As La Bamba sings: “In The Research Two-Thousand, In The Research Two-Thousand!”

Markos Moulitsas at Daily Kos:

A bit over two weeks ago, a group of statistic wizards (Mark Grebner, Michael Weissman, and Jonathan Weissman) approached me with a disturbing premise — they had been poring over the crosstabs of the weekly Research 2000 polling we had been running, and were concerned that the numbers weren’t legit.

I immediately began cooperating with their investigation, which concluded late last week. Daily Kos furnished the researchers with all available and relevant information in our possession, and we made every attempt to obtain R2K’s cooperation — which, as I detail in my reaction post here — was not forthcoming.  The investigators’ report is below, but its conclusion speaks volumes:

We do not know exactly how the weekly R2K results were created, but we are confident they could not accurately describe random polls.

The full report follows — kos

More Kos:

Since the moment Mark Grebner, Michael Weissman, and Jonathan Weissman approached me, I took their concerns seriously and cooperated fully with their investigation. I also offered to run the results on Daily Kos provided that they 1) fully documented each claim in detail, 2) got that documentation peer reviewed by disinterested third parties, and 3) gave Research 2000 an opportunity to respond. By the end of last week, they had accomplished the first two items on that list. I held publication of the report until today, because I didn’t want to partake in a cliche Friday Bad News Dump. This is serious business, and I wasn’t going to bury it over a weekend.

We contracted with Research 2000 to conduct polling and to provide us with the results of their surveys. Based on the report of the statisticians, it’s clear that we did not get what we paid for. We were defrauded by Research 2000, and while we don’t know if some or all of the data was fabricated or manipulated beyond recognition, we know we can’t trust it. Meanwhile, Research 2000 has refused to offer any explanation. Early in this process, I asked for and they offered to provide us with their raw data for independent analysis — which could potentially exculpate them. That was two weeks ago, and despite repeated promises to provide us that data, Research 2000 ultimately refused to do so. At one point, they claimed they couldn’t deliver them because their computers were down and they had to work out of a Kinkos office. Research 2000 was delivered a copy of the report early Monday morning, and though they quickly responded and promised a full response, once again the authors of the report heard nothing more.

While the investigation didn’t look at all of Research 2000 polling conducted for us, fact is I no longer have any confidence in any of it, and neither should anyone else. I ask that all poll tracking sites remove any Research 2000 polls commissioned by us from their databases. I hereby renounce any post we’ve written based exclusively on Research 2000 polling.

I want to feel stupid for being defrauded, but fact is Research 2000 had a good reputation in political circles. Among its clients the last two years have been KCCI-TV in Iowa, WCAX-TV in Vermont, WISC-TV in Wisconsin, WKYT-TV in Kentucky, Lee Enterprises, the Concord Monitor, The Florida Times-Union, WSBT-TV/WISH-TV/WANE-TV in Indiana, the St. Louis Post-Dispatch, the Bergen Record, and the Reno Gazette-Journal. In fact, just last week, in an email debate about robo-pollsters, I had a senior editor at a top DC-based political publication tell me that he’d “obviously” trust Research 2000 more than any automated pollsters, such as SurveyUSA. I didn’t trust Research 2000 more than I trusted SUSA (given their solid track record), but I did trust them. I got burned, and got burned bad.

I can’t express enough my gratitude to Mark, Michael, and Jonathan for helping bring this to light. Sure, our friends on the Right will get to take some cheap shots, and they should take advantage of the opportunity. But ultimately, this episode validates the reason why we released the internal numbers from Research 2000 — and why every media outlet should do the same from their pollster; without full transparency of results, this fraud would not have been uncovered. As difficult as it has been to learn that we were victims of that fraud, our commitment to accuracy and the truth is far more important than shielding ourselves from cheap shots from the Right.

Soon, we’ll have a new pollster (or pollsters) to work with, helping us to fulfill our vision of surveying races and issues that are often overlooked by the traditional media and polling outfits. As for Research 2000, the lawyers will soon take over, as Daily Kos will be filing suit within the next day or two.

Justin Elliott at Talking Points Memo:

R2K president Del Ali tells TPMmuckraker in an email, “I have much to say, however, I am following my attorney Richard Beckler’ ESQ’s counsel and referring all questions to him. I will tell you unequivocally that we conducted EVERY poll properly for the Daily Kos.”

Beckler is the co-chair of the Securities Litigation, Government Enforcement and White Collar Defense practice at Howrey LLP in Washington. Kos, for his part, tell TPMmuckraker that Daily Kos is represented by Adam Bonin of Cozen O’Connor in Philadelphia. Bonin’s bio says he “represents clients in campaign finance, election law and lobbying compliance matters and has been a leader in efforts on behalf of the rights of online speakers.”

Kos has promised to sue R2K in the next day or two.

R2K is an extremely well-established pollster that has been employed and quoted by countless national and state-level newspapers as its website describes:

Some of our most active media clientele include the Bergen Record, The Raleigh News & Observer, The Concord Monitor, The Manchester Journal Inquirer, The New London Day, The Reno-Gazette, The Fort Lauderdale Sun-Sentinel, The Spokesman-Review, KCCI-Television in Des Moines, Iowa, WRAL-Television in Raleigh, North Carolina, St. Louis Post-Dispatch, and KMOV-Television in St. Louis, Missouri.Our Polls can be seen on CNN’S “Inside Politics” and are also mentioned frequently in the National Journal’s “Political Hotline”, The Washington Post, The New York Times, The Christian Science Monitor, and The Wall Street Journal

Kos said in a post today that R2K has not offered a substantive response to the statistical analysis. R2K President Ali is described in his bio as having “analyzed over 2000 political races for media as well as for advocacy organizations and businesses.

Late Update: Here’s more from lawyers for Kos and Ali.

Nate Silver:

Although I expect to proceed fairly carefully with respect to Research 2000, which Daily Kos will be suing for alleged fraud, I have suggested here and to at least one reporter that I had my own suspicions about Research 2000 which paralleled some of the findings in the study by Mark Grebner, Michael Weissman, and Jonathan Weissman. I want to be a bit more explicit about what I mean by that.

This is a copy of two e-mails that I sent to Mark Blumenthal of Pollster.com in the wee hours of the morning on February 4th. Like the examples in the Grebner study, they point toward cases in which Research 2000’s data appeared to be other-than-random (although, as I declaim in the e-mails, not necessarily triggered by fraud).

from Nate Silver [xxx@xxx]
to Mark Blumenthal [xxx@xxx]
Mark Blumenthal [xxx@xxx]
date Thu, Feb 4, 2010 at 4:17 AM
subject Research 2000 weirdness
mailed-by gmail.com

Mark,

Not to sound too conspiratorial, but to be honest I'm getting
a little bit suspicious about Research 2000, or at least the
polling they've conducted for Markos over the past two years.
Do you know those guys at all?

I'll keep this pretty brief.  In part it's because of the
occasionally really weird result they turn out -- for instance,
they had only 27 percent of Republicans or something in favor
of gays in the military whereas Gallup and ABC/Post have had
those numbers in the 60s.  There are two or three other examples
like this I could point to.  For another, their contact
information and web presence is pretty sketchy relative to that
of other pollsters and there's not a lot of detail about the
scope of their operations.

But mainly, it's that that their data feels way too clean for
me.  Take a look at the attached chart, for example: these are
the age breakdowns in the Democratic vote share for the last
20 contests surveyed by R2K and PPP, respectively.  The age
breakdowns in Research 2000's numbers are almost always close
to "perfect" -- in 20 out of 20 cases, for instance, the
Democrat gets a lower vote share from among 30-44 year olds
than among 18-29 year olds.  PPP's data, on the other hand,
is *much* messier -- which is what I think we should expect
when comparing small subsamples, particularly subsamples of
lots of different races that are subject to different
demographic patterns.

Likewise, take a look at their Presidential tracking numbers
from 2008 (http://www.dailykos.com/dailypoll/2008/11/4).
They published their daily results in addition to their
three-day rolling average ... and the daily results were
remarkably consistent from day to day.  At no point, for
instance, in the two months that they published daily results
did Obama's vote share fluctuate by more than a net of 2
points from day to day (to reiterate, this is for the daily
results (n=~360) and not the rolling average).  That just
seems extremely unlikely -- there should be more noise than
that.

Megan McArdle:

To his eternal credit, that evidence is being published by Kos himself.  I wish more commentators were this forthright when they’ve been taken in by bad data.

I also wish he hadn’t been taken in by bad data, including this infamous poll which proves that Republicans are, like, the awfulest, stupidest people in the entire known universe.  Given the patterns that the investigators who published on Kos have found, it seems like that poll is probably bunk–concocted or massaged to fit the thesis of the person who commissioned it.  (Just to be clear, I don’t think Kos knew that this was happening.  A customer who is told what he wants to hear is a customer who is happy to buy more polls.)

However, the fact that R2K has apparently been discredited will not prevent this poll from popping up, over and over, in the work of pundits and bloggers.  The speed and openness with which Kos has exposed this problem is admirable.  But we need to be even faster at sniffing out bad apples before they pollute the data barrel.

Jonah Goldberg at The Corner

Jim Geraghty at NRO:

Even worse for Markos Moulitsas, he apparently was working on a book, entitled “American Taliban” that relied heavily on a Research2000 poll of American conservatives.

UPDATE: Markos declares via Twitter, “No premises in American Taliban depend exclusively on R2K polling.”

On his site, he elaborates, “Of references to R2K, except in two instances where I couldn’t do so without affecting page count (too late for me to do that since the index was done), but those two examples also references other supporting polling, so my premise didn’t depend on the R2K results.”

Moe Lane:

Jim Geraghty is too polite to admit that he’s laughing his ass off at the thought that Markos Moulitsas is going to go sue R2000 for supposedly faking up the polls done for his website.

I’m not.

HAHAHAHAHAHAHAHAHAHAHAHAHAHAHA
HAHAHAHAHAHAHAHAHAHAHAHAHAHAHA
HAHAHAHAHAHAHAHAHAHAHAHAHAHAHA
HAHAHAHAHAHAHAHAHAHAHAHAHAHAHA
HAHAHAHAHAHAHAHAHAHAHAHAHAHAHA
HAHAHAHAHAHAHAHAHAHAHAHAHAHAHA
HAHAHAHAHAHAHAHAHAHAHAHAHAHAHA

Did I mention that we’ve been calling these polls crap for a long time?  And that no one with the sense that God gave a drunken wombat took these polls seriously?

HAHAHAHAHAHAHAHAHAHAHAHAHAHAHA
HAHAHAHAHAHAHAHAHAHAHAHAHAHAHA
HAHAHAHAHAHAHAHAHAHAHAHAHAHAHA
HAHAHAHAHAHAHAHAHAHAHAHAHAHAHA
HAHAHAHAHAHAHAHAHAHAHAHAHAHAHA
HAHAHAHAHAHAHAHAHAHAHAHAHAHAHA
HAHAHAHAHAHAHAHAHAHAHAHAHAHAHA

Guess who was writing a book that relied on R2000 telling Markos everything that he wanted to hear about how awful we RepubliKKKans are?

HAHAHAHAHAHAHAHAHAHAHAHAHAHAHA
HAHAHAHAHAHAHAHAHAHAHAHAHAHAHA
HAHAHAHAHAHAHAHAHAHAHAHAHAHAHA
HAHAHAHAHAHAHAHAHAHAHAHAHAHAHA
HAHAHAHAHAHAHAHAHAHAHAHAHAHAHA
HAHAHAHAHAHAHAHAHAHAHAHAHAHAHA
HAHAHAHAHAHAHAHAHAHAHAHAHAHAHA

mistermix

Ed Morrissey:

Give Markos Moulitsas some credit for full disclosure, at least.  It would have been very easy for dKos to quietly drop Research 2000 as their pollster and put their resources into a more worthy vendor without ever explaining why.  That would have certainly cost them less than hiring attorneys to sue the pollster.  Given the fact that the firm claims to be operating “out of a Kinko’s office,” it seems less than certain that Markos will ever see much of his money returned to him, assuming he can prove fraud, which is not an easy task.

I’m also not inclined to crow over the failure of dKos polling.  That ship had sailed long ago anyway, and Markos knew it, which is why he launched an independent study of the data.  Political campaigns commission their own polls and so do traditional media outlets.  If Markos had the resources to do the same, why not?  It allowed him to play with the bigger outlets, although in the end the task proved beyond his ability to manage it.  Still, one never knows until one tries.

He also makes another good point:

Sure, our friends on the Right will get to take some cheap shots, and they should take advantage of the opportunity. But ultimately, this episode validates the reason why we released the internal numbers from Research 2000 — and why every media outlet should do the same from their pollster; without full transparency of results, this fraud would not have been uncovered. As difficult as it has been to learn that we were victims of that fraud, our commitment to accuracy and the truth is far more important than shielding ourselves from cheap shots from the Right.

Again, I’m disinclined to take cheap shots, mainly because Research 200o did have an existing clientele that lent respectability to their product.  It’s also important to note that this is Markos’ take, and that the firm itself will probably have a very different position on the end of their relationship.  But Markos is right when he warns that pollsters should provide full transparency for their surveys so that people can assess performance properly, such as sample size, question formulation, demographics, and so on.  Without that information, polls can easily manipulate both respondents and readers alike.

The only criticism that comes to mind is the lawsuit itself.  Why bother?  Markos would be better advised to cut his losses.

UPDATE: Greg Sargent

Dan Amira at New York Magazine

UPDATE #2: Patrick Ruffini at The Next Right here and here

Allah Pundit

UPDATE #3: David Freddoso at Washington Examiner

Advertisements

2 Comments

Filed under New Media

2 responses to “As La Bamba sings: “In The Research Two-Thousand, In The Research Two-Thousand!”

  1. Pingback: What We’ve Built Today « Around The Sphere

  2. Pingback: What We’ve Built This Weekend « Around The Sphere

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s