ABC's Sneaky Plame Poll
By The Horserace Blogger Posted in Miscellanea — Comments (31) / Email this page » / Leave a comment »
I have a confession to make: I really have no interest in this Plame/Wilson/Rove “scandal.” As far as I can tell, this is one of those typical stories that the Washington press corps cooks up to find something to do while the President is on vacation and Congress is out of town.
I do, however, have an interest in bad polling done by major media outlets -- bad polling done with the ostensible purpose of adding lighter fluid to a story that seems to be on a permanent low-burn.
ABC News has decided to provide us with such a poll -- a poll which screams in an unequivocal voice: “I am garbage, please debunk me!”
Gladly...
The first major conclusion of the poll is that a majority people are actually paying attention to the story. ABC reports that 21% of the public is following the issue “very closely,” 32% “somewhat closely,” 18% “not too closely” and 29% “not closely at all.” Pretty impressive right? At first glance one would guess that the American public is as interested in this story as, say, Ron Brownstein or Tim Russert. However, this is not necessarily the case. I would say it is not even likely the case.
Consider the question which ABC News posed to the respondents:
As you may know, a federal prosecutor is investigating whether someone in the White House may have broken the law by identifying an undercover CIA agent to some news reporters. One reporter has gone to jail rather than reveal her source. How closely are you following this issue - very closely, somewhat closely, not too closely or not closely at all?
Take a close look at that introductory clause: “as you may know.” This is a big no-no if one’s agenda is to gain an accurate impression of the public’s level of knowledge. This clause is a priming or cueuing mechanism. Most people have heard of the story, I would agree, but this is designed to get it to the front of people’s minds. But the ostensible object of the question is to gauge the extent to which the question is at the front of people’s minds! In other words, that clause gets people into thinking, “Oh sure, I do know about that story.” If one is looking for a reliable estimate about the public’s level of knowledge, one cannot make suggestions to people about their level of knowledge -- even subtextual ones. This clause really has no other purpose that I can divine. The question could just as easily been asked absent it (indeed, prior polling done by ABC did not make use of this clause).
These problems with suggestion move beyond that first clause, though -- for the question itself gives the respondent a great deal of information about the investigation. The respondent learns (A) A CIA agent’s undercover identity may have been leaked; (B) Somebody in the White House may have leaked it to (C) somebody in the press; (D) A prosecutor has been investigating the matter and (E) has put press people in jail. It seems to me that knowing all of these facts could, in the mind of a respondent, constitute following the matter “fairly closely.” But more than this, though, these facts act as frames for the respondent to collect all of the random bits and pieces of information they have heard and collate them into one sensible story. In other words, before the question, many respondents probably had heard bits and pieces of the story from local tv news, a blurb in their local paper, a few minutes of NBC Nightly News caught while doing the dishes -- but had not really put all the pieces together. This question, however, puts those pieces together and therefore enhances one’s ability to say they have been paying somewhat close attention, even though they would not have said that if they did not receive all of this data.
In other words, what I am arguing here is not that the public is lying, but that ABC News is playing a subtle psychological trick with the public here -- trying to make them respond that they are paying attention when they are not actually paying attention. They do this by (A) telling them that they “might know”; (C) telling them basic facts of the story; (C) giving them a way to organize the random facts they might have heard into a coherent narrative.
Generally, questions that give answers like these are highly suspect because they go against what political scientists have known about the public since roughly 1948: they do not know much about politics. They suffer from incredibly low levels of information. Shocking low. So shockingly low that the main question that has been animating much of political science since the publication of The American Voter is whether the public is too ignorant for democracy -- and, if they are not, how they get around their low levels of information. For a poll to publish results about a decidedly inside-the-beltway story that shows more than half of the public is paying attention, you know something is fishy.
There is one final problem with this question that is the most devastating. It gets to the issue of self-selection. When you let the public self-select itself into a category like ABC is allowing, you are opening yourself up to big problems. Somebody might say, “Sure! I’ve been paying attention! You bet!” when in fact they are not really paying attention (read: their definition of attention is radically different than yours). To find out, objectively, how much attention the public has been paying, what you really need to do is quiz them on the story itself. Otherwise, you can have happen to you what happened to the authors of The Changing American Voter in 1979. This book was a response to The American Voter. The latter argued that only about 3% of the public was ideological. The 1979 response argued that, no no -- many more, like 25%, were capable of ideological thinking. The problem with this latter argument is that they were using self-identifications to make the argument. In other words, instead of asking people their opinions on taxes, foreign policy, welfare reform, etc and measuring one’s conservativeness or liberalness, the authors of the 1979 book were simply using what people told them they were. As it turned out, later scholars (most notably Donald Kinder in 1982) used the same data to find that, though people were saying they were conservative, they were not actually conservative. They had liberal, moderate and conservative opinions. So when it comes to this poll, the real issue is whether people are paying attention or whether they are just saying they are paying attention. My guess is the latter.
The second question is even more disingenuous than the first. ABC asks,
Do you think this is a very serious matter, somewhat serious, not too serious or not serious at all?
42% of the public responds “very serious.” 33% responds, “somewhat serious.” 11% responds “not too serious.” 8% responds “not serious at all.” They do not provide any kind of cross-tabulation to see if those who are not paying attention are the ones who think it is serious. Any serious person examining this poll would have this as the first question in their mind -- do these people who think it is serious actually know what is going on?
Beyond this issue, though, there is yet another issue of framing. One might be inclined to say that this is a very large number of people who think that this is at least somewhat serious. But consider the question people just finished answering before this one. In that question, ABC News tells the respondent that somebody has gone to jail over the issue! Hell! For most people, going to jail indicates a very serious matter. If they have not heard the context of the story, i.e. if they have been not been paying attention (and the previous question does nothing to argue that this is not the case), then they would probably be inclined to conclude that the matter is serious based only on the questions ABC News is asking.
You don’t need to be a statistician or a political scientist to realize this is ridiculous. In this poll, the questions themselves suggest which answer the respondent should select. This is patently absurd and lousy -- or should I say tendentious -- polling.
The third question they ask has the same trouble. It reads,
Do you think the White House is or is not fully cooperating with this investigation?”
25% say the White House is cooperating, 47% say the White House is not cooperating. 20% have no opinion. We cannot conclude that this means anything because, once again, they provide no cross-tabulation. 47% of the public admits they are not paying attention, and 47% of the public thinks the White House is not cooperating. Coincidence? Probably...but where are the cross-tabs, ABC? Why should we care what the uninformed think on the matter!
Beyond this, they have committed more grave methodological errors on this third question. Before they have even asked this question, they have hinted, in the subtext, at an answer. In the first question we reviewed, they implied that it is possible that the White House is responsible for this whole situation, that they were the leakers -- and that, once again, as a consequence of this whole fiasco, some journalist is in jail. This can potentially prime the respondent into an anti-White House position. If you have been paying very little attention to the story, and ABC News calls you up one night and tells you, in moment one, that the White House might be responsible for the leak; and, in moment two, asks you whether the White House has been cooperating...what are you going to say?
Take Home Point: The chances are very high that the average American is not paying attention to this. And political scientists have found that when people are not paying much attention to an issue, they are quite susceptible to “framing effects” that can be created through question wording and question ordering (for more detail, see John Zaller’s The Nature and Origins of Mass Opinion, 1992). These sorts of framing effects can be designed in such a way to give the impression that the public thinks something that it does not.
It seems very clear to me that this is the what is driving the numbers in this garbage ABC News poll. Intentionally or not, they have mucked around with question wording and question order to make it seem like (A) the public is paying attention, (B) the public thinks this issue matters and (C) the public is disatisfied with the White House.
Jay Cost, a graduate student of political science at the University of Chicago, is creator of The Horse Race Blog. He can be reached at jay_cost@hotmail.com.
« Today is worthy of celebration — Comments (7) | RedState: Better than PBS — Comments (14) »
ABC's Sneaky Plame Poll 31 Comments (0 topical, 31 editorial, 0 hidden) Post a comment »
which I am surprised you passed up.
someone in the White House may have broken the law by identifying an undercover CIA agent to some news reporters
Doesn't this also set up in the respondent's mind that leaking and breaking the law are one and the same? This will also drive the "is this serious?" question.
Why should we care what the uninformed think on the matter!
Because they vote. In fact, they generally vote Republican.
I don't know how many polls I've read that struck me as preposterous.
The results of this poll, made me think that something skewed the whole thing. Have Americans become sufficiently inured to the polling practices of the media to smell a rat before it walks in the door? Or am I just being overly cynical?
Mencken said that a cynic is someone who (upon smelling flowers) looks around for a coffin (or something to that effect). That's me all over.
You say: "42% of the public responds "very serious." 33% responds, "somewhat serious." 11% responds "not too serious." 8% responds "not serious at all." They do not provide any kind of cross-tabulation to see if those who are not paying attention are the ones who think it is serious. My guess is that the more one pays attention, the more one learns how ridiculous this story is."
ABC News says [pdf]: "Fifty-three percent are following the issue closely - a fairly broad level of attention. Those paying close attention (who include about as many Republicans as Democrats) are more likely than others to call it very serious, to say the White House is not cooperating, to say Rove should be fired if he leaked, and to say Miller is doing the right thing."
I agree that it would be nice to see the crosstabs, but it appears that your guess was wrong.
I changed the post after reading through the poll, but forgot to change the title.
I am concerned that too many people are not taking this brew-ha-ha seriously. There is a lot of lying, denying and falsifying going on. Obviously we won't know exactly what has happened until the prosecutor reveals his info but ..
I am probably older than the majority of people who read this blog and was around when the Watergate mess began and the similarities are great.
Pres. Bush should clean out the bad no matter who is involved. There are more important issues coming down the road.
that this whole affair is more tedious than important I think you also betray your own biases when you say this...
My guess is that the more one pays attention, the more one learns how ridiculous this story is
It is only ridiculous if nothing comes of it. Since no one knows, at this point, what will come of it, I think it is reasonable to find the matter serious at this juncture, if you so choose.
As I noted yesterday in my somewhat "over-the-top" post about this story, I don't expect the media to let this one go until the public's perception of the story matches the numbers they want to see in the polling, and the public says what the media wants them to say.
Today the Plame Leak story, in one form or another, is still front-page news at the New York Times, the Washington Post, and the Boston Globe. Don Imus was discussing it again this morning on MSNBC, and a trip over to News.Google.Com reveals that it is the #1 story under their "U.S." category, leading off with the San Francisco Chronicle and 1,567 related articles. I didn't check FOX or CNN this morning to gauge what was going on over there, but Bush's statements about firing anyone who broke a law is a front-page link on CNN's website, so depending on their story rotation I expect that it will be getting air throughout the day on CNN.
And today is relatively light, compared to yesterday, when you could watch it simultaneously on all three of the major networks on the morning news programs. In stereo.
So it may be a ridiculous story, but the media is taking it pretty darn seriously, at least right now.
Pres. Bush should clean out the bad no matter who is involved.
In the United States, the President has no authority to fire the press. (sorry)
In seriousness, though: I'm pretty disappointed in the administration's failure so far to defend itself against this attack by the MSM. How can they have let this go on for so long without pointing out what was in the amicus brief shows that she was outed early in Clinton's 2nd term? Even some Republicans have started to believe the media spin on this one. I was listening to the radio show hosted by former Republican Congressman Peter Blute this morning, and he was talking about how Bush "promised to fire the leaker". JEEEZ.
as a front page item the moment that President Bush nominates a Supreme Court justice.
The problem is there isn't much going on right now that the MSM considers newsworthy. Iraq is the same old same old. SCOTUS is on recess. Congress is on recess. Middle of the summer. However the MSM still has the same amount of space to fill with SOMETHING. Heck there isn't even a pointless murder/child abduction/pedophile story for the MSM to wrap their arms around. No natural disasters to talk about.
It is a dry news cycle. That is why Rove is front page stuff right now.
Are you watching the same TV programs I am? Hurricane Emily is the #1 News Item in the country right now, because it is going to make landfall at the Texas/Mexico border, apparently. The other big story today is the death of General Westmoreland.
Now, if the buzz surrounding Clements is correct, we may have a major SCOTUS story today which should push the Plamegate story back a notch or too.
But I don't expect that to last very long. I expect Plamegate to diminish gradually but stay very close to the surface throughout the rest of the summer, or at least until Fitzgerald concludes his investigation. IMO the media is going to continue to dig for new ways to keep this story in the public mind all the way through November. I hope I'm wrong.
that the White House is staying mum. Right now there is very little fact and a lot of speculation. They have made several tactical errors already regarding this story. McClellan should NEVER had denied Rove's involvement in this matter. President Bush should never have laid out consequences for any potential leaker.
This story will die of a lack of inertia shortly. It will likely build up momentum again in October when the Grand Jury is dismissed. However if the White House starts talking up the story all they will do is extend the life of this story and distract from the BIGGER story of their upcoming Supreme Court nomination.
is NOT a natural disaster. Perhaps when it makes landfall it might be newsworthy and perhaps it will completely peter out and be irrelevant. RIGHT NOW it is only a minor story.
Westmoreland passing away is not exactly a huge story. It is today's story and will be largely forgotten tomorrow.
I suspect that the MSM will continue to keep this story on the burner until the Grand Jury is dismissed but it will stop being front page stuff pretty soon.
If Bush nominates a SCOTUS, Plamegate becomes inconsequential.
My guess is that it will continue to be a big story. We'll see how it plays out.
One should consider that even with an inherently biased poll, question skewing only goes so far.
If leading questions were only responsible for 5 points of skewed results, then the results are still worth paying attention to. If leading questions are responsible for 80 points of skewing then bush must be ready to walk on water.
In my opinion the results are interesting but not conclusive. In other words, they demonstrate a definite trend that the republicans should be paying attention to when they decide to fall on political grenades to protect people like Rove (whom GHW Bush justifiably fired).
I don't know where you get your news, because Hurricane Emily at this very moment is the #3 link at CNN, the #1 World News story at news.google.com, the #1 International News story at abcnews.com, it's the top link next to Westmoreland's face at cbsnews.com, it is the Big Story at MSNBC.MSN.COM, the list goes on...
What planet are you reading the news on?
Did I say that no one is talking about Emily? What I SAID was there is a reason why the Rove story is still a top story. Sheesh we are talking about why Plamegate is getting so much coverage and now you're telling me how other stories are getting more coverage. OK. So then what's the big deal?
Everyday there will be new stories. Sometimes they catch and the MSM covers it for a while. Oftentimes they don't.
Last week Dennis was the story. Next week it will be Franklin. The week after that it will be Gert. But unless they cause some major damage they won't be lasting stories.
Leon, the second link, You Are Entering a Republican Zone was very helpful. Seriously.
Conservative and Liberal thought are really two very different world views, and it is honestly, and I make a genuine effort, honestly difficult to understand where you are coming from.
Thanks.
WH makes decisions on its own polls, not statistically worthless polls like ABC. If you want a true Rovelation, take your local phone book,(as long as you don't live inside the beltway), and call random numbers until you ask 10 adults, "Do you know who is Karl Rove". My bet is one, maybe two out of 10 can come close to a correct answer. Then, just for fun, ask that one or two how he is related to Judith Miller.
That you and I have a disagreement on how much emphasis the media is going to continue to place on the Plamegate story for reasons of their own. You claim that there are "no natural disasters" and that it's a "dry news cycle" when in fact Hurricane Emily is among the top news stories in the country and the world (that's a little more than just a smattering of people talking about it.) You claim that Plamegate is only news because there isn't anything else for the MSM to report, as though they can't walk and chew gum at the same time, and aren't keenly interested in this story, and I simply disagree with you. You claim that a SCOTUS nomination will push Plamegate off the front page, and I think it will make it a little less of a story, but that it will continue to be developed and placed in the public view rather prominently regardless of what is going on with SCOTUS or anything else that happens.
There will be an easy way to test whether I'm right or wrong -- watch the frequency of the Plamegate stories, op-eds, and related stories over the next few weeks alongside whatever else the MSM considers newsworthy. I said I hope that I'm wrong, and that's the most direct way to test it. As of today, I see no signs of it becoming a marginal story. We'll see.
I suspect you believe what you do because you think that the MSM is some monolithic group hellbent on further some Liberal agenda.
I see the MSM as a big pot of various businesses hellbent on making a profit. They have no overarching agenda.
I turned on Fox News this morning and the FIRST STORY they started talking about was Plamegate. They had political analysts on to discuss it. Commentary was aplenty. Then they mentioned that Westmoreland died. Then they mentioned that Emily was looking like trouble.
Of these three stories Plamegate is EASILY the most newsworthy in the sense that it it contentious and most likely to get the viewers attention. Westmoreland passing away has some momentary interest but isn't terribly contentious or emotional. Emily MAY become a big personal tragedy story but right now is more along the lines of "we're keeping track of this".
But as you said we shall see.
I really do hope I'm wrong, because in the absence of genuinely new information, this story should drop off the front pages for a while, if the Media is as professional as it would like us to consider it, in order to regain some semblance of the credibility they've lost (and I'm not the only one saying they've lost that credibility - Nick Kristof agrees with me.)
So let's find out just how seriously they take their mission of informing the public with actual news, instead of manufacturing it for one reason or another. I have an open mind about it, but I'm skeptical of them for exactly the reason you suggest, and I think you're being honest when you say that you're skeptical of them for yours. But I fail to see how you could be arguing for them to stop covering this story, because it is fairly widely recognized as doing damage to the Administration. If they stop covering it, I guess you'll argue that it will reinforce the case that they're not being hard enough on the Administration, and then Frank Rich will write some op-eds about how the media is letting the story die when they should be keeping it alive.
I became a Horserace Blogger fan when you definitively called the election in Ohio well ahead of anyone - particularly following your solid and highly educational lead up work on polls. I even came second in Norm Geras' US election poll contest riding on your excellent predictive work. That said, this is another superb post because, once again, it harnesses your academic skills in the public interest - in this case to defrock a particularly nasty modern propaganda method - the push poll. As to Palmegate - the name says it all - its just deja vu all over again. Still it could have a nasty effect on the President if they manage to kinda, sorta hang somthing on someone, anyone on his staff. Rove, on the other hand, looks like he has managed to stay submerged long enough to prove he is not guilty of witchcraft and will manage to draw breath on the far shore. Its pretty clear since 'Rathergate' that the MSM will just plain fabricate stories. This post outs how far they will go to create phoney backup up for a story of little or no substance.
depends on how many leaks we have, especially through August during the reccess.
I expect there to be leaks from both sides in this over the next few weeks, and as those leaks come to light, the pot will get stirred and it will come to the forefront.
why you think that I would want a story to be carried simply because it damages the Administration.
I don't consider myself a partisan. While I will freely admit I lean leftwards I am more than willing to vote for a Republican. And if Iraq had never occurred I would probably be generally supportive of this President.
I think the MSM takes their mission very seriously. However their mission is to make money.
The pundits are the most insidious people in these types of stories because they are the most ardent peddlers of these non-stories because it pays their bills. Frank Rich needs to convince us why this is a big story and Rich Lowry needs to convince why it isn't a big story. By doing so they keep their own names relevant.
Because I think that goes right to the heart of it. Thank you.
"...even with an inherently biased poll, question skewing only goes so far."
Is ABC News consciously or subconsciously biased? I'm not sure which is worse: that they are biased and don't know it, or that they are knowingly trying to turn opinion against the Bush administration.
Saying that "skewing only goes so far" is like saying, "only one of watermelon in this field is poisoned."
on methodology we used to discuss bad poll questions all the time.
Media polls were one of my proff's favorite things to use as examples of how not to do polls-and the polls have only gotten worse in the last 13 years.