The WordPress.com stats helper monkeys prepared a 2011 annual report for this blog.
Here’s an excerpt:
The concert hall at the Syndey Opera House holds 2,700 people. This blog was viewed about 36,000 times in 2011. If it were a concert at Sydney Opera House, it would take about 13 sold-out performances for that many people to see it.
By David W. Moore, University of New Hampshire
Responses to my article in the August 2011 issue of Survey Practice, “Contemporary Issues with Public Policy Polls,” suggest there are possibly three general schools of thought with respect to measuring opinion. These are somewhat arbitrary classifications, and I don’t hold to them tenaciously, but I think they may be useful as a heuristic device to stimulate discussion about the various views that scholars and practitioners have about “public opinion.”
Literalism
At one end of the spectrum is Literalism, what Howard Schuman (in the article in this issue) refers to as “survey fundamentalism” – “the belief that some polls tell us the literal truth about public opinion.”
I believe that most public policy polls reported by the major news media outlets fall into this category. Associated with this school is the implicit, though prevalent, view that public opinion is what the pollsters say it is – regardless of whether pollsters measure intensity of opinion, non-opinion, or hypothetical opinion (see my Survey Practice article for an elaboration of these points).
Often times, pollsters asking about the same issues will produce conflicting results, but rather than assessing which results might be more valid, the effort is to harmonize the results – to explain that differences occur because of legitimate differences in question wording, questionnaire context, and timing, but that essentially all are valid measures of a complex issue. Rarely are judgments made that perhaps one approach is more valid than another. Long live diversity. The more polls, the more nuances we see in what the public is thinking. Pollster.com on Huffingtonpost is a good source for this type of no-fault analysis. I wrote many such articles when I was employed by the Gallup Organization.
Nihilism
At the other end of the spectrum is what I would call Nihilism. It’s the notion, articulated by Schuman and Scott (1987), that public opinion measures are so tenuous that no matter how carefully worded, they cannot provide a valid measure of public opinion. The solution to this problem, the authors write, “requires giving up the hope that a question, or even a set of questions, can be used to assess preferences in an absolute sense…” In his article in this issue, Schuman writes that “Study of change over time or of the differences between educational levels, can provide a plausible basis for a judgment about public opinion, but the marginals in any simple sense should almost never be taken literally, no matter the wording.”
The implication here is similar to the Literalism school – that almost any question wording approach is no better than any other, at least in the sense of providing an accurate picture of public opinion. But rather than say they all provide valid measures of public opinion, this school holds that none of them provides valid measures, because public opinion itself is too nebulous a concept to measure in any absolute sense.
That’s Schuman’s criticism of the bike lane expansion example I present in the August 2011 Survey Practice article. “Were New Yorkers faced with voting in a referendum on the bicycle lane issue, it’s hard to know which of the questions [presented in a split sample experiment] would be more predictive, if we take predictive validity to be important.”
(Just a reminder: One question showed a substantial majority support for expansion of bike lanes, with a paltry 4% without an opinion – though the same poll showed 40% of respondents paying little to no attention to the issue, and only 28% paying a lot of attention. The other question showed a little over a fifth of the public in favor, about the same amount opposed, and just over half with no meaningful opinion.)
That Schuman suggests it’s impossible to make a judgment as to which of these two wildly different results provides a more realistic assessment of public opinion is consistent with what I term the nihilistic school of thought.
Explicit with this school of thought is an indeterminate definition of public opinion, which essentially argues that public opinion is too vague a concept to permit any poll to actually measure what it is.
Realism
In between the two ends of the spectrum is what I would call the Realism school of thought. It holds that polls can give a meaningful measure of public opinion, even in an absolute sense, if they are conducted correctly. It takes into account both non-opinion and opinion intensity, and attempts to differentiate – in the words of Daniel Yankelovich (1991) – between the public’s “top-of-the-mind, offhand views (mass opinion) and their thoughtful considered judgments (public judgment)” – which Yankelovich criticizes most media polls for failing to do.
In the bike lane expansion issue mentioned earlier, the Realist school would argue that a realistic picture of the public, taking into consideration both admitted non-opinion and intensity, suggests that the opinionated public is about evenly divided over the issue, with a little more than half of the residents so unengaged in the issue, they have formulated no meaningful opinion. The exact percentages are less important than the overall picture.
This interpretation clashes with the Literalist school’s view, which accepts the technique of pressuring respondents to make a choice, resulting in 96% appearing to have an opinion – when initially not even a third were following the issue closely.
Implicit in the Realist school of thought is that opinions, as opposed to non-opinions, are views that respondents feel strongly enough about that they want their elected representatives to take such views into account. That was George Gallup’s explicit view (in Gallup and Rae, 1968), when he said polls could provide elected leaders with an ongoing picture of what the public was thinking, so public opinion could be incorporated into leaders’ decisions.
Survey Practice Articles
I would classify all the articles in this issue, except for Schuman’s, in the Realism camp. Initially it appeared that the article by George Bishop and Stephen Mockabee embraced Nihilism. Their critique suggests that measurements over time, even using the same question, do not necessarily provide a realistic picture of trends in public opinion – because the meaning of the questions (even if identical at all time periods) could change from one period to the next. But when I suggested to them that their critique implied no meaningful measures over time could ever be taken, they added a section that recommends using various experimental methods – among them the random probe (originally described by Schuman, 1966) – to clarify how respondents interpret the questions.
The other articles in this issue of Survey Practice all clearly imply that polls can meaningfully measure public opinion (the “will of the public”) on an absolute basis, but only if the polls are conducted properly.
Mike Traugott’s concern, for example, is that pre-election polls this year are producing a large variance in their estimates (i.e., significantly different results from each other), which suggests disaster this presidential election season, similar to what happened four years ago in the Democratic nomination contests, when “the pre-election polls systematically underestimated the winner’s share of the vote…by an amount that was typically greater than sampling error would admit.” But it’s not clear this year why there are such divergent poll findings, because poll methodologies are not fully available. Traugott would like all the polling organizations “to be more forthcoming about their methods now rather than trying to recover such information after the fact.”
Traugott also objects to the use of national polls of Republicans to characterize how primary voters feel, because the results don’t necessarily reflect the views of early caucus and primary voters in Iowa and New Hampshire. “National polls that include Democrats and Republicans in their samples do not provide any guidance about what might happen in the caucus and primary in these two states… and in fact they may be confusing some journalists who are covering the first two events.” Yet national polls are widely used to talk about primary voters (mostly because it’s easier to poll nationally than it is to poll in individual states, even if the national polls are of the wrong electorate).
Seth Rosenthal gives two extended examples of how the wording of the questions and the actual results were not consistent with the widespread interpretation of those results. Often pollsters have to speculate on the meanings of their results, given the ambiguity of some questions, and here Rosenthal writes that “we should consistently be willing to recognize the point at which our data ends (or becomes difficult to disentangle), and our own interpretation begins.”
In an extensive research article, Mark Nance and Michael Cobb examine the consequences of not measuring non-opinion in the area of trade. Their conclusions are worth noting: “First, non-attitudes appear rampant. Secondly, they alter the aggregate distribution of trade preferences, in many cases changing whether a majority supports or opposes it… Overall, our early findings suggest that the variables of most interest to researchers in this field may be affected by non-attitudes and, as such, researchers should be careful to account for the impact of non-attitudes in their analyses.”
Sid Groeneman concurs that in public policy polling, “non-opinions are frequently overlooked or too casually glossed over.” He later notes: “I might also add that findings of low intensity views, like extensive non-opinions, can run counter to the agenda of poll sponsors, who may resist disclosure of such results.”
In the other research paper in this issue, Patrick Murray reports on an example of hypothetical public opinion. The issue: Whether New Jersey Senator Frank Lautenberg, at age 84, was too old to run for re-election in 2008. When respondents were given his age, a majority said he was too old; when respondents were not told his age, a majority said he was not too old. Murray concludes: “Informing the sample of Lautenberg’s actual age skewed the results in a way that no longer reflected what the population of voters actually felt about Lautenberg’s age, but rather how they may have felt if everyone was aware of his age. In reality, most voters did not consider his age to be an issue, either because they underestimated his age or simply did not know what it was and the issue was not salient.”
By not distinguishing between hypothetical opinion and actual extant opinion, the Quinnipiac Poll, which consistently informed its respondents of the Senator’s age, led to misleading media stories about the damaging age factor in the campaign.
Murray’s caveat about how to deal with hypothetical opinion is important: “The bottom line is if you are measuring the potential salience of factual information on opinion formation then be forthright about what you are doing. If, on the other hand, you wish to tap extant opinion representative of a larger population, make sure your question does just that. How pollsters present their findings has as much, if not more, of an impact on the public debate as the questions and results themselves.”
Groeneman agrees that asking questions that produce hypothetical opinion is a useful way to speculate about what public opinion might be, but in such cases pollsters need to carefully qualify their presentation of the results to avoid giving the impression the results represent what the public is already thinking.
Finally, in his commentary in this issue, AAPOR’s current vice president and president-elect, Paul Lavrakas, makes three suggestions “for improving the value that public policy polling has for our nation’s decision-makers and the public at large”: conduct more question wording experiments, pay more attention to the quality of public opinion as described by Daniel Yankelovich, and improve the data analysis.
Lavrakas is particularly concerned about the quality of opinion – differentiating between “offhand views” and “considered judgment,” or what I earlier characterized as the whim vs. the will of the public. He writes: “Here is an arena that I believe AAPOR can and should make a much more muscular effort to raise the quality of public opinion polling by providing more education to public policy pollsters and editors/journalists about how to better measure and interpret the public’s opinion on matters that matter.”
However, Lavrakas is skeptical about improvement in polls: “I am not sanguine that any of these suggestions will be implemented soon or that a meaningful change will result in the quality by which public policy pollsters measure public opinion.” Still, he writes, it’s “another area that AAPOR can (and I believe should) take more aggressive action in the coming decade.”
Howard Schuman
David’s thoughts on the measurement of public opinion, and his example comparing three polls on the expansion of bicycle lanes in New York City, seemed to me interesting and of value. And how could I disagree with his emphasis on the importance of taking into account Non-Opinion and Intensity. Many pages in my book, Questions & Answers in Attitude Surveys: Experiments on Question Form, Wording, and Context, written with Stanley Presser (1981/ 1996) are devoted to just these issues. But this raises one of two reservations I had on reading his column.
David writes as though the issues have not been much studied in the past. He mentions in passing early recommendations by Dan Katz (1944), but ignores the fact that hundreds of experiments have been carried out on these issues between that time and today. No attempt is made to review these, whether critically or in any other way, nor consider seriously what has been learned and what still needs to be investigated. This disregard for the past is not uncommon among survey practitioners, perhaps because of a focus on the very latest data about presidential candidates, the right or wrong direction of the country, city bike lanes, and the like. Such results are certainly of interest to most of us, but if surveys are to contribute to knowledge and understanding, and not only to today’s news, more is needed. Those who cannot remember the past are doomed to repeat it, warned Santayana many years ago. Read More »
George F. Bishop and Stephen T. Mockabee, University of Cincinnati
A cardinal assumption we make in asking any survey question is that it should mean essentially the same thing to all respondents. Experienced survey research practitioner Floyd Fowler, Jr. (1995, p.84; cf. Belson, 1981) has expressed this principle in his recommendations for improving question wording: “…A survey question should be worded so that every respondent is answering the same question [emphasis added].” Furthermore, as survey methodologist Robert Groves (1989, p. 450) summarized the issue: “Although the language of the survey questions can be standardized, there is no guarantee that the meaning assigned to the questions is constant over respondents.” This becomes critical because “A fundamental tenet of scientific measurement is that the measuring device is standardized over different objects being measured (Groves, 1989, p.449).” Indeed, without such standardization how would we determine if our measurements are reliable and valid? Over a quarter of a century ago, political scientist Henry Brady (1985, p.269) reminded us that the “lack of interpersonal comparability of survey responses” was a greatly neglected and “…serious difficulty… largely ignored by social scientists.” Nearly 20 years later, Gary King and colleagues (2004) reminded us again of the potentially “devastating consequences” of ignoring the incomparable responses given to most survey questions, while calling for the increased use of anchoring vignettes to enhance the comparability of measurements. Their wise counsel, and that of Belson, Brady,Groves, and Fowler, among others, has gone largely unheeded—at our paradigmatic peril. Read More »
Michael W. Traugott, University of Michigan
Voting in Iowa and New Hampshire is only a few weeks away, but variance in their early estimates suggest that there are storm clouds on the horizon for the pre-election pollsters – and therefore for everyone involved in the polling and survey research business. Based upon a variety of analyses conducted after the 2008 campaign and the problems that pre-election pollsters had in those primaries, we know that the image of the entire industry rests substantially on the performance of the pre-election polls (1). This is due partly to the fact that pre-election polls have a peculiar external validation in the actual outcome of the election that most other polls do not have. In addition, the way that the news media report on “the polls” in the aftermath of an estimation error, especially when they systematically get the winner wrong as they did in New Hampshire, lays generic blame on the method and all those who apply it. Read More »
Seth A. Rosenthal, Merriman River Group and Center for Public Leadership, Harvard Kennedy School
As survey researchers often looking to gain the biggest headline or snag the biggest client, we sometimes lose sight of the differences between the answers our data can provide, and those that it can’t. Ideally, we would treat our data as the key that helps unlock the latent constructs (i.e., feelings, beliefs, and/or behaviors) that underlie the responses we receive. But unfortunately, we, and the people who report on our polls, often neglect our data’s limitations. We speculate about the underlying reasons for particular results without making it clear to our audience where our data ends and our conjecture begins. Our musings can be taken very seriously, often more seriously than the actual toplines and crosstabs we report. Because the interpretations of our polls are what are most likely to shape public discourse and opinion, I believe that we have the responsibility to police ourselves and our colleagues when our interpretations stray too far from our data. Read More »
Michael D. Cobb, North Carolina State University; Mark T. Nance, North Carolina State University
In a recent Survey Practice article, David Moore (2011) argues that policy polling is not living up to its democratic potential. As he sees it, and we generally agree, polling frequently manufactures a fictitiously interested and attentive public. Pollsters’ sins include pushing respondents to reveal preferences when no real opinion exists and failing to examine the (lack of) intensity of respondents’ answers.
While Moore’s criticisms of contemporary survey research are not entirely unique, more novel and controversial is his method of asking respondents if they would be upset if their preferences went unheeded in order to distinguish between “permissive” and “directive” opinions. Further, some pollsters are skeptical about the applicability of his critique to every issue (Traugott, 2009). The question is an important one. If Moore is right that public opinion usually reflects the questionable practices of survey researchers rather than the aggregation of thoughtful and deeply held preferences, why should public opinion, as it is most commonly measured, influence policy debates? Read More »
Patrick Murray, Monmouth University Polling Institute
One of the most challenging aspects of polling public opinion on policy issues is crafting the actual questions. They must be able to elicit a range of responses without affecting the pre-existing knowledge base of the survey sample. For most policy questions, the concern is that the question will be skewed to one side or another. In many cases, “explanatory” wording is included to balance perceived bias in one part of the question.
For example, a question gauging support for a “jobs bill” may be seen as inherently positive. After all, who can be against job creation? A pollster can attempt to balance that by including information on the cost, e.g., “$450 billion jobs bill.” Introducing that factual information, though, can have another unintended impact on responses. It introduces factual information to the survey sample that the full population does not necessarily know. It is then unclear whether the pollster is measuring extant support for a particular policy or hypothetical support if the full population had this information at hand.
A question wording experiment run during the 2008 U.S. Senate race in New Jersey provides evidence that including factual information in a poll question can be consequential when trying to portray where the public stands on an issue. While this particular experiment did not focus on a policy issue per se, the implications for measuring public attitudes on policy issues are apparent. Read More »
Sid Groeneman, Groeneman Research & Consulting
David Moore’s August 2011 article highlights three enduring challenges facing public opinion researchers. First off, non-opinions are indeed frequently overlooked or too casually glossed over. My approach to minimizing this problem is to judge whether all or nearly all to be surveyed can reasonably be expected to hold a pre-existing view about the policy, program, issue, individual, or institution of interest. If the answer is yes, then the response options presented may exclude “don’t know,” “not sure,” or “no opinion.” If the answer is no—probably the more common situation—then one such option should be included. In cases when “don’t know,” “not sure,” or “no opinion” is not offered, it should be accepted by respondents who volunteer it. Whether or not “don’t know,” “not sure,” or “no opinion” is presented, it is generally good practice to make it easy for respondents to opt out by noting that “some people have not heard of or thought much about” [topic]. Admittedly, judging the extent to which all are likely to hold an opinion is not always easy. When uncertain, it is better to assume not. One further point: it is often important to try to distinguish respondents who have given some thought to the issue and come down in between (neither for or against but with some qualified position) from those who have not thought about it at all (true non-opinions). Read More »
Paul J. Lavrakas, Independent Consultant
There are three suggestions I have for improving the value that public policy polling has for our nation’s decision-makers and the public at large. They all have to do with improving the quality (reliability and validity) of poll findings.
First, as Fowler (2004) has observed, experimentation to improve the wording of survey questions (and for other goals) is woefully underutilized. Why might this be the case in something as important as public policy polling? One answer may be that many of those who sponsor and conduct these polls may have too narrow a perspective of the costs and benefits of incorporating experiments on question wording into their survey instruments. Read More »
