In November 2009, the American Association for Public Opinion Research announced its Transparency Initiative, a program to place the value of openness at the center of our profession, and to encourage and make it as easy as possible for survey firms to be transparent about their research methods. The initiative is a work in progress, and your input will help it succeed. Please share your thoughts on this initiative in the comments for this post.
PDF Files
10 Comments
Welcome to the discussion of the Transparency Initiative. For the next few months, we would like to get your views on how AAPOR can advance the cause of methodological openness in public opinion and survey research. By the time of our annual meeting in May, I would like to have a plan basically in place, in time for the increased publication of polls leading up to the 2010 elections.
Here are elements of the project as I see them, each of which can benefit from your experience and analysis. First, AAPOR will give its commendation to those survey organizations who disclose and archive at the Roper
Center all of the elements required by our minimal disclosure standards at the time of a poll’s release to the public. Second, AAPOR will give special recognition to those organizations who share not only those items of information required by the minimal disclosure standards, but also deposit original data for secondary analysis. Third, AAPOR will offer to certify that the information disclosed by an organization is accurate. Those organizations who opt to have the information they volunteer audited will receive our highest rating.
This system, the details of which need to be specified with your help, will supplement our longstanding practice of censuring organizations who fail to disclose the methodological details of their work as specified in our Code. Rather than rely exclusively on standards complaints to uphold our values, the Transparency Initiative would allow us to publicize our values much more widely and continuously.
There are other aspects of the initiative that will benefit from your thoughts. We need to do a much better job communicating why methodological disclosure is important and how the consumer of disclosed information should interpret it. We also need to make it clear to sponsors of survey research why it is important to work with organizations who are transparent in their work. And we need to make sure that this venture is not a “flash in the pan,” but an integral part of what we do on behalf of the profession.
As is often said about any proposal, “the devil is in the details.” I trust that our discussion here will surface those matters that need to be addressed if we are to pull this off. You can read more about the initiative in the AAPOR Newsletter, out this week, and you will hear much more about it as we get closer to May.
If you sponsor surveys whose results are made available to the public, if you conduct such surveys, if you analyze their data, if you rely on them to understand the state of our society, you have a stake in this project. Please lend your voice to the discussion. All best wishes. Peter Miller
While I applaud the efforts of AAPOR to promote transparency in survey research procedures and methods, I am very concerned about the first of the five bullet points outlining the contemplated elements of the program (i.e., AAPOR’s public recognition of excellence in transparency).
This, in my view, would be a huge blunder, one on the magnitude of Grady Little sticking with Pedro in the 2003 ACLS.
Such recognition would, in essence, amount to official endorsement of survey research service providers. Absence of recognition would carry the inference that a provider is not transparent and does not operate with “excellent” practices. But how would we know this is or is not true?
The reality is that self-certification assurance, as has been proposed on the NexGen group communication posted on LinkedIn, does not do it because anyone could simply “certify” they meet the standards. And I don’t think AAPOR wants to get into the business of playing policeman.
At the same time, other than this point, I generally support this initiative and the remaining four elements outlined in the initiative’s announcement.
Thanks very much for your comment. I would be very interested in what others think about the idea of certifying methodological transparency. It is certainly a major departure from past practice, and it does present a number of nettlesome issues. I do intend, however, that consumers of survey data would examine our list of ‘transparent’ organizations and see if it contains ones in whose data they are interested. If the list does not contain a particular organization, I hope that consumers will inquire of that entity why they have not joined the initiative and agreed to make methodological information routinely available through the mechanism I hope we will set up. AAPOR might not be a policeman directly, but it would provide a system for the community to advocate for more transparency.
ASA, CASRO, ESOMAR, NCPP, WAPOR are listed as collaborators. Why not the American Evaluation Association as well? Their professionals use survey research methods with profound impacts on policy.
Thanks for this important addition. I did not mean to exclude AEA. If others think of additions, please do not hesitate to post.
I’ve often wondered why should anyone trust a poll? Where is the transparency? For goodness sake, what if some closet Tea Partier is working for Gallup taking polls? And the numbers don’t look socialist enough for his liking or his clients liking.
For starters, pollsters need to include who paid for the poll.
Secondly, do we really need to track the daily rise or fall in job approval for the President? For example, Obama use to be in BOLD print at the top of the Gallup website back when he was at 46%. I noticed as his numbers started to rise the daily tracking of him got smaller and smaller. Plus the daily media attention got less and less.
Third, show me the papers. Show me everything that was asked, who asked it, how it was asked, why did you ask it, who paid for it, how much did they pay, show me everything. Not just the most damaging stuff or the most helpful stuff.
Fourth, I’d like to know the financial part of places like Gallup. How much does the CEO get paid, how does he get paid and who pays them. I’d also like to see some IRS statements. Only babies are not biased. Adults are biased when it comes to politics and religion. And you can’t tell me people can’t be bought. We all have a price.
How about ARF as a collaborator on this initiative? Many commercial survey organizations use them as a partner or resource.
I think this is a great initiative. Lots of details to sort out, but remember, AAPOR could not only do a favor to American public debate, but be a model for polling around the world (we do polling in the South Caucasus, and would probably adopt your transparency standards, to set an example to others).
After a couple of experiences in finding pollsters who were worse than just sloppy, I have a take opposite to Brill’s. Think of various federal regulatory agencies. Industries (e.g. meat-packing) that need to maintain public confidence after some horrendous scandals use more than paper self-certification schemes. They have real paid inspectors, at least on a spot-check basis. I think AAPOR (or some spin-off) needs to have a few inspectors who can spot-check the actual accuracy of the “pollster’s” claims. That means people showing up at alleged calling centers on very short notice. Yes, this would slightly raise the cost of approved polls, just as it does with meat inspections, but it would provide some real assurance to the consumers.
I’m speaking on behalf of local residents as a whole, and as a concerned citizen. Why do these survey results only pertain to Organizations and Governmental Agencies. Speaking as a public citizen, I know that recent research was next to impossible for me to obtain on certain health related issues in my community. I hope your initiative is successful, I will then applaud your efforts.
One Trackback/Pingback
[...] November 24, 2009 Two efforts dealing with transparency caught my eye recently – one from AAPOR (the American Association for Public Opinion Research) and the other from Don Bruzzone (of Bruzzone Research) and Jack Bookbinder (at Kaiser Permanente). In just the last few weeks AAPOR announced its Transparency Initiative program which is intended to “place the value of openness at the center of our profession, and to encourage and make it as easy as possible for survey firms to be transparent about their research methods.” Peter Miller, president of AAPOR, emphasizes that this program is a “work in progress” and encourages researchers to voice their comments on the initiative at the online journal Survey Practice. [...]