A note on sampling error:
In almost any poll or survey, the error due sample size is the least likely cause of measurment error. News stories almost always report the margin-of-error due to the sample size only because it smallest and most easily measured source of error.
If you flipped a coin 100 times, you have a 95% of the number of heads coming out somewhere between 40 and 60. Flip a coin 400 times and you have a 95% chance between 45% and 55% percent of the tosses being heads . Flip the coin 50,000 times and heads will fall between 49.6% and 50.4% of the time.
A percentage estimate based on a sample of 100 will tend to have an error rate of 10%; a sample of 400 and error rate of 5%. There is a simple formula for sampling error (below) and a more complicated one that produces the same result. In general, every time you quadruple the sample, the error is cut in half.
sample
size (N)100
400
900
1600
6400sampling
error10%
5.0
3.3
2.5
1.25The following more-or-less randomly selected news stories
"A CBS News poll in May found that only 24 percent of those surveyed had a favorable view of the bureau [the FBI], down from 43 percent just one year earlier. (The poll, conducted by telephone, collected views of 1,063 Americans and had a margin of error of plus or minus 3 percentage points.)" New York Times, September 9, 2001, Section 4; Page 3
"Eighty-two percent of adults surveyed in the CBS 2/Sun-Times poll said binge drinking was either very serious or somewhat serious. Of the 500 adults surveyed, 48 percent believed binge drinking to be very serious and 34 percent thought it to be somewhat serious. . ....The CBS 2/Sun-Times survey was conducted by Survey USA and has a margin of error of plus or minus 4.5 percentage points." Chicago Sun-Times, August 30, 2001, p. 30
"Sixty-nine per cent of respondents recalled hearing or reading about space activities that involved Canada. The poll report also found 81 per cent feel it is important Canada continue to have a space program.... The telephone survey of 1,600 Canadians over the age of 18 was done May 49 in eight cities where the bus panels appeared. It has a margin of error of plus or minus 2.4 percentage points, 19 times out of 20." The Ottawa Citizen, August 26, 2001, p. A5
"About 33 percent of born-again Christians have ended their marriages, compared to 34 percent for those who have not embraced Jesus, said researchers.....
The survey is based on telephone interviews with 7,043 adults nationwide between January 2000 and this July. The study has a margin of error of plus or minus 2 percentage points." The Atlanta Journal and Constitution August 18, 2001, p. 1B"In the latest Star-Ledger/Eagelton-Rutgers Poll, released on Sunday, Mr. Schluter was supported by 4 percent of voters. The poll of 660 registered voters found Mr. McGreevey, the Democrat, at 48 percent, 19 points ahead of Mr. Schundler, the Republican, at 29 percent. Nineteen percent were undecided. The margin of sampling error was plus or minus 5 percentage points." The New York Times August 28, 2001, p. B5
"Overall, about half of Canadians in the poll say the right number of immigrants are coming into the country and that immigration has a positive effect on Canadian communities. Only 16 per cent view it as a negative impact while one-third said it had no impact at all...
The telephone survey of 2,000 Canadians was conducted between March 20 and March 30. It has a margin of error of plus or minus 2.1 percentage points, 19 times out of 20." The Ottawa Citizen August 17, 2001 p. A6"Seven in 10 blacks expressed confidence that Simpson did not kill the pair -- and an identical proportion of whites said that regardless of the verdict, he probably did....
A total of 312 randomly selected blacks and 315 randomly selected whites were interviewed Wednesday through Friday. Margin of sampling error for the overall results is plus or minus 4 percentage points and plus or minus 6 percentage points for the white and black subsamples." Chicago Sun-Times October 8, 1995, p. 29"Harris Interactive said 81% of those surveyed, including those who hadn't previously been familiar with the concept, approved of genetic testing....
This poll was conducted by telephone between May 15 - 21 among a nationwide cross section of 1,013 adults. Figures for age, sex, race, education, number of adults and number of voice/telephone lines in the household were weighted where necessary to align with population proportions. Harris Interactive estimates the results have a statistical precision of plus or minus three percentage points compared with results if the entire adult population had been polled with complete accuracy." Wall Street Journal 6/6/02The following table summarizes the sample size and sampling error reported in these stories compared the the sampling error given by the simple sampling error formula (The New york Times has the largest descrepency between the reported and the actual sampling error).
number interviewed type of persons interviewed sampling error reported actual
sampling error312 blacks 6.0 5.7 315 whites 6.0 5.6 500 adults 4.5 4.5 660 registered voters 5.0 3.9 1013 adults 3.0 3.1 1063 Americans 3.0 3.1 1600 Canadians 2.4 2.5 2000 Canadians 2.1 2.2 7043 adults nationwide 2.0 1.2 Note however, that all of the sampling errors reported and calculated in the above table are based on a critical assumpltion that is in all likelihood known to be false. The critical assumption being that the persons interviewed are in fact a random sample of the all the persons indicated, i.e., "blacks', "whites", "Canadians", "Americans", "adults", "registered voters" and "adults nationwide". In fact, these are more or less random surveys of of blacks, whites, Candians, adults, et. cetera, who chose chose to be interviewed and who chose to answer the pollsters' specific questions. For the poll to be an accurate measure of the entire population of these groups, we have to assume that those how did not respond to the poll would have responded the same way as those who did.
In general a poll's response rate is a far better indicator of the reliability of the poll than the sampling error measure that is reported. Unfortunately, newspapers and polling organizations generally refuse to publicly divulge the response rate for their polls. In recent years, the response rates to most polls have fallen dramatically.
The monthly Current Population Survey enjoys a response rate of 94 60 96 percent.
Almost every newspaper reports poll results with the same standard disclaimer "..a margin of error of X percentage points"
Here's a test. Do a search of Lexis\Nexis newspaper stories for the past six months on the word "poll" and "margin of error". I found 595 articles (on June 6, 2002). Now do a search of "poll" and "response rate". I found 1 for the past six months and 3 over the past 10 years (one of which was a letter to the editor complaining about
Reference:
Don Van Natta, Jr. "Ideas & Trends: Silent Majorities; Polling's 'Dirty Secret': No Response," New York Times November 21, 1999, Section 4;1,1