Literary Digest and Predicting the 1936 Presidential Elections: An Example Of How Not To Conduct A Poll

In statistics, we are taught that samples should be representative of the overall population, but what happens if they are not? Would just amping up a larger sample make for a better, more representative prediction? Emphasized by Mr. Anderson’s sarcastic threats of immediately failing his students if they ever thought otherwise, and as notably shown in some opinion polls on the 1936 general election―which saw Republican Kansas Governor Alfred Landon or the incumbent president Franklin D. Roosevelt as its candidates―in no way does relying on sampling more individuals to reduce bias work out as well as a well-conducted sample.

In 1936, Literary Digest, one of the most trusted magazines at the time (which had a history of accurately predicting presidential election outcomes), had a plan to tap into the American electorate. Their method of sampling was ambitious: based on all the sources of lists of people in the United States they can get their hands on (telephone books, club lists, and others), they chose a sample of ten million people, sent them each a mock ballot, and waited for returns with marked their responses. They even went as far as to claim, “When the last figure has been totted and checked, if past experience is a criterion, the country will know to within a fraction of 1 percent of the actual popular vote of forty million [voters].”

Unfortunately, their poll had major flaws and their optimism for this project ran short. Firstly, selection bias plagued the poll, as the sources usually came from upper- or middle-class voters and leaving out the lower class, who would seem to be the most active, considering 1936 was still in though nearing the end of the Great Depression. Secondly, out of the ten million people who were on the mailing list, only 2.4 million of them actually responded, attributing to nonresponse bias, which skews the results of the poll towards the most opinionated individuals of the population and leaving out the opinion of those less active on the issue. There’s no feasible way to get the rest of the 7.6 million people who didn’t respond. In the end, the magazine predicted that 57 percent of the sample would go to Landon and Roosevelt would get 43 percent of it. When it came to the actual results, Roosevelt won 61 percent of the vote and Landon won 38 percent.

The bottom line: though larger, representative samples could be a plus for a representative sample, an improperly chosen large sample underperforms as opposed to a well-chosen small sample, due to the sheer skewness in the prediction created by selection and nonresponse bias. Although a perfect sample does not exist, pollsters can more closely match their findings with the populous’ opinions if they took account of factors they cannot control, such as the population's accessibility, demographics, and most importantly, passions and opinions about whatever issue considered.

Source: https://www.math.upenn.edu/~deturck/m170/wk4/lecture/case1.html