700 People Said It. America Has Apparently Spoken.

There is a very specific kind of sentence that arrives in the inbox with impressive regularity: “Seventy-three percent of Americans say…” What follows can be almost anything. Americans are changing how they work. Americans no longer trust a certain technology. Americans secretly want to reorganize their kitchens. Americans would apparently abandon civilization itself if the hotel Wi-Fi were slow enough. 

Somewhere near the bottom of the release, after the headline has already converted a percentage into a national mood, there is often a methodology note explaining that the finding came from 700 or 1,000 or 1,200 respondents surveyed through some panel on behalf of the company whose product happens to sit directly beside the problem the survey has just discovered.

This does not make the survey useless. It does mean we should stop letting the press release decide what the survey proves.

That distinction matters because a lot of interesting reporting starts with imperfect material. Press releases are not peer-reviewed journals, corporate surveys are not the U.S. Census, and sponsored polling is not suddenly transformed into neutral public knowledge because somebody put a percentage sign next to it. But journalism would become remarkably dull if the only things worth discussing were facts produced under laboratory conditions with pristine methodology and no interested party anywhere near the room. The more practical question is whether a survey gives us enough information to begin a conversation, and whether we are disciplined enough not to confuse beginning that conversation with validating every claim wrapped around it.

Sample size is part of that judgment, but it is also one of the easiest details to misunderstand. Seven hundred respondents sounds absurdly small next to a country of more than 300 million people, and emotionally it is difficult not to look at the number and think: That is who we have appointed to speak for everybody? Statistically, however, a well-designed probability sample of that size can produce useful estimates. Under ideal simple-random-sample assumptions, 700 respondents would put a percentage near 50 percent within roughly four percentage points in either direction at the conventional 95 percent confidence level. The population does not need to be interviewed one household at a time for sampling to work.

But that is precisely why “we surveyed 700 people” is not the reassurance it is often presented as. The interesting questions begin after the sample size. How were those people found? Did every person in the population have some meaningful chance of being selected, or did respondents come from an opt-in online panel? Was the sample weighted, and to what? Who did not answer? What did the question actually say? In what order were the questions asked? Were respondents screened for a particular experience before being counted as “Americans”? Are we looking at the full sample or a subgroup that may contain 83 people? Was the fieldwork conducted over three days immediately after a news event that could have changed how people answered?

The American Association for Public Opinion Research treats those details as core disclosure issues, not decorative footnotes. Its guidance for journalists distinguishes probability samples from nonprobability samples, stresses the importance of recruitment and weighting methods, and warns that conventional margins of sampling error do not simply transfer to opt-in samples. It also reminds reporters that larger is not automatically better. A giant pool assembled badly can still be a giant pool assembled badly. Meanwhile, question wording, nonresponse and weighting can introduce uncertainty that the tidy “±3.5%” at the bottom of a chart does not magically erase.

None of this survives particularly well inside the press-release machine. A company commissions research because “512 respondents from a commercial opt-in panel expressed this preference under these conditions” is not a headline. “Americans are demanding a new way to work” is a headline. The transformation from one sentence to the other is where a lot of certainty gets manufactured. A limited observation becomes a trend. A trend becomes a national mood. A national mood becomes an inevitability. Then the company announces that, fortunately, it has just launched the product designed for the future everybody has apparently agreed upon.

There is a temptation to respond by rejecting the entire genre. I do not think that is especially useful either. A corporate survey can still surface a real tension, behavior or emerging vocabulary worth examining. Sometimes the sponsor is asking a self-interested question because the question happens to sit directly on top of a real change. Sometimes 600 people saying something strange is interesting even if we have no intention of announcing that America has spoken. Sometimes the percentage is not the story at all; it is simply the door into the story.

That is the line we keep coming back to in our own reporting. We are not publishing a survey because we have independently certified that 64 percent of the United States now feels a particular way. We are publishing because a company, trade group, research firm or professional organization asked a group of people something, the answers reveal an idea worth examining, and the idea holds up as a useful conversation even after we strip away the theatrical certainty of the release.

The recent Avid research on artificial intelligence and professional editors is a good example. Avid surveyed 120 professional editors. That is not “editors have decided the future of AI,” and it would be ridiculous to treat it that way. But the responses were still useful because they exposed a coherent workplace tension: editors appeared much more interested in AI helping them execute their own creative intentions and giving them more time for storytelling than in handing the creative intention itself to the machine. The sample did not need to establish a universal law of editing for the underlying question to be worth pursuing. It gave us a place to ask something larger: What if the best use of workplace AI is not replacing the meaningful part of a job, but removing the administrative shell that accumulated around it?

That is where a survey becomes valuable without becoming sacred. The percentage does not have to carry the entire article. In fact, it probably should not. It can tell us that a group of respondents encountered a question in a certain way. From there, reporting, context, experience, other evidence and plain old human observation can do the heavier work. If the argument collapses the instant 73 percent becomes 66 percent, there probably was not much of an argument there to begin with.

This also helps separate survey skepticism from survey cynicism. Skepticism asks what the data can reasonably support. Cynicism decides in advance that sponsored research can never tell us anything. The first is journalism. The second can become its own kind of laziness. Companies spend money researching customers, workers and industries because those populations matter to them. Their incentives should be visible in the story, but incentives do not automatically make every observation false. They make the observation something we should handle with the proper grip.

For us, that grip is fairly simple. Attribute the finding. Name the population that was actually surveyed whenever possible. Do not quietly upgrade “respondents” into “Americans” just because the release did. Look for the methodology and note meaningful limitations when they affect the claim. Be especially suspicious of tiny subgroups wearing full-sample percentages like an oversized coat. Distinguish between what the survey found, what the sponsor says it means, and what we think is actually interesting about it. Most importantly, make sure the story still has a reason to exist if the survey is treated as a signal rather than a verdict.

That last test is the most useful one. Suppose the exact percentage is wrong by five points. Suppose a better-designed study later finds that the trend is weaker. Suppose the sponsor framed the question too aggressively. Is there still a real issue underneath it? Are people actually wrestling with the behavior, technology, workplace problem, consumer ritual or cultural shift the survey points toward? Can we connect the finding to something observable beyond the release itself? If yes, we may have a story. If no, we probably have marketing collateral with a chart.

There is also a difference between information that is imperfect and information that falls below the conversation line entirely. A survey with a disclosed sponsor, identifiable sample, intelligible questions and enough methodology to understand what happened can be discussed with appropriate restraint. A mysterious “study” with no accessible methodology, no indication of who was asked, percentages that cannot be reconciled, or claims that bear no relationship to the questions is something else. Journalism does not owe every statistic a rehabilitation project.

The point is not to become terrified of numbers. It is to stop being hypnotized by their formatting. A percentage to one decimal place looks precise even when the underlying measurement is wobbly. A chart can be beautifully rendered while the sample is badly constructed. “National survey” can describe geography without establishing representativeness. And 700 respondents can be either a legitimate statistical sample or 700 people who clicked a button because they had ten minutes and wanted the gift card.

So yes, we will probably keep writing from surveys that arrive through PR Newswire. Some of them will have small samples. Some will be sponsored by companies with an obvious interest in the result. Some will contain claims we would never repeat in our own voice without attribution. That is not an accident in the process. Handling those tensions is the process.

The survey does not get to tell us what America thinks simply because the headline says it does. But it may still tell us what question is worth asking next.

And for a newsroom interested in the strange, practical, structural and very human things hiding underneath the daily flood of corporate announcements, that is often enough to get started.

Methodology note. This essay draws on current guidance from the American Association for Public Opinion Research on survey disclosure, probability and nonprobability samples, weighting and reporting precision, as well as Pew Research Center explanations of sampling error and survey design. The approximate margin-of-error example for n=700 is an illustrative simple-random-sample calculation and does not describe the precision of any specific commercial survey.

SOURCE NOTES

• AAPOR — A Journalist’s Guide to Understanding Polls & Surveys
• AAPOR — Disclosure Standards
• Pew Research Center — Understanding the Margin of Error in Election Polls