How to read the margin of error in a survey

Imagine two surveys asking the same question about library opening hours. The first estimates that 44% of eligible residents want longer opening hours. The second estimates 48%. That is a four-point gap between estimates. Before treating it as a change in opinion, look at how much uncertainty the surveys attach to their results.

The range comes from drawing a sample

The UK Office for National Statistics explains why another sample from the same population could produce a different number. A confidence interval gives a range using the estimated sampling uncertainty. In a table that reports a margin of error of five percentage points, 44% plus or minus five points spans 39% to 49%.

The confidence level describes the method. Under the assumptions behind a 95% procedure, if we could repeatedly draw samples and construct intervals the same way, about 95% of those intervals would cover the true population value. Each new sample can put the interval in a different place.

Put the uncertainty around the difference

In the invented example, imagine the second estimate also has a five-point margin. Its interval spans 43% to 53%. Those ranges overlap. An analyst who wants to assess the four-point change should use a method for comparing the estimates. The U.S. Census Bureau, for instance, publishes guidance and a tool for testing differences in American Community Survey estimates.

The comparison depends on how the figures were produced. Estimates that share observations can have related errors. Comparing two years in a rolling multiyear survey, for example, can require particular care. Use the guidance for that source and look for a published uncertainty measure or test for the difference itself.

Check what a margin can cover

Sampling uncertainty is one issue. Question wording, coverage gaps, nonresponse and reporting mistakes can also affect an estimate. The ONS distinguishes these from sampling error. Check that the question, eligible population, collection mode and time period are comparable before concentrating on the interval.

Finally, decide how large a change would matter in practice. A precise estimate of a tiny shift may deserve less attention than a larger shift with uncertainty that calls for more data. In a headline, keep the point estimates, intervals and comparison method together so readers can see the basis for the claim. If you can find only the point estimates, report that limitation alongside the numerical difference.