How to tell whether a comparison supports a cause

A gardener notices that plants given a feed are taller. The fed plants also happen to grow in the sunniest corner of the garden. The observation is worth investigating. To learn how much the feed contributed, the gardener needs a comparison that separates it from the effect of light.

Start with the other differences

The Open University uses misleading correlations to show how a third factor can help explain a pattern. In this fictional garden, sunlight is a plausible confounder. It varies with whether the plants were fed and could affect the result. There might also be differences in watering, starting height or the types of plants used.

You can still learn from the first observation. It suggests a question and makes the competing explanations visible. Write down what the groups share and what else varies with the treatment. The list can help decide what needs measuring or controlling in a follow-up study.

Change how the groups are formed

One way to build a stronger comparison is to choose comparable plants, record their starting heights, and give each plant a randomly assigned treatment. In a simple example, number the pots and shuffle identical slips marked feed or no feed, one for each pot. Use the same watering plan, potting mix and measurement schedule for all the plants.

The NIST/SEMATECH engineering statistics handbook describes designs in which the treatment assigned to each experimental unit is determined by chance. Here the unit is a pot. Chance assignment reduces the opportunity for the gardener to give one treatment systematically to plants that were already favoured.

Read the result and its limits together

Even with random assignment, a small group can start with uneven characteristics by chance. A report should tell you how many plants were studied, how growth was measured, and how uncertain the difference was. Measuring change from the starting height is one way to make the outcome clear in this example.

The National Research Council distinguishes experiments, where researchers control assignment, from observational studies. It also discusses how far evidence from one setting can travel. A carefully measured effect for one plant variety under one season’s conditions gives the best information about those conditions. Another variety or site may be worth testing separately.

When you read a study headline, find the part that describes how the groups were formed and what was measured. Then ask whether those groups, measurements and conditions support the claim being made.