If you’re a journalist who reads academic research, you’ve likely seen the term “standard deviation” many times. If you’re not sure what it means or how to explain it to audiences, keep reading because we’re going to break it down for you.

Here are four key things you need to know:

1. The standard deviation of a dataset is a number that indicates how much variation there is within the data.

When researchers analyze quantitative data such as birth rates, temperature readings and student test scores, they typically calculate the standard deviation of the data to gauge how close or far apart the data are. A higher standard deviation means the data are more spread out. The lower the standard deviation, the more closely data cluster around the average value of the data.

Deborah J. Rumsey, a statistics professor at The Ohio State University, points out in her book Statistics for Dummies that the measure provides critical context.

“Without it, you’re getting only part of the story about the data,” she writes. “Statisticians like to tell the story about the man who had one foot in a bucket of ice water and the other foot in a bucket of boiling water. He said on average he felt just great! But think about the variability in the two temperatures for each of his feet. Closer to home, the average house price, for example, tells you nothing about the range of house prices you may encounter when house-hunting. The average salary may not fully represent what’s really going on in your company, if the salaries are extremely spread out.”

2.  Scientists can use standard deviation to make predictions, investigate trends and answer other key research questions.

The standard deviation of a dataset plays a limited role in many academic studies. Scientists might only note standard deviation values in a table or list or mention them within the body of an academic article.

Sometimes, however, researchers rely heavily on the measure to help them answer questions central to their studies. For example:

• Researchers can make predictions about the weather, voter behavior, tax revenue, healthcare usage and a host of other things based partly on the standard deviation of data gathered over time.
• Equities researchers typically use the standard deviation of stock prices to measure market volatility, with a high standard deviation indicating high volatility.
• Researchers examining student test scores can use the standard deviation to determine whether most students perform at or close to the average or whether test scores are all over the place. The measure also allows researchers to estimate the proportion of students who need more help mastering the material.

Here’s a brief explanation of how to calculate standard deviation.

3. In some studies, scientists report their findings in terms of standard deviations instead of a unit of measurement such as inches or pounds.

When datasets have data points with different units, scientists often need to standardize, or rescale, the data before they can draw comparisons and look for relationships. For instance, scientists might want to examine the relationship between orange juice consumption, measured in ounces, and flu vaccination rates, measured as the number of vaccines administered each month per 100,000 U.S. residents.

The process of standardizing data includes dividing each numerical data point by the standard deviation of the dataset. Doing this changes the units of measurement. Instead of expressing findings using common units such as ounces, inches and pounds, they must be reported in terms of standard deviations.

Hypothetically, scientists looking at orange juice consumption and flu vaccination rates could conclude that a one standard deviation increase in juice consumption is associated with a one standard deviation reduction in vaccination rates.

While standardizing datasets can make them easier for researchers to work with, Brian Healy, an associate professor of neurology at Harvard Medical School, notes many people might have difficulty understanding the results. He urges journalists to read these papers closely.

“The problem is, unless you look really closely in the paper, you’ll have no idea what a one standard deviation means,” says Healy, who’s also the lead biostatistician for the Partners Multiple Sclerosis Center at Brigham and Women’s Hospital in Boston.

“Do understand the units that results are being shown in,” he adds. “If there is a number reported, you want to make sure you understand how to interpret the number and you can’t understand how to interpret the number without knowing the units.”

4. Scientists can use standard deviation to help them confirm whether a data point they consider an outlier actually is an outlier.

Outliers are extremely high or low values that can complicate statistical analyses and skew results. Many researchers will remove or alter outliers caused by error — for example, an error in collecting or entering data.  

When you look at a graph of all the data in a dataset, some data points appear to be outliers because they differ so much from the others. Since the standard deviation of a dataset takes into account how far away individual values are from the average, scientists often use it to gauge whether an unusual data point is an outlier. This method works well for datasets that follow the pattern of a symmetrical, bell-shaped curve in which the majority of data converge near the center of the bell, where the average value is located.

After calculating the standard deviation for that dataset, it’s easy to spot outliers. A general rule of thumb for data that follows a bell-shaped curve is that approximately 99.7% of the data will be within three standard deviations of the average. Data outside this boundary are usually deemed outliers.

Although the standard deviation of a dataset is affected by outliers, journalists should not assume a large standard deviation indicates data quality problems. As Rumsey writes in Statistics for Dummies, “a large standard deviation isn’t necessarily a bad thing; it just reflects a large amount of variation in the group that is being studied.”

The Journalist’s Resource would like to thank Troy Quast, a professor of health economics at the University of South Florida’s College of Public Health, and Brian Healy, an associate professor of neurology at Harvard Medical School, for their help creating this tip sheet.

Check out our other tip sheets on research-related terms such as statistical significance, peer review and margin of error.

The post What’s standard deviation? 4 things journalists need to know appeared first on The Journalist's Resource.

It’s easy to misunderstand and misuse one of the most common — and important — terms in academic research: statistical significance. We created this tip sheet to help journalists avoid some of the most common errors, which even trained researchers make sometimes.

When scholars analyze data, they look for patterns and relationships between and among the variables they’re studying. For example, they might look at data on playground accidents to figure out whether children with certain characteristics are more likely than others to suffer serious injuries. A high-quality statistical analysis will include separate calculations that researchers use to determine statistical significance, a form of evidence that indicates how consistent the data are with a research hypothesis.

Statistical significance is a highly technical, nuanced concept, but journalists covering research should have a basic understanding of what it represents. Health researchers Steven Tenny and Ibrahim Abdelgawad frame statistical significance like this: “In science, researchers can never prove any statement as there are infinite alternatives as to why the outcome may have occurred. They can only try to disprove a specific hypothesis.”

Researchers try to disprove what’s called the null hypothesis, which is “typically the inverse statement of the hypothesis,” Tenny and Abdelgawad write. Statistical significance indicates how inconsistent the data being examined are with the null hypothesis.

If researchers studying playground accidents hypothesize that children under 5 years old suffer more serious injuries than older kids, the null hypothesis could be there is no relationship between a child’s age and playground injuries. If a statistical analysis uncovers a relationship between the two variables and researchers determine that relationship to be statistically significant, the data are not consistent with the null hypothesis.

To be clear, statistical significance is evidence used to decide whether to reject or fail to reject the null hypothesis. Getting a statistically significant result doesn’t prove anything.

Here are some other things journalists should know about statistical significance before reporting on academic research:

1. In academic research, significant ≠ important.

Sometimes, journalists mistakenly assume that research findings described as “significant” are important or noteworthy — newsworthy. That’s typically not correct. To reiterate, when researchers call a result “statistically significant,” or simply “significant,” they’re indicating how consistent the data are with their research hypothesis.

It’s worth noting that a finding can be statistically significant but have little or no clinical or practical significance. Let’s say researchers conclude that a new drug drastically reduces tooth pain, but only for a few minutes. Or that students who complete an expensive tutoring program earn higher scores on the SAT college-entrance exam — but only two more points, on average. Although these findings might be significant in a mathematical sense, they’re not very meaningful in the real world.

2. Researchers can manipulate the process for gauging statistical significance.

Researchers use sophisticated software to analyze data. For each pattern or relationship detected in the data — for instance, one variable increases as another decreases — the software calculates what’s known as a probability value, or p-value.

P-values range from 0 to 1. If a p-value falls under a certain threshold, researchers deem the pattern or relationship statistically significant. If the p-value is greater than the cutoff, that pattern or relationship is not statistically significant. That’s why researchers hope for low p-values.

Generally speaking, p-values smaller than 0.05 are considered statistically significant.

“P-values are the gatekeepers of statistical significance,” science writer Regina Nuzzo, who’s also a statistics professor at Gallaudet University in Washington D.C., writes in her tip sheet, “Tips for Communicating Statistical Significance.”

She adds, “What’s most important to keep in mind? That we use p-values to alert us to surprising data results, not to give a final answer on anything.”

Journalists should understand that p-values are not the probability that the hypothesis is true. P-values also do not reflect the probability that the relationships in the data being studied are the result of chance. The American Statistical Association warns against repeating these and other errors in its “Statement on Statistical Significance and P-Values.”

And p-values can be manipulated. One form of manipulation is p-hacking, when a researcher “persistently analyzes the data, in different ways, until a statistically significant outcome is obtained,” explains psychiatrist Chittaranjan Andrade, a senior professor at the National Institute of Mental Health and Neurosciences in India, in a 2021 paper in The Journal of Clinical Psychiatry.

He adds that “the analysis stops either when a significant result is obtained or when the researcher runs out of options.”

P-hacking includes:

• Halting a study or experiment to examine the data and then deciding whether to gather more.
• Collecting data after a study or experiment is finished, with the goal of changing the result.
• Putting off decisions that could influence calculations, such as whether to include outliers, until after the data has been analyzed.

As a real-world example, many news outlets reported on problems found in studies by Cornell University researcher Brian Wansink, who announced his retirement shortly after JAMA, the flagship journal of the American Medical Association, and two affiliated journals retracted six of his papers in 2018.

Stephanie Lee, a science reporter at BuzzFeed News, described emails between Wansink and his collaborators at the Cornell Food and Brand Lab showing they “discussed and even joked about exhaustively mining datasets for impressive-looking results.”

3. Researchers face intense pressure to produce statistically significant results.

Researchers build their careers largely on how often their work is published and the prestige of the academic journals that publish it. “‘Publish or perish’ is tattooed on the mind of every academic,” Ione Fine, a psychology professor at the University of Washington, and Alicia Shen, a doctoral student there, write in a March 2018 article in The Conversation. “Like it or loathe it, publishing in high-profile journals is the fast track to positions in prestigious universities with illustrious colleagues and lavish resources, celebrated awards and plentiful grant funding.”

Because academic journals often prioritize research with statistically significant results, researchers often focus their efforts in that direction. Multiple studies suggest journals are more likely to publish papers featuring statistically significant findings.

For example, a paper published in Science in 2014 finds “a strong relationship between the results of a study and whether it was published.” Of the 221 papers examined, about half were published. Only 20% of studies without statistically significant results were published.

The authors learned that most studies without statistically significant findings weren’t even written up, sometimes because researchers, predicting their results would not be published, abandoned their work.

“When researchers fail to find a statistically significant result, it’s often treated as exactly that — a failure,” science writer Jon Brock writes in a 2019 article for Nature Index. “Non-significant results are difficult to publish in scientific journals and, as a result, researchers often choose not to submit them for publication.”

4. Many people — even researchers — make errors when trying to explain statistical significance to a lay audience.

“With its many technicalities, significance testing is not inherently ready for public consumption,” Jeffrey Spence and David Stanley, associate professors of psychology at the University of Guelph in Canada, write in the journal Frontiers in Psychology.“Properly understanding technically correct definitions is challenging even for trained researchers, as it is well documented that statistical significance is frequently misunderstood and misinterpreted by researchers who rely on it.”

Spence and Stanley point out three common misinterpretations, which journalists should look out for and avoid. Statistical significance, they note, does not mean:

• “There is a low probability that the result was due to chance.”
• “There is less than a 5% chance that the null hypothesis is true.”
• “There is a 95% chance of finding the same result in a replication.”

Spence and Stanley offer two suggestions for describing statistical significance. Although both are concise, many journalists (or their editors) might consider them too vague to use in news stories.

If all study results are significant, Spence and Stanley suggest writing either:

• “All of the results were statistically significant (indicating that the true effects may not be zero).”
• “All of the results were statistically significant (which suggests that there is reason to doubt that the true effects are zero).”

5. The academic community has debated for years whether to stop checking for and reporting statistical significance.

Scholars for decades have written about the problems associated with determining and reporting statistical significance. In 2019, the academic journal Nature published a letter, signed by more than 800 researchers and other professionals from fields that rely on statistical modelling, that called “for the entire concept of statistical significance to be abandoned.”

The same year, The American Statistician, a journal of the American Statistical Association, published “Statistical Inference in the 21st Century: A World Beyond p < 0.05” — a special edition featuring 43 papers dedicated to the issue. Many propose alternatives to using p-values and designated thresholds to test for statistical significance.

“As we venture down this path, we will begin to see fewer false alarms, fewer overlooked discoveries, and the development of more customized statistical strategies,” three researchers write in an editorial that appears on the front page of the issue. “Researchers will be free to communicate all their findings in all their glorious uncertainty, knowing their work is to be judged by the quality and effective communication of their science, and not by their p-values.

John Ioannidis, a Stanford Medicine professor and vice president of the Association of American Physicians, has argued against ditching the process. P-values and statistical significance can provide valuable information when used and interpreted correctly, he writes in a 2019 letter published in JAMA. He acknowledges improvements are needed — for example, better and “less gameable filters” for gauging significance. He also notes “the statistical numeracy of the scientific workforce requires improvement.”

Professors Deborah Mayo of Virginia Tech and David Hand of Imperial College London assert that “recent recommendations to replace, abandon, or retire statistical significance undermine a central function of statistics in science.” Researchers need, instead, to call out misuse and avoid it, they write in their May 2022 paper, “Statistical Significance and Its Critics: Practicing Damaging Science, or Damaging Scientific Practice?”

“The fact that a tool can be misunderstood and misused is not a sufficient justification for discarding that tool,” they write.

Need more help interpreting research? Check out the “Know Your Research” section of our website. We provide tips and explainers on topics such as the peer-review process, covering scientific consensus and avoiding mistakes in news headlines about health and medical research.

The Journalist’s Resource would like to thank Ivan Oransky, who teaches medical journalism at New York University’s Carter Journalism Institute and is co-founder of Retraction Watch, and Regina Nuzzo, a science journalist and statistics professor at Gallaudet University, for reviewing this tip sheet and offering helpful feedback.

The post 5 things journalists need to know about statistical significance appeared first on The Journalist's Resource.

Journalists can bolster public trust in science if they present science as a process of exploration, scrutiny and self-correction instead of focusing on novel or sensational research and controversial retractions of published papers, a new study suggests.

Prior research has found that news stories about science in the U.S. tend to follow a few common themes, including characterizing science as being “broken” or “in crisis” and drawing attention to research retractions based on scientific misconduct, plagiarism or ethics violations.

The new study builds upon that research, finding that negative coverage of science without adequate context can erode public trust in scientists and induce negative beliefs about them. The study, “The Effects of Media Narratives About Failures and Discoveries in Science on Beliefs About and Support for Science,” was published this month in the journal Public Understanding of Science.

Journalists need to change how they report on and frame scientific mistakes, according to the authors, Yotam Ophir, an assistant professor of communication at the University at Buffalo, State University of New York, and Kathleen Hall Jamieson, a communication professor and director of the University of Pennsylvania’s Annenberg Public Policy Center.

Ophir says news outlets usually fail to recognize the role retractions play in advancing scientific knowledge. Discovering that findings from a peer-reviewed paper no longer hold true or that results cannot be replicated by other researchers is part of science’s self-correction process.

Those important details are largely missing from news stories, he adds.

“Sure, talk about scientific failures — we want [journalists] to talk about scientific failures,” Ophir says. “The public should know when scientific studies are successfully replicated and when they are not. But we also want journalists to remind their readers that that’s how science works.”

Jamieson says news coverage should emphasize that knowledge is provisional — research builds upon research over time.

“When something goes wrong in science and something gets retracted, the journalist has a choice about how to frame that retraction,” Jamieson explains. “Those instances in which something really important was found and turned out to be an error — they are building blocks. They become the foundation for the knowledge move.”

How they did the research

To gauge how news media accounts influence public perceptions of science, Ophir and Jamieson randomly assigned 4,497 U.S. adults to read one of five types of news articles. The articles were fabricated but based on real news coverage of science. Marketing research company Research Now recruited participants, who closely resembled the U.S. population in terms of age, gender, education and geographic region of residence.

Researchers assigned participants to four treatment conditions and one control condition, created for comparison purposes. People assigned to the control group read an article about baseball, chosen because the article’s structure and length were similar to the science stories but on a topic unrelated to science.

Participants assigned to one of the treatment groups read stories about science built around narratives common in actual news coverage:

• Discovery: This type of coverage, the most prevalent of the four, “features terms such as ‘advance,’ ‘path-breaking,’ and ‘breakthrough’ and tells the story of scientists who have advanced knowledge through a finding cast as new and important, as a discovery,” Ophir and Jamieson write in their paper. “These stories rarely acknowledge dead ends or false starts, and often fail to emphasize the need for additional ongoing research.”
• The counterfeit quest: These news stories focus on a scientist or group of scientists “whose journey to ‘discovery’ and a resulting finding have been found wanting and purged from the scholarly record through retraction.”
• Science is broken or in crisis: This narrative “concentrates not on individual scientists but on broader and more systemic problems in a specific scientific discipline or in science writ large.” It draws attention to “a problem that science as an institution or collective community has ignored or downplayed.”
• Problem explored: These stories spotlight “scientists exploring and hence potentially remedying one of the problems focal to the crisis or broken narrative.”

After reading the articles, participants answered questions aimed at measuring their trust in scientists and their beliefs in areas such as whether science has benefited the U.S. and whether funding for science should be increased or reduced.

For example, people were asked to respond to the following prompts with a number ranging from 1 to 5, with 1 representing “rarely” and 5 representing “often:”

• “When a study is flawed, the scientists involved in it catch and correct the mistake prior to its publication.”
• “When fraud occurs in scientific research, how often do you think it is caught?”
• “When scientists make mistakes in their research, how often do you think other scientists catch it”?

People assigned to read stories featuring the discovery theme expressed the strongest level of trust in scientists. Those who read stories indicating science is in crisis had the lowest.

Ophir and Jamieson also learned that people who expressed higher levels of trust were more likely to believe science is self-correcting — meaning that scholars continually uncover new ideas and evidence and build upon and correct older ones.  

None of the story types had a statistically significant effect on opinions about whether science funding should rise or fall.

A potential solution

Ophir and Jamieson assert that news coverage featuring a problem-explored narrative can help the public understand that research findings are subject to ongoing scrutiny. The problem-explored narrative could “yield more positive beliefs and attitudes about science and scientists, by better communicating scientific norms of continuing exploration, scrutiny, and skepticism,” they write in their paper.

The authors note that better science communication doesn’t depend solely on journalists, however.

Scientists must make changes, too.

News coverage of science, they write, “is the product of a negotiation between scientists and journalists, both of which may be incentivized to prioritize more sensational, novel stories [at] the expense of the somewhat pedestrian, yet crucial, topic of self-correction.”

Ophir and Jamieson add that “scientists and those who communicate about their findings need to develop narratives that reflect the nature of scientific inquiry and its norms and practices as well as the practices it uses to detect and correct error as well as fraud.”

We obtained this image from the Flickr account of Paul VanDerWerf. It is being used under a Creative Commons license. No changes were made.

The post By changing their framing of scientific failures and discoveries, journalists can bolster trust in science: New research appeared first on The Journalist's Resource.

If you’re a journalist, you might feel more comfortable with words than numbers. If you’re reading this, you might also be interested in research, which, more often than not, involves math — usually statistics. One of the more important statistical concepts used in interpreting research is effect size,  a measure of the strength of an association between two variables — say, an intervention to encourage exercise and the study outcome of blood pressure reduction.

Knowing the effect size will help you gauge whether a study is worth covering. It also will help you explain to audiences what the study’s findings mean.  To make effect size as easy to understand as possible, we spoke with Regina Nuzzo, who is the senior advisor for statistics communication and media innovation at the American Statistical Association as well as a freelance science writer and professor of statistics at Gallaudet University.

Researchers can be an invaluable resource in interpreting their study’s key findings. Start by simply asking about effect size: “I think just talking about effect size is a huge win and great for journalists to do,” Nuzzo says. “We [journalists] have not been trained to do that, and some researchers make it hard to do that, and many articles and press releases don’t do it, so you might have to work for it. But it’s so important and rewarding.”

Nuzzo explains that the term effect size can be misleading: it doesn’t actually tell you anything about cause and effect. And effect size alone can’t tell you whether findings are important or newsworthy.

For more on what effect size is, and isn’t, read Nuzzo’s five tips on understanding and interpreting effect size.

Tip #1: Look for the different terms researchers use to describe effect sizes.

“Most of the time in an article… they’re not going to put a big highlight at the top saying, ‘effect size here and here,’” Nuzzo says. “You have to go hunting.”

With that in mind, here are terms that signal “effect size here!”

Some you might recognize: correlation, odds ratio, risk ratio (a.k.a. relative risk), hazard ratio, mean (average) difference.

Some you might be less familiar with: Cohen’s d, Eta-squared, Cramer’s V, R-squared.

Nuzzo offers the following guidance on interpreting the more common types of effect sizes you’ll encounter:

Risk ratio: This is a ratio of the probability of a certain outcome happening in two different groups. For example, suppose a study looked at the incidence of heart attacks in night shift-work nurses compared with nurses who work regular day shifts. To get the risk ratio (RR), which tells you the effect size of night shifts on heart attacks, you take the probability that a night-shift nurse had a heart attack and divide it by the probability a day-shift nurse had a heart attack.

RR = probability of outcome in group A / probability of outcome in group B

“Since it’s a ratio, we can have three different possibilities,” Nuzzo adds. “It can be equal to one, bigger than one, or smaller than one. But it can never be negative.”

• If the RR is equal to 1, that means the risk of a heart attack is the same in both groups
• If the RR is greater than 1, that means the risk of a heart attack is greater in night-shift workers than day workers
• If the RR is less than 1, that means the heart attack risk is lower in night-shift workers than day workers.

It’s not too difficult to translate a risk ratio into statistics you can use.

• If the risk ratio is greater than 1: then the difference between the risk ratio and 1 (Subtract 1 from RR) represents the how much higher the risk of an outcome is for group A compared with B.
  • For example: RR = 1.5 → 1.5 – 1 = 0.5 → The risk of heart attack is 50% higher in night-shift workers than in regular day-shift workers.
• If the risk ratio is less than 1, then the difference between the risk ratio and 1 (Subtract 1 from RR) represents how much lower the risk of an outcome is for group A compared with B
  • For example: RR = 0.75 → 1 – 0.75 = 0.25 → The heart attack risk is 25% lower in night-shift workers than in day workers
• You can also flip the risk ratio — just divide 1 by the risk ratio. So a risk ratio of 0.75 for night workers versus day workers is equivalent to a risk ratio of 1.33 (= 1/0.75) for day workers versus night workers. That would mean that the heart attack risk is 33% higher for day workers than night workers.

Nuzzo adds that it’s helpful to mention absolute risk along with relative risk. If the absolute risk of a certain outcome occurring is very low, that can help contextualize the reduction or increase you see in terms of relative risk.

Odds ratio: This is the same as the risk ratio, with a slight difference in how the probability of an outcome is measured. It uses odds, which is a ratio of probabilities; think of a coin toss — the odds of getting heads is 1:1, or a 50% chance of happening. If something has a 25% chance of happening, the odds are 1:3.

You interpret an odds ratio the same way you interpret a risk ratio. An odds ratio of 1.5 means the odds of the outcome in group A happening are one and a half times the odds of the outcome happening in group B.

Hazard ratio: A hazard ratio (HR) is an annual risk of death (or some other outcome, e.g., cancer recurrence, heart attack) over a specific period, Nuzzo explains. The period of time being studied is important, because everyone has a 100% chance of dying at some point in their lives.

Here’s how you’d translate the following hypothetical example into plain language: If you’re looking at a study analyzing daily meat consumption and risk of death over a 20-year time frame, and the hazard ratio is 1.13 for people who eat red meat every day compared with vegetarians, that means that meat eaters have a 13% increased yearly risk of death over the 20-year study period compared with vegetarians. (We got to this percentage the same way we did for risk ratios.)

But what does a 13% increased yearly risk of death over 20 years really mean? Here’s how to calculate the probability a person in the daily meat-eating group will die before a person in the vegetarian group: HR / (1 + HR)

So in this case, you’d do the following: 1.13/(1 + 1.13) = 0.53

That means there’s a 53% chance that a person who eats red meat every day will die before someone who doesn’t eat red meat at all.

As a quick comparison, you can calculate the probability as though eating red meat had no effect (HR = 1). 1/(1+1) = 0.5. In this case, the chances the meat lover will die before the meat-abstainer is 50% — essentially a heads-or-tails flip.

Tip #2: Put effect size into context.

There are guidelines for what statisticians consider small, medium and large effect sizes. For an effect size called Cohen’s d, for example, the threshold for small is a 0.2, medium is a 0.5, and large is a 0.8.)

But what do small, medium and large really mean in terms of effect size? We might have different frames of reference that we use to interpret these terms. Luckily, statisticians have come up with ways to translate Cohen’s d into what’s called “common language effect size,” as well as effects we can visualize or understand more intuitively. Suppose, for example, that a researcher measures the difference in verbal fluency between teenage boys and teenage girls in a certain neighborhood, and she gets a Cohen’s d of 0.9, which is considered a large effect. You can look up this effect size in a table to find that it translates to a 74% chance that any randomly chosen teenage girl in that neighborhood would be more verbally fluent than any randomly chosen teenage boy in the neighborhood.

Tip #3: Don’t assume effect size indicates causality.

Put simply, effect size cannot prove causation between two variables — that one caused the other to change in some way. It’s just a measure of the strength of the relationship between two things. In general, the larger the effect size, the stronger the relationship. But effect size alone can’t tell you if there’s a causal link between the variables being studied. For example, let’s say a study found that the correlation between leafy vegetable intake and improved sleep quality in children has a large effect size. That doesn’t mean leafy greens cause much better sleep. It just indicates that children who ate a lot of leafy greens had much higher sleep quality than those whose diets were low in greens.

Tip #4: Don’t confuse effect size with statistical significance.

If a result is found to be statistically significant, it’s unlikely to be a chance occurrence. Statistical significance is often understood in terms of p-values. We explained it as follows in an earlier tip sheet:

“P-values quantify the consistency of the collected data as compared to a default model which assumes no relationship between the variables in question. If the data is consistent with the default model, a high p-value will result. A low p-value indicates that the data provides strong evidence against the default model. In this case, researchers can accept their alternative, or experimental, model. A result with a p-value of less than 0.05 often triggers this.

Lower p-values can indicate statistically significant results, but they don’t provide definitive proof of the veracity of a finding. P-values are not infallible — they cannot indicate whether seemingly statistically significant findings are actually the result of chance or systematic errors. Further, the results of a study with a low p-value might not be generalizable to a broader population.”

Just because a result has a small p-value — indicating it’s probably not due purely to chance — does not mean there is a strong relationship between the variables being studied. The effect size reflects the magnitude of the finding.

“You can have something that’s really statistically significant — a tiny p-value — but it also has a tiny effect size. Then you can say, ‘so what?’” Nuzzo says. “It’s really important to look at that effect size — and researchers don’t always brag about it, because sometimes it’s really small. They’d rather say, ‘Oh, but look, my p-value is really good,’ and avoid the fact that, okay, it’s a ‘who cares’ effect size.”

For example, a study might find that the average marital happiness in people who meet on a dating app is higher than the average marital happiness in those who meet in a bar, with a p-value of 0.001, which indicates the finding is statistically significant. But look closer and you might find that the average difference in happiness between the two groups is only 0.2 points on a seven-point scale. This small effect size might not have practical importance.

Tip #5: Remember that effect size is not the sole indictor of a study’s importance or newsworthiness.

The things you should be looking for besides the effect size are:

• How the effect size compares to others in a particular field. In educational or psychological studies, small effect sizes might be the norm, because of the difficulties associated with trying to measure or change behavior. On the other hand, randomized trials in medicine commonly have bigger effect sizes, because drugs tested in tightly controlled settings can have large effects. “It’s important to look at it in the context of that field,” Nuzzo says. “I think journalists can definitely push researchers and say, ‘Okay, what are other effect sizes in this field, in this particular area?’” She also suggests asking the researchers questions that bring the findings closer to the individual-level, including: “What percent of the sample did this treatment actually work for? What percent did it not help at all? What percent got worse?”
• Whether the result is unexpected. “If it goes against everything that we know about theory or past experience, it might be telling us something really cool and really unexpected, so it’s fine if it’s a small effect size,” Nuzzo adds.
• If a study focuses on an intervention, the cost of the intervention might be noteworthy. “Maybe it’s a small effect, but this is a super cheap and simple and easy intervention, so it’s worth writing about,” she says.

For more guidance on reporting on academic research, check out our tips for deciding whether a medical study is newsworthy.

The post Statistics for journalists: Understanding what effect size means appeared first on The Journalist's Resource.

Media scholar John Wihbey, an assistant professor at Northeastern University, co-authored a recent study that found a clear link between the Twitter accounts journalists follow and the partisanship of their published work. In his new book, The Social Fact: News and Knowledge in a Networked World, Wihbey examines the issue in greater depth, raising questions about how journalists use social media to report the news. The book also calls attention to a challenge many newsrooms face — journalists’ struggle to gain competence in reporting about numbers and research.

Wihbey’s book is based partly on data he obtained from us here at Journalist’s Resource. Each year, we survey journalists and journalism faculty and ask questions such as: How important is it for journalists to interpret statistics from sources? On your own, how well are you equipped to do statistical analysis? What are some of the barriers to your using academic research more often in your journalism?  (We’ll be sharing some of the results of the 2019 survey in an upcoming article on JR.)

Wihbey, a former assistant director of JR who continues to help with our annual survey, analyzed the responses we received to identify patterns and problems.

His overall advice, which he explains in a piece he wrote for The Conversation: Reporters should spend more time studying math than they do studying Twitter.

We reached out to Wihbey  via email to ask more about his book and what he thinks journalists can learn from his research. Below are our eight questions and his answers, which offer insights on rebuilding public trust, cultivating diverse sources and developing statistical literacy.

In case you’d like a peek at the book, Wihbey gave us permission to share Chapter 7, “Journalism’s New Approach to Knowledge,” which draws on data from our surveys.

———————-

Journalist’s Resource: What are the top three things you’d like journalists to take away from your book?

John Wihbey: First, I’d like to think they could come away with a deeper sense of how information networks typically behave, how they open up opportunities for both reporting and listening but how they also have structural biases and readily surface misinformation and false trends. The book really tries to dig into the network science literature and pull together some useful material. Second, the book emphasizes the degree to which knowledge and subject matter depth is key to successful and valuable reporting. I try to make a strong case, which echoes the mantra of Journalist’s Resource itself, that we need to take journalism to a new level in this regard. Finally, the book tries to situate our historical moment in the broader sweep of media history. I think it can be empowering to see the changes in the media business as part of a long history of disruption, both culturally and technologically.

JR: In the book, you explain how you examined journalists’ Twitter networks and found a link between the partisanship of the accounts that a journalist follows on Twitter and the partisanship of his or her work. Can you talk more about how journalists are or could be influenced by whom they’re interacting with on Twitter and whether they should be intentionally following and interacting with people holding diverse opinions on a range of issues? 

Wihbey: The section of the book to which you are referring came out of a study we did that was published in New Media & Society. I should note that we did not design the research to look for causation. We merely noted a correlation, in an attempt to show the dangers that may be present in terms of partisan sorting and a potential through-line from news media to social media and back. Spending a lot of time within social networks certainly may color an individual journalist’s view of what is true and interesting. As journalists, we have to be careful to recognize the limits and constraints of our environments. Something may be trending in the online world that we construct (and that is algorithmically influenced), but it may not be as important as many other stories that are unfolding in other spaces. I do think that ensuring an intellectually diverse set of sources is crucial. We need to be more aware of how digital spaces construct our sense of reality.

JR: If news outlets have a left- or right-leaning partisan slant and the partisanship of journalists’ Twitter networks tends to show in their work, then can members of the public really rely on the news media to provide the facts they need to know in a clear and unbiased way?

Wihbey: I would contend that there is no real monolith that constitutes “the news media.” There are thousands of individual journalists operating within diverse institutions. There may be general tendencies and trends that characterize some journalists, but they have fierce codes of independence. The profession is full of brilliant and dedicated people. We should have every expectation that as individuals, journalists are striving toward producing clear and unbiased work. And I think we need to renew the call for greater depth and independence as polarization accelerates within society.

JR: In your book, you draw on data that Journalist’s Resource has been collecting through surveys for several years. Can you talk about why you decided to use this information and how it contributed to your findings?

Wihbey: We launched the first survey in 2015, and we’ve done a bunch of surveys since then. It’s completely fascinating work, and it’s been an honor to help oversee these surveys. We’ve asked a lot of questions about how journalists and journalism educators prepare for their work, how they see the world, how they execute their jobs, and what their aspirations for the future are. It’s a rich dataset from a large community of people involved in media. I’ve drawn on it because it paints a nuanced and textured picture. The survey datasets contributed to the book in an overall sense insofar as they provided me with a reasonably accurate, though evolving, picture of what journalism is currently and where it needs to go.

JR: Is it reasonable to expect most or all reporters to be able to interpret statistics and do their own statistical analyses?

Wihbey: Probably not. Obviously, much depends on one’s particular beat and job needs. In general, I think being able to interpret statistics from sources is crucial. As a baseline, statistical literacy is very helpful in terms of being able to vet claims and detect misinformation. Likewise, having basic data literacy — the ability to, for instance, manipulate data in rows and columns — is a corollary skill that I think is necessary, as well. That said, I’m not sure every journalist should be able to perform statistical analysis, or run a regression analysis or whatnot. However, it’s not a bad thing to at least understand the underlying concepts involved in statistics, whether it’s sampling, [statistical] significance, probability or causation. It can sharpen our thinking and make us less susceptible to slips in logic and framing.

JR: You’ve said you see your book as sort of a Part 2 to the book, Informing the News, by Harvard Kennedy School Professor Tom Patterson, who helped found Journalist’s Resource and serves as its principal investigator. How does your book build on Tom Patterson’s book?

Wihbey: I should hesitate to suggest my work is even remotely in the same category as Professor Patterson’s! Still, I consider his work hugely influential on mine. He and I had many conversations about how my research might take up his challenge and idea of knowledge-based journalism and see how it might apply in the context of social media and a world of increasingly democratized publishing.

There is a big movement in journalism toward greater audience engagement and social listening. But I think it’s in need of richer thinking and theoretical grounding; we need a greater sense of purpose other than just getting clicks and likes, or just getting people talking. I wanted to articulate how the move toward knowledge is the essential ingredient for engaged journalism. In the book, I say that knowledge-driven journalism fuels a virtuous cycle in the online world: journalists must be able to grasp and articulate the stakes for citizens in issues. Increasingly, for citizens to engage in issues that have complexity, they need a level of understanding to organize. The more engaged citizens are, the more they will be informed, as interest in issues is an incentive to learn. By knowing more, journalists are therefore more likely to engage the public.

In one respect, our work around knowledge-based journalism has to grapple with the old Lippmann-Dewey debate, which for nearly a century has framed up an enduring divide between top-down expertise and gatekeeping ([Walter] Lippmann) and bottom-up grassroots learning and agenda-setting ([John] Dewey). Tom Patterson and I talked a lot about how to, in a sense, resolve this debate by emphasizing the application of knowledge within networked practice – knowledge by the journalist as the key to public engagement. I’m in Tom’s debt for the good suggestions in this regard. I think the social web opens creative possibilities that were not available to Lippmann and Dewey. I hope my book provides a somewhat convincing account of how we can be both smarter as journalists and more engaged with the public.

JR: To whom would your book be most useful: individual journalists or newsroom managers?

Wihbey: I would hope that both might take something away that is useful or thought-provoking. For individual journalists, I think the discussion can help provide greater critical reflection on practice. For editors and news leaders, I hope it might clarify the mission of good journalism in a networked environment, where journalists are publishing material alongside millions of other publishers and voices on social platforms. In this regard, I think my central idea of trying to foster mutual “networks of recognition” is useful in terms of clarifying the civic function of journalism in a networked age.

JR: If you could offer journalists one tip for helping them rebuild public trust, what would it be?

Wihbey: I believe the current movement in journalism toward greater transparency regarding [reporting] methods is vital. But increasingly I’ve come to believe that it’s important for journalists to find ways to highlight uncertainty — what is not known, leaving open the possibility for revision, iteration, and admission of error. I think the public would respond positively to a humbler and more careful journalism, a journalism that proceeds a bit more like science and social science. This is not to say that everything needs to be hedged and boringly conveyed. But it needs to allow for the possibility that the first draft of history is just that – a draft; it will likely be radically revised as more information comes along. Trust will not come back instantly. But framing journalism more as an iterative process, and less as a final, deliverable output, may help open up space for the public to engage and help co-create knowledge.

Chapter 7 from The Social Fact: News and Knowledge in a Networked World by John P. Wihbey, © 2019 by John P. Wihbey. Reprinted by arrangement with the MIT Press, Cambridge, MA. www.mitpress.com.

The post Why journalists need statistics, diverse sources: 8 questions for John Wihbey appeared first on The Journalist's Resource.

Here at The Journalist’s Resource, we love research. Early in our careers, however, we as individual journalists didn’t always appreciate the value of research or interpret it correctly. We did not always use the best study to make a point or fact-check a claim. Learn from our mistakes. Here are some things we wish we knew years ago.

1. Academic research is one of the best reporting tools around.

• Reading studies early in the reporting process will give you a good general understanding of an issue. It also will help you ask better questions and understand the answers that sources give you.
• Use it to hold public officials accountable. Oftentimes, policymakers try new things because they assume a certain change will prompt a certain result. (For example, mandating uniforms in public schools to improve student achievement.) A review of the research often will help you gauge whether such a change will or could provide the result a policymaker wants. Research also will tell you what has and has not worked in other locations and under similar circumstances.
• Individual studies often offer ideas for other angles journalists might want to pursue.

2. General Google searches are not the best way to find good research.

• A better source is scholar.google.com, which lets you search for research published in peer-reviewed journals. Other good resources are PubMed, Microsoft Academic, PLOS and the National Bureau of Economic Research.
• Search academic journals for studies. But don’t limit yourself to journals that focus on your specific topic area. For example, if you are writing about criminal justice, you can find quality research in Criminology as well as in Social Problems, PNAS (Proceedings of the National Academy of Sciences) and The Quarterly Journal of Economics.
• At the end of each academic study, authors list the research that is cited within the study. These lists are great resources for additional research on a particular topic.

3. Researchers generally are accessible and like to talk about their work.

• We have found that researchers respond more quickly to email than phone calls. They also may share free copies of their work or tell you how to access them for free.
• If you are confused by a data analysis and don’t have a strong background in statistics or research methods, reach out to someone who does. Many scholars are eager to help journalists describe their research findings correctly.

4. When something is described as “significant,” that doesn’t necessarily mean it is important or noteworthy.

• Scholars often refer to their findings as being “significant” or “statistically significant” to indicate that a relationship they have discovered or a difference they have observed is not likely to be the result of chance or a sampling error. Determining whether something is “significant” or “statistically significant” is based on a mathematical analysis, not an opinion.

5. Correlation does not imply causation.

• Often, studies determine there is a relationship between two or more variables. Just because one variable changes in the presence of a second variable does not mean the second variable caused the change. Never make assumptions about what a study says or does not say. If in doubt, contact the author.

6. Don’t spend much time on the abstract.

• Many people think of the abstract as a summary of the most compelling findings. Oftentimes, this is not the case. The two best places to find information about key findings are 1) the “results” section, which typically is located in the middle of a research article and is where authors explain what they have learned and provide their statistical analyses and 2) the “discussion” or “conclusions” section, which is usually located at the end of the paper and offers a summary of findings as well as a discussion of the real-world implications of the author’s work.

7. Use caution when relying on research from think tanks, private consulting firms and special interest groups.

• The results of research from these organizations are not always independently reviewed prior to publication or distribution, whereas studies published in academic journals generally are.
• As a rule of thumb, journalists should avoid research funded or distributed by organizations with clear biases, including political affiliations.
• Sometimes, academic research does not exist on a certain topic, or there is little of it. Private consulting firms and other organizations often will try to fill that knowledge gap by doing their own research. While some of these organizations provide quality research, it’s important to give the information additional scrutiny.

8. The best research doesn’t always come from Harvard and Stanford.

• Scholars from prestigious universities such as Harvard and Stanford are considered among the best in their fields. But that doesn’t mean every research article and report they write is the best on a given topic. High-quality research comes from scholars in a variety of settings, including public universities and non-profit institutes.

9. The peer-review process does not guarantee quality research.

• Keep in mind that some peer-review processes are more rigorous than others and some academic journals are more selective than others. Publication in a top-tier journal is no guarantee that a piece of research is high quality. But it’s safe to assume research articles that are published in top-tier journals have been reviewed and given a stamp of approval by multiple top scholars.

10. Knowing statistics and research methods helps a lot.

• Many journalists shy away from math and science. But even a basic knowledge of statistics and research methods will help you understand the studies you’re reading and distinguish a good research study from a questionable one.

The post 10 things we wish we’d known earlier about research: Tips from The Journalist’s Resource appeared first on The Journalist's Resource.