Module Three:

Interpreting Results

Learning Objective

Learn how to interpret results and findings. 


What You Need to Know

Studies that have experimental design should make clear distinctions between two statistical terms: correlation and causation. 

  • Correlation indicates that two variables have some kind of relationship, whether positive or negative, and therefore may change together.

  • Causation exists when one variable causes a change in another. 

The two can exist at the same time, but correlation does not imply causation. 

Effect size measures the strength of a relationship between two measures on a common scale. It does not indicate whether there was correlation or causation. Effect sizes typically range from -0.2 to 1.2. Note that an effect can be described as “statistically significant,” but that doesn’t make it educationally meaningful. In Visible Learning, researcher John Hattie, determined that an effect size of 0.4 could be used as a marker to represent a year’s growth per year of schooling for a student.

Steps

  • 1. Assess the Quality of the Results and Findings

    If the study is experimental, look at the effect size and find out whether the relationship between the variables is correlational or causal. Determine whether the findings are consistent with previous research.

  • 2. Evaluate the Sample Size

    Check that an adequate/ and appropriate number of subjects were tested to make the results applicable to a larger group and were representative of different target populations.

  • 3. Check the Context

    Note that research conducted in a controlled environment will not necessarily transfer to a real-life classroom of students. Controlled experiments allow researchers to isolate certain aspects of learning and cognition, making results not universally applicable to real-life settings. Check on whether the researcher includes information about the results’ limitations.

  • 4. Beware of Bias

    When interpreting research results, be cognizant of the researcher’s and your own potential biases. Confirmation bias is our own natural preference for information that confirms what we know and the tendency to overlook information that contradicts or challenges our natural preference.

Guiding Questions

  • What are the implications of the results? Are they supported by previous research?

  • Can the results be applied to real-life classroom settings? What evidence is there that the intervention would work in your context? 

  • What are the limitations of this study according to the researchers?

  • Was this an independent study? Are there any potential biases that could have affected the results?

Case Study

Recognizing the high cost of tutoring, the superintendent wants to ensure there is impact. From the study of 9th and 10th graders who received high-dosage math tutoring during the 2013-14 and 2014-15 school years, researchers found that participation increased math test scores by 0.16 standard deviations (SDs) and increased grades in math and non-math courses.

In an article about the study, researchers reported that during the first randomized control trial (RCT) participants’ test scores were roughly double the math growth of an average high schooler over one year, and in the second RCT, those results roughly doubled. Students’ scores increased by six percentage points in the first year and 14 in the second year.

She is confident that students in her district can produce similar results since the sample sizes for the treatment groups in the two RCTs were adequate and representative (over 2,500 9th and 10th graders in two public schools in Chicago in both trials). 

She also is conscious of the possibility of bias. She checks to make sure that the study was conducted by independent researchers as opposed to the developers of the tutoring company.