Module Two:

Assessing Research Quality

Learning Objective

Understand the characteristics of quality research.

What You Need to Know

Research designs take different forms depending on the research’s purpose. The first step in choosing a design is deciding whether to take a qualitative or quantitative approach. 

A qualitative approach relies on data collected by the researcher from observations, interviews, questionnaires, focus groups, and artifacts. A quantitative approach utilizes observational/descriptive, correlational, experimental, and quasi-experimental designs. 

Quantitative Designs

Below are descriptions of quantitative designs, including randomized control trials, which some deem the “gold standard” for research methodologies in education.

  • Observational / Descriptive Design

    Non-experimental design that focuses on the characteristics of the population or question being studied (the ‘what’ vs. the ‘why). Data collection tools can include surveys, interviews, focus groups, ethnographic and case study methods.

  • Correlational Design

    Focuses on the relationship between two variables, but does not include manipulation or control of either variable by the investigator; therefore, causation cannot be determined when using a correlational design.

  • Experimental Design

    Has one variable that is controlled and one that is manipulated by the investigator to determine the existence of a cause and effect relationship.

  • Quasi-Experimental Design

    Does not rely on random subject assignment but is used when experimental design cannot be used for ethical or practical reasons and researchers still seek to find a cause and effect type relationship. Data collection includes large-scale surveys or panel data of a quantitative nature.

Steps

  • 1. Start with a Reliable Source

    Make sure the research or evidence has been peer-reviewed and/or vetted by a trusted source like an academic journal, clearing house reviewers, or a professional association with a well-developed and strong reputation for providing accurate and reliable information.

  • 2. Consult with a Subject Matter Expert or Researcher

    If possible, connect with a researcher or expert, who can help assess the quality of a study or evidence.

  • 3. Assess Critical Components of the Research or Evidence

    Such as methodology, the sample, outcomes, and replicability.

Guiding Questions

  • What was the research methodology that was used? Was there a control group?

  • Is the sample size adequate, inclusive, and representative? 

  • Was the research submitted for an independent expert review?

Case Study

After looking for research that provides evidence of impact and on effective design principles, the superintendent chooses to look more closely at a recent study of 9th and 10th graders who received high-dosage math tutoring during the 2013-14 and 2014-15 school years. 

To justify the high cost of a high-dosage tutoring program, she wants to ensure the study is high quality. She examines the following components:

  • Methodology:  Researchers conducted two randomized control trials (RCT) of the tutoring model. RCT is considered the ‘gold standard.’

  • Sample:  In the first RCT, there were 2,633 9th and 10th graders from two representative Chicago public schools in the treatment group, and in the second RCT, there were 2,710 9th and 10th graders.

  • Outcomes:  Participation in the tutoring program increased math test scores by 0.16 standard deviations (SDs) and increased grades in math and non-math courses, with results persisting into future years.

  • Replicability:  Researchers replicated the results of the first RCT in a separate RCT with 2,710 students and found even larger math test score impacts and similar grade impacts. 

She concludes that the research design and outcomes are strong. She also finds an article that she can share with her board to help communicate the intervention’s effectiveness.