Return to: How Do I Know If the Research Is Trustworthy?
Who were the participants in the study? How were they selected?
The research report should describe the number of participants in the study, as well as their characteristics. This includes not only the characteristics of persons, but also those of entities such as schools and districts. Look for characteristics that could influence the results such as the following:
- Student characteristics – Grade level, gender, socioeconomic status, ethnicity, language status (e.g., second language learner), prior student achievement
- Teacher (classroom) characteristics – Experience, grade level, class size, subject area, preparation, certification status
- School characteristics – Number of students, teachers, paraprofessionals, administrators and other certified staff; location; grade levels; socioeconomic status; ethnicity of students; mean student achievement data
- District characteristics – Number and grade levels of schools; number of students; number and types of teachers, administrators and other certified staff; location; community characteristics.
The study should describe how the participants were selected for the study
sample. Most researchers do not have the luxury of selecting a
random sample from a
population of participants. An exception is the U.S. Department of Education, which conducts random sampling to collect education survey data.
If the sample is not random, then conclusions about the population based on the sample can be erroneous. Valid conclusions can be made only about the sample of participants in the study.
A related issue is how participants were assigned to the different comparison groups in the study. Without random assignment, selection bias can occur. For example, if a researcher selected teachers to participate in one of two types of professional development based on school location, the results could be influenced by characteristics of the schools rather than the professional development.
Here are some examples of studies with and without random assignment:
Example of a study with random assignment:
A researcher uses an
experimental research design to study whether teacher professional development increases student achievement. Prior to the beginning of the school year, half the 4th-grade teachers in a school district are randomly assigned to receive professional development in reading and the other half are randomly assigned to receive no professional development in reading. At the end of the school year, the achievement gains in reading by the students of the two groups of teachers are compared. It is assumed that because teachers were randomly assigned to the two groups, teacher characteristics that might influence reading achievement favor neither the treatment group nor the control group.
Example of a study without random assignment:
A researcher uses a quasi-experimental research design to study whether teacher professional development increases student achievement. The researcher assigns teachers in School A to the treatment group. For the control group, the researcher finds a school with school and teacher characteristics similar to those of School A (e.g., similar student achievement, similar teacher qualifications). When matching is used, the researcher should report how the groups were matched and the degree to which matching was successful (i.e., the similarity of the matched groups).
Example of a comparative descriptive study:
A researcher conducts a study to determine whether teacher professional development is related to increased student achievement. The researcher examines the achievement gains in reading by students of teachers in two schools. In one school the teachers had participated in professional development in reading, while in another school the teachers had no professional development. This type of comparative descriptive study is called ex post facto because the research started after the fact – that is, after the professional development occurred.
On face value, this descriptive comparative study seems very similar to an experiment. The researchers, however, did not select teachers to participate in the two groups. In addition, the researchers did not implement the treatment (the professional development). While this study might be informative, a conclusion that professional development increased student achievement scores would be invalid. In a descriptive study, due to selection bias and the absence of treatment manipulation, the only conclusion that can be justified is about association, not causation.
Good education research also seeks to limit the impact of
extraneous variables regarding study participants. Extraneous variables are characteristics of participants and aspects of the study that are not intended to influence the results. Look for studies that use random assignment,
matching or
statistical controls, or that keep characteristics constant (e.g., using teachers with the same amount of experience), as ways to control extraneous factors.
