question one link question two link question three link question four link question five link question six link question seven link question eight link
 
 

Criteria Used by Guarino et al. for Acceptance of Studies for Review
Elaboration of Quality Criteria
Specific Quality Criteria for Quantitative Research
Specific Quality Criteria for Qualitative Research
Search Strategy and Results

Criteria Used by Guarino et al. for Acceptance of Studies for Review

In the Methodology section of their review, the RAND researchers provided an elaborate discussion of the criteria they employed for accepting or rejecting articles. What follows in this chapter is distilled from the RAND discussion.

The RAND literature review included all studies that met the following four general criteria:

  1. Relevance
  2. Scholarship
  3. Empirical nature
  4. Quality

A study was determined to be relevant if it illuminated issues related to the recruitment and retention of teachers in the United States. The researchers limited studies to those of teacher labor markets in the United States and to those published between 1980 and 2003. Due to limited time and resources, the researchers did not review the large body of research focusing exclusively on the recruitment and retention of special education and vocational education teachers.

A study was determined to be scholarly if it was published in peer-reviewed journals or by organizations with well-established peer-review processes, such as the National Center for Education Statistics, the Educational Testing Service, the Urban Institute and the RAND Corporation, among others. The researchers also included books, book chapters and monographs that offered empirical evidence and analysis. They excluded publications by any organizations that are not research institutions with a well-established peer-review process. The researchers also excluded working papers because (a) they are not peer reviewed, (b) it was not possible to systematically search for them, and (c) their findings are subject to change. Newspaper and magazine articles also were excluded.

A study was considered empirical if it offered evidence — quantitative or qualitative, or both — for conclusions, rather than simply opinion, theory or principles. The researchers excluded simple program descriptions that were not analytical or evaluative, as well as publications that offered only opinions, theory or principles without offering new or original evidence to support conclusions. Thus, they also excluded literature reviews and publications that only cited research performed by others.

A study was deemed to be of sufficiently strong quality if it met the generally accepted standards of rigor for quality in empirical research discussed in more detail below.

Although the preponderance of empirical studies the researchers found on the subject of teacher recruitment and retention were statistical in nature, they included both quantitative and qualitative research in their literature search and applied a similar baseline standard of rigor to each type of research. Quantitative methods are the research tools of choice when the phenomena being studied are well-described by large quantities of numerical or categorical data. This is the case for issues of teacher recruitment and retention because they so heavily involve the realities of the labor market. Quantitative methods can yield findings that can be generalized to larger populations, and they also can tease out more general facts or trends from those embedded in particular contexts.

A qualitative approach generally yields useful information when a quantitative approach is not possible because the phenomena being studied are too few to permit a large sample size, too impressionistic or meaning-dependent to quantify readily, or too new or too little studies to allow researchers to formulate an appropriate set of hypotheses or analytic approach. A qualitative approach also may be useful in providing a context for the interpretation of phenomena of interest. Thus, qualitative research is often useful in hypothesis generation and essential when concepts cannot be reliably captured via quantitative methods.

In the end, quantitative and qualitative research are complementary. Qualitative research often precedes and serves as a foundation for quantitative research, as it provides insight into meaningful distinctions that can then be used to categorize, quantify and collect data on a large scale. Quantitative research is grounded in qualitative pieces of information that can be described and gathered in quantifiable units in sufficient quantity to permit the use of statistical tools. This means that a qualitative understanding of the phenomena being studied is needed before quantitative data can be gathered.

[ back to top ]

Elaboration of Quality Criteria

The researchers included a study in their review if its research design and analytic strategy were appropriate to the topic under study, its methodology was applied in a careful manner, its focus was relevant to the research questions posed in the ECS report, and its interpretation was well-supported.

[ back to top ]

Specific Quality Criteria for Quantitative Research

The researchers' assessment of the quality of a quantitative study involved four considerations:

  1. The sample had to be adequate in size to support credible inferences; it had to reflect accurately the characteristics of the larger population that was sampled; it had to be selected randomly; it had to be complete and be adjusted, if necessary, to account for nonresponse of various groups in the larger population; and it had to draw on appropriate units of analysis: it is problematic, for example, if aggregated data (e.g., average state teacher salaries) are used to draw conclusions about smaller units of analysis (e.g., average teacher salaries in a specific district). Also, the researchers of the individual studies had to address any issues of sample incompleteness or attrition.
  2. The variables used in a study had to be measured reliably and with a high degree of validity.
  3. The statistical models used had to be efficient and free from significant bias, and they had to appropriately represent the phenomena under study. The researchers of the individual studies needed to demonstrate an awareness of the potential for any bias in their models and employ measures to attempt to correct for it.
  4. The interpretation placed on the findings had to be warranted and be consistent with the results. While an unjustified interpretation did not automatically disqualify a study, it was an important consideration and was noted in any critique of the RAND study.

[ back to top ]

Specific Quality Criteria for Qualitative Research

The researchers' assessment of a qualitative study involved three considerations:

  1. The methodology employed had to be appropriate to the research question. If the subject of investigation was the impact of particular program on teacher's attitudes about the teaching profession, for example, a case study or ethnography might well be the best approach to provide useful data.
  2. The study offered sufficient evidence to support its conclusions.
  3. The study provided an interpretation and analysis of the data that was of interest to other researchers in the field or led to the formulation of relevant hypotheses.

[ back to top ]

Search Strategy and Results

The RAND researchers conducted electronic database searches of Education Abstracts, Social Science Abstracts, Econlit, ERIC and JSTOR. They undertook table-of-contents searches on widely recognized journals, including the American Educational Research Journal, Educational Evaluation and Policy Analysis, the Journal of Policy Analysis and Management, the Economics of Education Review, and the Journal of Education Finance. And they searched the National Center for Education Statistics publications index, the Educational Testing Service publications index, the RAND report index, the Urban Institute report index, the American Institutes for Research report index and the book index of the University of California libraries.

Although peer review was an important criteria for selection, the fact that (a) the searches sometimes yielded inconsistent indications about whether a journal was peer reviewed or not and (b) the researchers also chose to study non-peer-reviewed publications of high enough quality and relevance to justify inclusion eventually led the researchers to re-run all searches without the peer-reviewed screen and then to assess the quality of the publications on a case-by-case basis.

In addition, the RAND researchers asked a number of scholars for their suggestions of relevant works to include in this study.

These searches yielded an initial total of 4,773 candidates for review. Of these, 2,989 were on topics insufficiently relevant to this project; 1,475 were either non-empirical in nature or located in journals not widely regarded as quality scholarly publications even though they might be peer reviewed; and 213 were of interest, but were either reviews of other work, early publications superseded by later work that was included, not quite on target regarding the specific research questions or rejected under the quality criteria as described in the previous section. This left 91 items in the final RAND review. [One article was subsequently discovered to be non-empirical by the author of the ECS report.]

* For additional insight into the methodological issues involved in the preceding discussion, see the sections titled "How Do I Know if the Research Is Trustworthy?" and "A Research Typology" (Appendix A) in A Policymaker's Primer on Education Research.

[ back to top ]

 

© 2005 Education Commission of the States