Every text included in a systematic review should be appraised using an explicit criteria. In medical science reviewers a hierarchy of evidence:
Ia: systematic review or meta-analysis of randomised controlled trials
Ib: at least one randomised controlled trial
IIa: at least one well-designed controlled study without randomisation
IIb: at least one well-designed quasi-experimental study, such as a cohort study
III: well-designed non-experimental descriptive studies, such as comparative studies, correlation studies, case–control studies and case series
IV: expert committee reports, opinions and/or clinical experience of respected authorities
V: anecdotal / opinion
Many scholars in the social scieces believe it is unfeasible and undesirable for management research to adopt the hierarchy of evidence. All academic fields are different and every study is different. What consitutes evidence and 'good quality research' depends entirely on its appropriateness to question being asked (Boaz and Ashby, 2003). As noted by Briner, Rousseau and Denyer (2009, Academy of Management Perspectives - forthcoming):
'If the question is “what effect does intervention X have on outcome Y?” then a meta-analysis of randomized trials may indeed be the best possible evidence. Similarly, if the question is about tools for personnel selection then meta-analysis of predictive validity studies is likely to be appropriate and may provide relevant evidence for practitioners (e.g., Le, Oh, Shaffer and Schmidt, 2007). For other questions, longitudinal studies or quasi-experiments might be the best evidence available. If, on the other hand, the question is “how do women interpret their role on male dominated boards?” then qualitative data will form the best evidence. If the question is “why or how does goal setting result in higher team performance?” then we need theory as well as evidence from which we can infer processes.'
Therefore, 'good quality' research can be quantitative, qualitative or theoretical. It is possible to create a set of generic quality criteria that are applicable to all study designs. For example,
However, these should be supplemented with some specific appraisal criteria for the particular methods employed in the original studies. We do not believe that it is possible to judge qualitative research using criteria designed to evaluate quantitative research and vice versa.
Quality checklists for different research designs are freely available on the internet, such as those produced by the Critical Appraisal Skills Programme.
Given that different subfields of management will have different perspectives on research quality, it is also important to take this into consideration. You should investigate the guidelines for reviewers in a number of journals in your subfield of management and organisation studies.
Some journals produce comprehensive guidance of appraising certain forms of research. For example, the Journal of Occupational and Organisational Psychology has an extensive set of - criteria for evaluating papers using qualitative research methods. If you want to construct some criteria for evaluating theoretical / conceptual papers you could refer to a journal such as the Academy of Management Review: This journal's Guidelines for Reviewers are provided below:
- Does the paper create, extend or advance management theory in a significant way?
- Is the topic important and interesting? Does the manuscript pass the “so what”test?
- Are the central constructs defined clearly? Are the underlying causal mechanisms behind proposed relationships explained clearly?
- Are underlying assumptions clearly recognized and discussed?
- Does the manuscript contain a well-developed and articulated theoretical framework?
- Do the propositions (if applicable) logically flow from the theory?
- Is relevant literature cited accurately? If relevant literature is missing, can you point the authors toward that literature?
- Does the paper have clear implications for future research?
- Is the paper’s contribution commensurate with its length?
By combining general quality criteria and guidance from key journals in the field it is possible to create a bespoke and practical quality appraisal tool.
The quality of a paper is not only related to methodological issues but also its contribution to the field. Further, some papers fail to report the methods of data collection and data analysis in sufficient detail to conduct an evaluation. Whilst this demonstrates poor scholarship, the paper may actually provide new insights. In such cases you may not wish to exclude papers that are deemed to be low quality, but to evaluate and report on the study and its limitations. If studies are excluded on the basis of quality it is crucial for the reviewer to document and justify the reasons for exclusion.
|
1. e.g. Does the paper address a clearly focused issue?
2.
3.
etc |
|
|
|
Quality criteria for quantitative papers |
|
|
|
1.
2.
3.
etc |
|
|
|
Quality criteria for qualitative papers |
|
|
|
1.
2.
3.
etc |
|
|
|
Quality criteria for conceptual / theoretical papers |
|
|
|
1.
2.
3.
etc |
|
|
|
Does the paper make a contribution to the field? Does it help address the review questions? |
|
|
|
Should the paper be included in the review |
|
|
[To download and use this document - click on the word document here
The text on this page was created by Dr David Denyer, Management Practice Fellow, Advanced Institute of Management Research (AIM), and Senior Lecturer in Organization Studies, Cranfield School of Management, david.denyer@cranfield,ac.uk