I’m often involved in teaching students about critical appraisal of educational / social science research. I’m not convinced by arguments that we should judge research only by the criteria that apply within a particular perspective or paradigm. Notwithstanding my prior post, based on Schatzki’s arguments, about why ontology is important and how it changes the game in terms of judging research, I do believe that there are some dimensions of research that can be subject to a broader-based critique.
This refers to a framework presented by Martyn Hammersley in chapter 2 of: Reading ethnographic research: a critical guide, published by Longman (eg. 1998 2nd edition).
I think Hammersley’s framework (originally written with a focus on ethnographic research) provides a sound basis for precisely such an approach. The content does not overly prescribe what good research is, nor does it replace rules, conventions and quality criteria associated with particular perspectives or approaches.
But, as I say in the podcast, I’ve yet to come across a piece of social science research where asking probing questions about the focus, empirical context / case, methods, claims [and their links to the case] and conclusions [and their links to the focus] have not been useful as a means for assessing research quality.
Reader-listeners will detect my strong attachment to the idea of ‘evidence’ in educational / social science research. I doubt everyone shares this, and I’d be surprised if everyone agrees with the views expressed in the podcast.
Part of my motivation for the podcast was a reaction to constructions of Hammersley (and others like him) as rather old-fashioned empiricists. I hope the podcast shows how a concern for evidence, quality of evidence, and relationships between claims and evidence does not automatically position one as a naive realist who’s never heard of the crisis of representation etc.
I conclude the podcast by arguing that the aesthetic dimension of research (something I’ve blogged about elsewhere, too), is something that is not excluded from Hammersley’s framework, but isn’t given the emphasis that it might deserve. I suggest that incorporating aesthetics into assessments of research quality (inspired by Silvia Gherardi, Antonio Strati and others), follows through on the original spirit of Hammersley’s framework. Hammersley is very careful in setting up a position that rejects a doctrine of immaculate perception, and has an explicit role for modes of writing, relationships between researchers and participants, and varying degrees of insight, inference and so on. I simply suggest that highlighting these complements and enriches a focus on claims and evidence.
In summary: a lot can be achieved in terms of critical appraisal of educational or social science research by thinking about:
1. The Focus (wider topic), its articulation (scope, boundaries), importance, relevance
2. The Case(s) studied [not that all research is a case study] – the spatially and temporally limited aspect of the wider focus that is the actual subject of empirical research
3. Methods – including processes through which data are generated [and not collected: see Pat Thomson’s blog for more on this], relationships between researchers and participants, analytic techniques etc.
4. Claims made about the case – different kinds of claim and the different kinds of evidence that would warrant them
5. Conclusions drawn – not letting go of evidence completely, but saying something about the wider focus, moving beyond the specific case (eg. via theoretical inference, empirical generalisation).
If we concern ourselves with these questions, and relationships between focus, case, methods, claims, and conclusions, while keeping a close eye on evidence (whatever that may look like), we can’t go far wrong. And if we are sensitive to aesthetic dimensions when we do this, too, so much the better!