- April 26, 2002; Brynjulf Stige (
- ):
Do We Need General Criteria for the Evaluation of Qualitative Research Articles, and If We do, How Could Such Criteria be Formulated?
Brynjulf Stige
Acknowledgement
- (1)The idea of starting a discussion in the web Forum of
Nordic Journal of Music Therapy
- (NJMT) grew out of the 4th International Symposium for Qualitative Music Therapy Research, organized by Mechtild Jahn-Langenberg in Sauen, April 2002. I want to thank all participants for these stimulating days, and especially for the members of the discussion group I belonged to (Kenneth Aigen, Gary Ansdell, and Eckhard Weymann) for helping me shaping these initial ideas. I hope that they, and others, will contribute in a discussion about this important topic. Eventually then, after a process of discussion in this Forum (and then in the Fourth Nordic Music Therapy Conference) it may be possible to conclude whether it will be feasible or not for NJMT to operate with general criteria for the evaluation of qualitative research articles.
Introduction
- (2)
Nordic Journal of Music Therapy
- publishes quantitative and qualitative research studies. For quantitative studies there are quite clear criteria for the evaluation of submitted texts: there are some firm methodological rules to follow, there are specific procedures for statistic analysis, and there is a high degree of consensus on what adequate means of reporting the results are. In qualitative research this is different, and there are many good reasons for proposing that it
should
- be different. Qualitative research is about exploring particularity and diversity, and there is a plethora of qualitative research approaches around with rather different answers to basic ontological, epistemological, and methodological questions.
(3)This diversity creates some problems though in the process of evaluating articles, maybe not because there is a lack of criteria for evaluation, but because there is a lack of consensus as to which criteria to use. Most traditions of qualitative research have their own criteria of evaluation, but how could a journal receiving articles from several traditions best deal with this issue? As there is little consensus as to which criteria to use, one may in fact imagine a situation where four sets of criteria operate in the evaluation of one text: the author is writing according to the criteria of his or her tradition of research, the two referees use two other sets of criteria, and the editor a fourth. While ideally one could think that each research project deserves to be evaluated according to the criteria of the tradition it belongs to, this is not always easy to exercise practically. First of all, authors are not always clear about what tradition their research belongs to (or their methodology may be eclectic, using bits and pieces from several traditions.) Second, it is not always realistic to think that a referee should be equally updated and qualified in the latest criteria proposed in any tradition or branch of qualitative research.
(4)Even if these practical problems were possible to overcome completely, which I doubt they are, another more basic problem would remain: If criteria idiosyncratic to each research tradition are used only, one runs into the risk of inhibiting a process where different positions may challenge each other and produce a healthy discussion among qualitative researchers about what the most relevant and important criteria of evaluation are.
What General Criteria Should Not Be, and What They Possibly Could Be
- (5)To me it is obvious that general criteria for the evaluation of qualitative research could not be based in the specific procedures advocated by one or another tradition of qualitative research. While phenomenological researchers may advocate the importance of triangulation and Grounded Theorists request systematic procedures of categorization, to base general criteria of evaluation in such research procedures certainly could alienate a researcher working for instance in a hermeneutic tradition. From this simple example I conclude that general criteria could not be formulated as a rule-book in relation to research procedures. Neither could they be formulated as prescriptive rules as to how to present a study, since many qualitative researchers advocate that not only the research method should be adopted to the object or phenomenon of study, but also the form and format of presentation of the research findings.
(6)As an alternative to criteria based upon research procedures etc., I would like to consider criteria based upon more basic metatheoretical assumptions related to ontology and epistemology, one could call them metacriteria if one likes. Let me – by playing with Wittgenstein’s (1953/1967) notions of language games and family resemblances – use the rules of games such as basket, soccer, and rugby as an analogy. It makes sense to group these games together, as they share certain characteristics. They are all games where two teams compete according to certain rules, manipulating a ball in order to have more points than the other team. It would not make sense for a referee to evaluate a soccer team by the rules of a basketball team though. In the first case you are supposed to move the ball with your feet and you are – within certain limits – allowed to be in physical contact with the players of the other team, while in the other case you are supposed to move the ball with your hands and no physical contact among players is allowed. If rugby is added when making this comparison, we have a game where both kicking and throwing is allowed, as well as physical contact at a much rougher level than what is allowed in soccer. In other words: a referee of a basket or soccer game would create chaos if using the rules of rugby, and vice versa.
(7)Still, at another level, there are some rules that seem to be shared by games such as soccer, basket, and rugby, and which are necessary in order for them to make any sense. Four examples could be: a) there is a set of rules for the particular type of game, and this set is known and accepted by both teams and all players, b) a player plays in one team only, and tries to help his or her own team to win the game, c) you are not allowed to hurt other players or with purpose put their health to risk, d) the referee has the final word when events and rules are disputed.
(8)In using this analogy we could propose that qualitative research represents a family of “games,” with particular rules for each game but also with some “shared rules” for the whole family. I will proceed by proposing a set of “shared rules” or metacriteria that could be relevant for the evaluation of qualitative music therapy research. The proposal is based on the premise that reflexivity is a basic characteristic of qualitative research. With reflexivity in relation to research I mean: The researcher acknowledges that she is not separated from the field she studies, she is herself positioned in it and must therefore reflect upon this position, which includes self-inquiry and examination of the assumptions guiding the research process (Ruud, 1998). I have discussed the concept of reflexivity in relation to music therapy and research in a forthcoming book (Stige, in press/2002), and some of the arguments below paraphrase arguments in this book.
Reflexive Methodology
- (9)Alvesson and Sköldberg (2000) have developed a conceptualization of four elements of reflexive qualitative research methodology that I find relevant to use as a starting point. The conceptualizations are based upon certain ontological and epistemological assumptions. For instance the authors try to avoid the polarized positions of realism with little awareness of the problems of representation of reality and social constructivism where the representations may lose contact with reality. The conception of data as merely reflections of the world is therefore rejected by these authors, and in this respect the
primacy of interpretations
- is accepted. In order to communicate this reservation, these authors therefore use the term
empirical material
- instead of data. Following these basic assumptions it is proposed that the basic principle for qualitative research methodology is
reflexivity
- . There are no fixed rules prescribing what to do and not to do, rather, to be reflexive in relation to research means to reflect upon the preconditions of one’s own activity and upon how one’s own personal and professional involvement influences the interaction with the phenomenon under scrutiny. Reflexivity in the research process, then, is awareness of the need for critical and sensitive interpretations of one’s interpretations of the empirical material. Part of this is the awareness that as researchers we construct ourselves while constructing the objects or phenomena that we study, which then suggests that self-reflexivity and reflexivity converge.
(10)Based upon these assumptions Alvesson and Sköldberg (2000) then propose four elements in reflexive research methodology: 1) Systematics and techniques in research procedures: There must be some kind of logic in the way the researcher interacts with the empirical material, and some kind of systematic approach when collecting it. 2) Clarification of the primacy of interpretation: A reflective researcher acknowledges the primacy of interpretation (that even “raw data” are interpretations), which implies that research cannot be disengaged from either theory or self-reflection. 3) Awareness of the political-ideological character of research: What is explored, and how it is explored, cannot but support or challenge some values and interests in the society. The political and ideological aspects of research must therefore be acknowledged. 4) Reflection in relation to the problem of representation and authority: There are problems of representation and authority connected to any research text, and awareness of rhetoric elements and the relationships between text, author, and world need to be examined.
(11)These four issues have been treated differently by different traditions of qualitative research, and a possibility for learning exists in examining these traditions. Consider for instance the concern about systematics and techniques found in Grounded Theory, empirical phenomenology, and other inductive approaches. Similarly, hermeneutics has much to offer concerning clarification of the primacy of interpretation, while Critical Theory, feminism, and related metatheories have given valuable inputs concerning the political-ideological character of research and knowledge. Postmodern (text-oriented) approaches, admired by some and held in contempt by others, should also have something to offer to all researchers, by sensitizing them to the problems of representation and authority in research texts. Alvesson and Sköldberg (2000) therefore argue that reflexivity in research involves some kind of search for a balance between these four perspectives, and that this is necessary in order to avoid reductionism with focus only on data, interpretation, critique, or language. The researcher thus must search for some kind of interplay between the four levels, and avoid totalization of one of them. These suggestions also have a practical aspect: reflexivity in research means to restrict the time you use for data collection and analysis, to be able to make room for interpretations of the interpretations that you make.
(12)An integration of the four concerns and orientations outlined above – from concerns about the empirical material to concerns about interpretation and critique – Alvesson and Sköldberg (2000) call “reflexive interpretation.” Again, their main message is the need to counteract any one of those orientations to totalize and “take over” the operations in the field. There are several levels of interpretation to deal with, from data-collection and analysis (“low-abstract” interpretations) to critical and self-critical interpretations that focus upon ideology and the selectivity of voices represented in the research text (Alvesson & Sköldberg, 2000, p. 238ff.).
ERIC: metacriteria for evaluation of qualitative music therapy research?
- (13)I believe Alvesson and Sköldberg’s (2000) eclectic approach is relevant for a multifaceted field like music therapy. The four metacriteria that I will propose below are therefore in many respects based upon their contribution. I have rephrased them from being descriptions of elements of reflexive research to being metacriteria for evaluation of qualitative music therapy research. I have also changed the sequence somewhat. This is partly of practical reasons, in order to be able to create an acronym that researchers could remember more easily than abstract principles, and partly to communicate how central the problem of representation is for researchers in a discipline where music is a major element. (It must be noted, though, that the criteria proposed are not set up according to any hierarchy. They are all equally relevant and important.) The acronym I propose is ERIC, composed of the following criteria:
“The researcher should demonstrate awareness of and sensitivity to:
- the relevance and solidity of the Empirical material
- the problem of Representation
- the primacy of Interpretation
- the obligation to produce pertinent Critique.”
(14)(Even though ERIC is a good old Nordic name, I would rather have preferred an acronym that could characterize qualitative research in some way, such as EPIC. I have not been able to produce that without stretching words and arguments though, so ERIC is my proposal at this point. Anybody that can help?)
(15)I will briefly exemplify the issues lurking behind each letter of the acronym, with a few references to some music therapy literature where such issues have been discussed. The list of references is in no way intended to be complete. The references are included just to demonstrate that the criteria proposed here relate to ongoing discussions in the discipline:
E – the relevance and solidity of Empirical material: Empirical materials – for instance musical recordings, field notes from therapy sessions, and interview transcriptions – will be the result of musical and/or verbal interactions with clients or other agents of the setting. While qualitative researchers may not need to have representative data according to the conventions of quantitative research, empirical material should not be arbitrary and fragmented, or colored by the preferences, personality, or values of the researcher with lack of discussion of such issues. A high degree of sensitivity of the researcher-as-instrument is thus asked for (Aigen, 1996).
R – the problem of Representation: This domain is multifaceted. As Ansdell (1999) has argued, problems of language and discourse are at the heart of a discipline with a non-verbal phenomenon such as music as a major element. In addition, there is the more general problem of representation in research, based on the assumption that language does not simply reflect reality. Any representation is a selection, distortion, and addition. Related to this are issues of power, discourse, and rhetoric. What is required of the researcher is of course not a particular stance in the above mentioned debates, but awareness of and relevant discussions of these dilemmas.
I – the primacy of Interpretation: Interpretation starts already with the choice and collection of data. The researcher therefore needs to demonstrate awareness of the hermeneutic problems inherent to all research, such as part-whole relationships when working with the empirical material, and the influence of the researcher’s pre-understanding upon the understanding that is developed. As a general criteria, what is asked for is of course not the use of hermeneutics as a specific research method, but awareness of the general problems of understanding and interpretation that have been discussed in philosophical hermeneutics (Ruud, 1998; Stige, in press/2002).
C – the obligation to produce pertinent Critique: This criteria is based upon the assumption that all research is situated in social and political contexts, and in some ways are influenced by and also influence these contexts. While some researchers – such as participatory action researchers – make the implication that research should contribute to social change and empower participants (Stige, in press/2002), this could hardly work as a general principle for evaluating qualitative music research. A minimum of awareness of this dimension could be requested though, ensuring that researchers at least do what they can to hinder that their research contributes to repression and dis-empowerment.
The DaRMI-principle
- (16)The above criteria could be summarized to
request for reflexivity
- . Based upon Alvesson and Sköldberg’s (2000) discussion of reflexive methodology I have created the acronym ERIC that could denote four domains of major relevance in qualitative music therapy research. These criteria are not meant to “replace” other standards proposed for the field, such as Bruscia’s (1996, 1998) discussion of authenticity and integrity. Rather, what is proposed are some domains in which the researcher must use his authenticity and integrity in order to develop the required reflexivity.
(17)For researchers and for referees evaluating submitted research articles, one question remains: How should reflexivity be demonstrated and documented? To produce thorough discussions of the issues implicit in the ERIC acronym could easily turn every research article into a book. I think there is no general answer to this question. Solutions must be found in relation to each research project. I have one general comment though, and that is that qualitative researchers – to a higher degree than quantitative researchers – need to know the history of ideas quite well. Since there are no fixed methodological rules to play by, qualitative researchers need to position themselves in relation to philosophical and metatheoretical discussions. This has, in fact, clear implications for the content and organization of qualitative research courses in music therapy training programs.
(18)Going back to the concrete question of how to demonstrate and document reflexivity, the issue of how to write becomes essential. My proposal is that at this concrete level, the DaRMI-principle could be helpful. DaRMI is a term I am making up, by spelling the established IMRaD-principle for quantitative research backwards. What I am trying to communicate is that the IMRaD-principle in many ways encapsulates what makes scholarly communication effective (for explication, see NJMT’s Guidelines for authors.) The principle could therefore not be easily dismissed. At the same time, a literal application of the IMRaD-principle, as it has been explicated within the quantitative tradition, would be limiting for many qualitative research projects. For instance, in most qualitative research studies it does not make sense to present the methods used with the purpose of making replication possible. On the other hand, explication of metatheoretical assumptions may need to be developed to a higher level than what is usual in quantitative research articles.
(19)With the acronym DaRMI I then try to communicate that the sections expressed by the IMRaD acronym should be present in most qualitative research articles (Introduction, Methods, Results, and Discussion), but that there could be considerable freedom as to their sequence and as to how to develop each section. The basic story of a research report is, according to the four sections suggested by the IMRaD-principle: “This is what I looked for and why I wanted to find it, This is how I did it, This is what I found, and This is what I think it means.” Is the story-line of qualitative research articles very different? I don’t think so. I do think, however, that some phenomena require flexibility and creativity in writing, so that the plot may be arranged differently. In this respect qualitative researchers sometimes may come close to authors of fiction. In a detective novel, for instance, the author may choose to disclose who the murderer is on page one, only to use the rest of the text to elaborate upon how and why this happened. Similarly I have read good research articles organized in other sequences than I-M-R-D (to rearrange the sequences may sometimes be effective, as in the detective novel, by leading to a closer examination of how and why instead of who and what).
(20)But: if I read a research article with no discussion of results, or with a lot of discussion but with no presentation of results, or with no clarification of method, or with no positioning of the study in relation to existing theory, do I read a good research article? I don’t think so. I think the I, the M, the R, and the D needs to be there, in some way or another, which I then have chosen to call the DaRMI-principle.
Conclusion
- (21)In the literature of qualitative research several sets of criteria for evaluation have been proposed. As there is no consensus, this is part of the problem. Authors may feel that their text is evaluated by the wrong criteria, and journal editors may feel that the evaluation process becomes vague, because of lack of shared criteria. In the long run this is a potential problem for the discipline and profession, as the situation may impede the development of high quality qualitative research. As soccer cannot be refereed by the rules of basketball, a participatory action research project informed by critical theory could not be refereed by the coding rules of Grounded Theory. My proposal is therefore not a set of specific rules, but a set of metacriteria denoting domains with a complexity of issues related to them. These issues are under continuous debate and have for a large part been discussed for centuries. The purpose of having general criteria of evaluation would then not be to have prescriptive rules for how exactly qualitative research should be done. Such a thing would be limiting for the content- and context-sensitivity that characterizes good qualitative research. Instead the purpose of having general criteria would be to develop consensus about a few major domains of which the researcher should demonstrate awareness of and sensitivity to.
(22)The reflexivity implicit in the ERIC acronym needs to be communicated effectively in a research article. I have tried to encapsulate a request for the combination of rigor and flexibility through making up the term “the DaRMI-principle.” The story told by the researcher may be organized in many ways, but we need to know what was looked for and why, how the (re)searching was done, what was found, and what the researcher thinks this means. If reflexivity – as outlined by the ERIC acronym – is integrated in the writing process, I think the result may be a good and interesting research article. This is one of the reasons why qualitative researchers need to know the history of ideas quite well. They need to be able to position their work in relation to existing traditions of thought (which often could be done in a few sentences and with a few references instead of re-inventing the wheel through pages of explications of every basic assumption).
(23)One could of course counter my arguments for ERIC by stating that researchers may disagree on metacriteria based on metatheoretical assumptions as much as they could disagree upon specific prescriptions about procedures. I still think a journal like NJMT have reasons for exploring if such metacriteria could be established, for at least the following reasons:
a) The existence of such criteria would ensure that the author knows the criteria used by the journal, which should be better than a hidden sets of criteria (to imagine no criteria is impossible, unless one imagines a process without evaluation, which is incompatible with the notion of a peer reviewed journal.)
b) When the author knows the criteria, he has the freedom of choosing to accept them, trying to change them (through initiating a debate), or simply choosing another journal. Again this must be preferable to being evaluated by criteria that are not explicit.
c) The fact that there could be disagreement about metacriteria such as those communicated by the acronym ERIC could in itself be fruitful for the discipline, as it could stimulate metatheoretical debate and discussion, which will be helpful for future refinement of theory, research and practice.
References
- Aigen, Kenneth (1995). “Principles of Qualitative Research.” In: Wheeler, Barbara (ed.).
Music Therapy Research. Quantitative and Qualitative Perspectives.
- Phoenixville, PA: Barcelona Publishers.
Alvesson, Mats & Kaj Sköldberg (2000). Reflexive Methodology: New Vistas for Qualitative Research. London: Sage Publications.
Ansdell, Gary (1999). Music Therapy as Discourse & Discipline. A Study of ‘Music Therapist’s Dilemma.’ London: Doctoral thesis, City University, Department of Music.
Bruscia, Kenneth (1996). “Authenticity Issues in Qualitative Music Therapy Research.” In: Langenberg, Mechtild, Kenneth Aigen & Jörg Frommer (eds.). Qualitative Research in Music Therapy. Beginning Dialogues. Gilsum, NH: Barcelona Publishers.
Bruscia, Kenneth (1998). “Standards of Integrity for Qualitative Music Therapy Research.” Journal of Music Therapy, XXXV (3), 176-200.
Ruud, Even (1998). Music Therapy: Improvisation, Communication and Culture. Gilsum, NH: Barcelona Publishers.
Stige, Brynjulf (in press/2002). Culture-Centered Music Therapy. Gilsum, NH: Barcelona Publishers.
Wittgenstein, Ludwig (1953/1967). Philosophical Investigations. Oxford: Blackwell.
- April 30, 2002; Jane Edwards (
- )
Response to Brynjulf Stige’s paper ‘General criteria for the evaluation of qualitative research articles’
- (1) I want to start by saying how much I enjoyed the ‘slip’ in an email advising of this forum where it was described that the criteria for ‘quanlitative’ research were to be discussed. I think this slip is, to some degree representative of the issues that might have led to Brynjulf’s plea for general criteria to be developed (and I think that Wittgenstein might have liked the slip in terms of the question ‘are we talking about what we think we are talking about?’).
(2) A discussion of the question ‘how do we ensure that criteria for assessing qualitative research submissions for publication allow for flexibility and originality in approach but also adhere to standards for quality in communication of the study undertaken?’ is welcomed. We might also think about the question ‘How do we avoid requiring a fixed written representation of all published qualitative research in the way that studies that employ quantitative research method have traditionally required?’ Any discussion of these questions in relation to refereed publication quickly broadens out to include a wider range of ideas and perhaps we see this in Brynjulf’s concern as to the implications of what he is proposing in regard to teaching about qualitative research.
(3) I do think that Brynjulf’s proposal to use ERIC could be a start to refining a process for review of qualitative research submitted for publication. Have people any examples of difficulties that they have encountered as editors and reviewers? Perhaps these could be discussed (carefully with respects to confidentiality) to see whether the approach proposed by Brynjulf would offer an answer or way through the difficulties people have previously experienced (for my own part these are mainly the inappropriate naming of an approach as ‘grounded theory’ or
‘qualitative research’ when writers are simply using rich clinical description in case report form submitted for review or examination).
(4) Although I agree with the authors Brynjulf quotes who state that reflexivity is one key to the undertaking of qualitative research, I believe that other aspects in Brynjulf’s paper are also important. These are that ontological and epistemological issues implicit in the research method chosen should be required to be addressed to some extent in written submission of qualitative research reports for publication (and I am not sure this is the same as the argument that political-ideological awareness is important to be demonstrated). I concur with Althusser’s perspective that the statement ‘I am not ideological’ is an insensible one however also including discussion of beliefs about ‘knowledge’ and ‘knowing’ is perhaps commendable in good qualitative research writing.
(5) The reasons I believe that requiring qualitative research reports to address ontological and epistemological perspectives inherent in the work have been stated eloquently by Peile (1988), a social work researcher, who advised that the choice of research methods in social work research should not be made merely on the basis of epistemological arguments, but “…should also be based on the compatibility of the research method with the researcher’s own preferred paradigmatic assumptions or worldview, and these should be made explicit so that they can be challenged.” (p. 8). I think we could have further interesting debates about broader positions in relation to research and challenge each other about these; not so that we might agree as to whose research voices should be privileged but rather so that we can be challenged to explore more deeply some of the ideas on which our research endeavours are based.
References:
Althusser, L. (1971). Ideology and ideological state apparatuses. In: Lenin and philosophy and other essays. London: New Left Books.
Peile, C. (1988). Research paradigms in social work: From stalemate to creative synthesis. Social Service Review, 62(1), pp. 1-19.
- May 3, 2002; David Aldridge (
- )
The politics of qualitative research criteria: A local solution within an ecosystemic ecology.
- 1)I would like to present here a set of criteria that we will be using to review studies for a systematic review project (see
http://www.musictherapyworld.net
- ). These criteria are for a specific purpose, and that is the point of my brief enclosed comments. The criteria for evaluation will be context specific, as such they will also be politic (see Aldridge,D. & Aldridge,G. A personal construct methodology for validating subjectivity in qualitative research. (in the archive at http://www.musictherapyworld.net). I see evaluative criteria as local and context specific according to the politic of the relationships involved – an ecosystemic paradigm. Trying to generalize is laden with difficulty, and while necessary (perhaps) for a journal, the end result will be that many of us will agree to differ. As editor of a website, my main criterion is “Is a contribution interesting”? Such an approach does not accord to a high-minded science but it gets people talking. Things however are different for the doctoral studies that I accompany on their way. Differing criteria then for differing circumstances, and probably in different places at different times.
2) If I am examining doctoral theses that use a qualitative approach, the assessment criteria are specific to me here at the Chair and to the institution. I would like them to be coherent with other Universities and institutions but this is idealistic, as we all have varying criteria. Like the validity problem in research, eventually the core questions of the argument are “Does this research make sense to me” and “Do I find it legitimate”? The making sense to me is a personal matter and I accept a very broad range of studies, some of which may be incompatible one with another (I also like quantitative research studies for the same reason). However, the legitimation of such studies is based upon a social understanding according to the workgroups within which I find myself. As an academic in the process of conferring an academic degree in a University responsible to a Ministry of Science then there is a chain of responsibilities, even if I am sometimes an advocate of subversiveness in that chain. Eventually, we face the dilemma of responsibility to the candidate in offering advice such that she will gain her doctoral qualification as opposed to her demand for a freely-structured work that would find no resonance with potential examiners. This is not an idle speculation, my own doctoral study had a hard time because in the early 1980’s it was difficult to get a qualitative ethnographic study examined in England (it has since been published and is used by a health care region in England as a practice model). Times change and so do the criteria for legitimacy.
3) Eventually, we want our studies, and those of our students to be accepted into the broader realm of academic life and professional practice, what I refer to as ” reflective practitioner in the community of inquiry” (Aldridge, D. 1996 “Music therapy research and practice in medicine” Jessica Kingsley: London). Regardless of the official criteria, there will be unofficial criteria for acceptance or rejection. Thus my emphasis on the politics of evaluative criteria. I applaud the effort in this journal to find such criteria but guess that eventually you will find your own benchmarks. Making them transparent is important and is probably a qualitative research study in itself, something like “The hidden criteria of validity for the acceptance of qualitative research studies in music therapy journals”. Whatever, the criteria will be known as the Stige criteria and I guess that is the hidden purpose behind suggesting generalizability.
4) Here is the research tool we are using for evaluating research studies in our project. I am open to comments. It will be changed and eventually be prepared as a data mask for entry into a database. Those interested in the project can write to me. They too, are part of the political process of negotiation, just like this commentary. The positive side is that which the European sociologists emphasise as an antidote to the solipsism of radical constructivism; a community of knowledge where we develop criteria together.
5) Finally, I am all for keeping the form of qualitative research open. I guess that eventually I would find a novel acceptable as a research form given that I cooperate with our Faculty of Humanities, and we are also considering “performance”, “installations” and “exhibitions” as a valid element of phenomenological studies here at the Chair of Qualitative Research University Witten Herdecke. Such formats are necessary for doctoral studies in dance therapy and art therapy. We will have to find evaluative criteria. They will be local. We can share them and then they will be glocal, part of a shared community of like minded participants.
6) I fully sympathise with the attempt to find some basis for assessing qualitative research. It is indeed a wide and varied field. The following argument has been for local solutions within a global ecosystemic ecology- that is, glocal. Below the reader will find a pragmatic, albeit limited, evaluative tool for those wishing to cooperate with us. This tool will eventually be available as a database entry template. Inevitably solutions are political and partisan. Brynjulf Stige presents his beloved ERIC as an appropriate acronym, which itself is Nordic and male. As a balance for some of our European cousins, we could have GIACOMO (Guided imagery and Consciousness of Musical Origins), JIMI (Journal of Improvisational Music Indices *), FRITZ (Free Radical Interpretation of Therapeutic Zeitgeist) or RITA (Relative Interpretations of Transcendental Actions). Maybe in the interest of our female colleagues, and using a name that is found in a variety of countries, and one that also has a hidden purpose, we can use ANNIE – Announcing Non-Neutral Interpretation Expectations (methodologically incorrect but realistic). Reflecting Stige’s DaRMI principle, I would like to invoke Trygve Aasgaard’s FART principle – Free All Research Therapists.
* names also give-away our biographies and our musical histories.
Evaluation Tool for Qualitative Studies
(1) Study overview
| Bibliographic Details | Author, title, source (publisher and place of publication), year |
| Purpose | What are the aims of the study? |
| If the paper is part of a wider study, what are its aims? | |
| Key Findings | What are the key findings of the study? |
| Evaluative Summary | What are the strengths and weaknesses of the study and theory, policy and practice implications? |
(2) Study, setting, sample and ethics
| Phenomena under Study | What is being studied? |
| Is sufficient detail given of the nature of the phenomena under study? | |
| Context I: Theoretical Framework | What theoretical framework guides or informs the study? |
| In what ways is the framework reflected in the way the study was done? | |
| Within what geographical and care setting is the study carried out? | |
| Context II: Setting | Within what geographical and care setting is the study carried out? |
| What is the rationale for choosing this setting? | |
| Is the setting appropriate and/or sufficiently specific for examination of the research question? | |
| Is sufficient detail given about the setting? | |
| Over what time period is the study conducted? | |
| Context III: Sample (events, persons, times and settings) | How is the sample (events, persons, times and settings) selected? (For example, theoretically informed, purposive, convenience, chosen to explore contrasts) |
| Is the sample (informants, settings and events) appropriate to the aims of the study? | |
| Is the sample appropriate in terms of depth (intensity of data collection – individuals, settings and events) and width across time, settings and events (For example, to capture key persons and events, and to explore the detail of inter-relationships)? | |
| What are the key characteristics of the sample (events, persons, times and settings)? | |
| Context IV: Outcomes | What outcome criteria are used in the study? |
| Whose perspectives are addressed (professional, service, user, carer)? | |
| Is there sufficient breadth (e.g. contrast of two or more perspective) and depth (e.g. insight into a single perspective)? |
(3) Ethics
| Ethics | Was Ethical Committee approval obtained? |
| Was informed consent obtained from participants of the study? | |
| Have ethical issues been adequately addressed? |
(4) Data collection, analysis and potential researcher bias
| Data Collection | What data collection methods are used to obtain and record the data? (For example, provide insight into: data collected, appropriateness and availability for independent analysis) |
| Is the information collected with sufficient detail and depth to provide insight into the meaning and perceptions of informants? | |
| Is the process of fieldwork adequately described? (For example, account of how the data were elicited; type and range of questions; interview guide; length and timing of observation work; note taking) | |
| What role does the researcher adopt within the setting? | |
| Is there evidence of reflexivity, that is, providing insight into the relationship between the researcher, setting, data production and analysis? | |
| Data Analysis | How were the data analysed? |
| How adequate is the description of the data analysis? (For example, to allow reproduction; steps taken to guard against selectivity) | |
| Is adequate evidence provided to support the analysis? (For example, includes original / raw data extracts; evidence of iterative analysis; representative evidence presented; efforts to establish validity – searching for negative evidence, use of multiple sources, data triangulation); reliability / consistency (over researchers, time and settings; checking back with informants over interpretation) | |
| Are the findings interpreted within the context of other studies and theory? Researcher’s Potential Bias |
|
| Are the researcher’s own position, assumptions and possible biases outlined? (Indicate how those could affect the study, in particular, the analysis and interpretation of the data) |
(5) Policy and practice implications
| Implications | To what setting are the study findings generalisable? (For example, is the setting typical or representative of care settings and in what respects? If the setting is atypical, will this present a stronger or weaker test of the hypothesis?) |
| To what population are the studys findings generalisable? | |
| Is the conclusion justified given the conduct of the study (For example, sampling procedure; measures of outcome used and results achieved?) | |
| What are the implications for policy? | |
| What are the implications for service practice? |
(6) Other comments
| Other Comments | What were the total number of references used in the study? |
| Are there any other noteworthy features of the study? | |
| List other study references | |
| Reviewer | Name of reviewer |
| Review date |
May 27, 2002; Gary Ansdell ():
Goethe’s Response: On Qualifying the Qualifiers
Brynjulf Stige’s recent and timely paper on possible criteria for the assessment of qualitative research studies and papers (and the subsequent responses to this) led me back to the Berlin Symposium on Qualitative Research in April, where these ideas were aired in our work group1. For reasons of mutual interest (and some chance factors) the spirit of Goethe hovered above many of our discussions there. Not the poet Goethe, but the scientist – who spent a significant part of his life studying botany, physics and meteorology, famously challenging Newton’s theory of colour.
Goethe’s work in these spheres can be seen both as a road not taken in mainstream European scientific thinking, but also as one of the progenitors of modern traditions of qualitative research. For, above all, what Goethe strived for was a way of doing science which could validate qualitative ways of knowing about the world, its objects and processes. His method of ‘delicate empiricism’ aimed to contemplate but not destroy the things it tried to study. Goethe was also perhaps the first ‘reflexive’ researcher – realising that all knowledge (including scientific knowledge) is historically and culturally situated, and must understand itself as such. He was much maligned for his scientific efforts – seen as a quack, an amateur, as not understanding mathematics enough, as being methodologically dubious (does this ring a bell, music therapist colleagues?). Yet he remained passionately concerned with the methods and standards of science – and remains still part of the ‘official opposition’ to reductionist positivism.
What then would he make of Stige’s “ERIC”? Would he have something to say about our current concern to qualify the qualifiers? Would he recognise our contemporary dilemmas in this area?
I took a Bank Holiday break to Jena and talked to the grand old man of European thought2:
Gary: You are both an artist and a scientist. Are there different criteria of judgement for artistic and scientific work?
Goethe: Young Man, I have been considering this question since the 1790s! If you will excuse a passing moment’s grandiosity, you will recall how one of your contemporaries will say that philosophy is no more than footnotes to Plato! Well equally, your current concerns are footnotes to Goethe. At one level it is easier to locate error in those using the protocols of mathematical science and its styles of writing-up. But one should be aware of making too simplistic a conclusion. ‘One hears that only mathematics is certain; it is no more certain than any other form of knowledge and activity. It is certain when it confines itself wisely only to things about which certainty can be attained, and in so far as certainty is attainable’. In the domain of the arts, and in the science of qualities, the intention is often broader, and hence less easy to judge…
Gary: Are you saying that quality can only be judged in relation to the researcher’s chosen methodology and purpose?
Goethe: As one of my contemporaries has it ‘Beauty is Truth/ Truth Beauty…’ – the shape of the whole alongside the truth of the parts! Quite the same in the arts and the sciences. There is beauty in writing, there is beauty in a mathematical proof, in an elegant piece of research, in a well-turned phrase. What of beauty, then, is true? Right relationship – the fitting of perception, understanding, execution, control, economy of means, logic balanced by feeling, and awareness of consequence. We do all recognise this when we see it! Your generation has put much effort into cleaving the arts and the sciences – I hear people are rarely both now – and with this you pretend that the criteria for the quality of artistic and scientific work are essentially different. I beg to differ!
Gary: To put the question the other way around: How do you discern bad work?
Goethe: Both the incompetent and the malicious fail to convey truth. It matters little what the sign is; whether the vehicle be words, numbers or tones. ‘It is true that human capacity to understand and reason is basic to language, but its use does not really presuppose clear understanding, developed reason and an upright will in the use of the one using language. It is a tool that can be used to a purpose and arbitrarily; one can use it just as much for subtly confusing dialectics as for confused-shady mysticism’. We must then, as the Bible tells us, ‘Search out the truth in the inward parts’. My friend Beethoven likes to quote this. Were your music still contrapuntal you would know what I mean, young man! The same with science: look around and inside it. Look at the relationship between the inward parts and the shape of the whole. Look at the integrity of the work, and of the integrity of the researcher. The wise man already knows he’s a fool, faced with the complexities of the world. ‘The highest achievement of the human being as a thinking being is to have probed what is knowable and quietly to revere what is unknowable’.
Gary: Can we now apply these ideas to the problem of establishing fair and general standards of evaluating research work? My age is one of ‘criteria’, attempting to rationalise judgements, so let me now suggest to you my colleague’s criteria for evaluating qualitative research studies: ERIC….
Goethe: It sounds as if you want it both ways, young man! You would wish to qualify the qualifiers through evaluating their value! Your times sound worse than mine!
Gary: Sorry! But let’s try: the first criterion is ‘E’, representing the relevance and solidity of empirical evidence – sometimes difficult, given the fact that the researcher himself is the knower…
Goethe: ‘Insofar as we make use of our healthy senses, the human being is the most powerful and exact scientific instrument possible’, but nevertheless ‘the manifestation of a phenomenon is not independent of the observer – it is caught up and entangled in his individuality’. An empirical stance certainly helps, but we must not pretend it provides a certain foundation of knowledge. If there is a foundation of knowledge away from the ground we stand upon, that is! As Neurath will say: ‘We are like sailors who have to rebuild their ship on the open sea, without ever being able to dismantle it in a dry dock and reconstruct it from the best components.’
Gary: The second criterion ‘R’ stands for the problems of representation, and how language mediates both our knowledge and the communication of this…
Goethe: ‘How difficult it is to refrain from replacing a thing with its sign, to keep the object alive before us instead of killing it with a word’. Wittgenstein will caution against ‘the bewitchment of our intelligence by means of language’, and so too I have argued that we attain our knowledge of the world through many modes – intuitive, sensual, aesthetic, analytical, synthetic, spiritual… so we must guard against our accounts being monological. Many things can be shown – few can be said!
Gary: The third criterion is ‘I’, for the primacy of interpretation…
Goethe: All understanding involves pre-understanding! We all have a story to tell, a plot to thicken! ‘The ultimate goal would be: to grasp that everything in the realm of fact is already theory. The blue of the sky shows us the law of chromatics. Let us not seek for something behind the phenomena – they themselves are the theory.’ We need to understand the relationship between the phenomenological experience of the world, the hermeneutic ‘reading’ of its texts, and the uses of theory. Beware theory as explanation! Beware especially when explanations replace phenomena.
Gary: The last criterion is ‘C’, for the obligation to produce pertinent critique, to be aware of its stance and potential effects…
Goethe: My studies of some of the oddities of physical and natural research of my time led me to believe that science reflects its time and place, and helps make the world it claims merely to study. ‘We might venture to suggest the statement that the history of science is science itself’. My critics had their reasons to discredit a science of qualities – as I believe they do in your own time. Qualitative knowledge seldom supports the values of industrial or post-industrial societies. I may, of course, have been wrong in some ways, though wrong, I think, in a right fashion.
Gary: What are your final thoughts?
Goethe: That your generation realise that your current endeavours – well-founded though they be – are part of a long tradition of critical scholarship: one which has considered the foundational relationship between (i) ontology – our agreement as to the things of the world; (ii) epistemology – how knowing is done concerning such things, and (iii) methodology – systematic ways of promoting such knowing. Only from such a perspective can come any critical surety: ‘The highest achievement of the human being as a thinking being is to have probed what is knowable and quietly to revere what is unknowable’. Now, if you’ll excuse me, young man, I have a screenplay of Faust to be getting-on with…
Note 1.
Thank you to Brynjulf Stige, Ken Aigen and Eckhard Weymann for bearing with me while I essayed Goethe’s possible relevance to the matters in hand!
|back to text|
Note 2.
Goethe’s answers in this dialogue are partly fashioned from his extant remarks on scientific work and thinking. My sources are: Stopp, E. (Ed.)(1998). Johann Von Goethe – Maxims & Reflections. Penguin; Naydler, J. (ed.)(1996). Goethe on Science. Floris ; (1996).The Wholeness of Nature – Goethe’s Way of Science Floris .
|back to text|
June 4, 2002; Jane Edwards (Jane.Edwards@ul.ie):
Another contribution to the forum
(1) Although the other contributors to the discussion have made many valuable points that would benefit further consideration, I find myself preoccupied with Brynjulf’s question as to whether we can find a way to make the acronym EPIC. This suggests I am still interested in finding a publication criteria that could be useful and am wondering whether EPIC contributes more successfully to recall and reflects better the endeavours of the qualitative researcher.
(2) I have two suggestions – One is to use the word Presentation along with Representation and write it as Presentation/Representation. In writing up a qualitative research project for publication, both presentation and representation are important. In a sense the representation components are a re-presentation of the presented elements following a process of distillation.
(3) The second suggestion is to use the word ‘perspective’ instead of ‘representation’. This addresses concerns I raised in my response to Brynjulf’s initial paper that the perspective of the researcher to the research and their theoretical underpinnings in choosing their method require consideration and inclusion – see my previous response (4) & (5). This issue was also expolored extensively in discussion and debates in the Sauen symposium group consisting of myself, Michele Forinash, Barbara Wheeler and Trygve Aasgard and I wish to acknowledge the influence of ideas we discussed in that group in making me so keen to find a way to require inclusion of not just the rationale for the research but also the theoretical and philosophical clarity that makes for interesting work (and here I make reference to David’s response (1) that whether a study is interesting or not is key for him).
© 2002: Nordic Journal of Music Therapy
(last updated June 4, 2002 by Rune Rolvsjord)