The purposes of this article are to position mixed methods research

(mixed research is a synonym) as the natural complement to tradi-

tional qualitative and quantitative research, to present pragmatism

as offering an attractive philosophical partner for mixed methods re-

search, and to provide a framework for designing and conducting

mixed methods research. In doing this, we briefly review the para-

digm “wars” and incompatibility thesis, we show some commonali-

ties between quantitative and qualitative research, we explain the

tenets of pragmatism, we explain the fundamental principle of mixed

research and how to apply it, we provide specific sets of designs for

the two major types of mixed methods research (mixed-model de-

signs and mixed-method designs), and, finally, we explain mixed meth-

ods research as following (recursively) an eight-step process. A key

feature of mixed methods research is its methodological pluralism

or eclecticism, which frequently results in superior research (com-

pared to monomethod research). Mixed methods research will be

successful as more investigators study and help advance its concepts

and as they regularly practice it.

For more than a century, the advocates of quantitative andqualitative research paradigms have engaged in ardent dis-pute.1 From these debates, purists have emerged on both sides (cf. Campbell & Stanley, 1963; Lincoln & Guba, 1985).2

Quantitative purists (Ayer, 1959; Maxwell & Delaney, 2004; Popper, 1959; Schrag, 1992) articulate assumptions that are con- sistent with what is commonly called a positivist philosophy.3, 4

That is, quantitative purists believe that social observations should be treated as entities in much the same way that physical scientists treat physical phenomena. Further, they contend that the observer is separate from the entities that are subject to ob- servation. Quantitative purists maintain that social science inquiry should be objective. That is, time- and context-free gen- eralizations (Nagel, 1986) are desirable and possible, and real causes of social scientific outcomes can be determined reliably and validly. According to this school of thought, educational re- searchers should eliminate their biases, remain emotionally de- tached and uninvolved with the objects of study, and test or empirically justify their stated hypotheses. These researchers have traditionally called for rhetorical neutrality, involving a formal

Mixed Methods Research: A Research Paradigm Whose Time Has Come by R. Burke Johnson and Anthony J. Onwuegbuzie

writing style using the impersonal passive voice and technical ter- minology, in which establishing and describing social laws is the major focus (Tashakkori & Teddlie, 1998).

Qualitative purists (also called constructivists and interpretivists) reject what they call positivism. They argue for the superiority of constructivism, idealism, relativism, humanism, hermeneutics, and, sometimes, postmodernism (Guba & Lincoln, 1989; Lincoln & Guba, 2000; Schwandt, 2000; Smith, 1983, 1984). These purists contend that multiple-constructed realities abound, that time- and context-free generalizations are neither desirable nor possible, that research is value-bound, that it is impossible to dif- ferentiate fully causes and effects, that logic flows from specific to general (e.g., explanations are generated inductively from the data), and that knower and known cannot be separated because the subjective knower is the only source of reality (Guba, 1990). Qualitative purists also are characterized by a dislike of a de- tached and passive style of writing, preferring, instead, detailed, rich, and thick (empathic) description, written directly and some- what informally.

Both sets of purists view their paradigms as the ideal for re- search, and, implicitly if not explicitly, they advocate the in- compatibility thesis (Howe, 1988), which posits that qualitative and quantitative research paradigms, including their associated methods, cannot and should not be mixed. The quantitative versus qualitative debate has been so divisive that some gradu- ate students who graduate from educational institutions with an aspiration to gain employment in the world of academia or re- search are left with the impression that they have to pledge alle- giance to one research school of thought or the other. Guba (a leading qualitative purist) clearly represented the purist position when he contended that “accommodation between paradigms is impossible . . . we are led to vastly diverse, disparate, and to- tally antithetical ends” (Guba, 1990, p. 81). A disturbing fea- ture of the paradigm wars has been the relentless focus on the differences between the two orientations. Indeed, the two dom- inant research paradigms have resulted in two research cultures, “one professing the superiority of ‘deep, rich observational data’ and the other the virtues of ‘hard, generalizable’ . . . data” (Sieber, 1973, p. 1335).

Our purpose in writing this article is to present mixed meth- ods research as the third research paradigm in educational re- search.5 We hope the field will move beyond quantitative versus qualitative research arguments because, as recognized by mixed methods research, both quantitative and qualitative research are important and useful. The goal of mixed methods research is not to replace either of these approaches but rather to draw from theEducational Researcher, Vol. 33, No. 7, pp. 14–26



15OCTOBER 2004

strengths and minimize the weaknesses of both in single research studies and across studies. If you visualize a continuum with qualitative research anchored at one pole and quantitative re- search anchored at the other, mixed methods research covers the large set of points in the middle area. If one prefers to think cat- egorically, mixed methods research sits in a new third chair, with qualitative research sitting on the left side and quantitative re- search sitting on the right side.

Mixed methods research offers great promise for practicing researchers who would like to see methodologists describe and develop techniques that are closer to what researchers actually use in practice. Mixed methods research as the third research paradigm can also help bridge the schism between quantitative and qualitative research (Onwuegbuzie & Leech, 2004a). Meth- odological work on the mixed methods research paradigm can be seen in several recent books (Brewer & Hunter, 1989; Creswell, 2003; Greene, Caracelli, & Graham, 1989; Johnson & Christensen, 2004; Newman & Benz, 1998; Reichardt & Rallis, 1994; Tashakkori & Teddlie, 1998, 2003). Much work remains to be undertaken in the area of mixed methods research regarding its philosophical positions, designs, data analysis, va- lidity strategies, mixing and integration procedures, and ratio- nales, among other things. We will try to clarify the most important issues in the remainder of this article.

Commonalities Among the Traditional Paradigms

Although there are many important paradigmatic differences be- tween qualitative and quantitative research (which have been fre- quently written about in the Educational Researcher and other places), there are some similarities between the various approaches that are sometimes overlooked. For example, both quantitative and qualitative researchers use empirical observations to address research questions. Sechrest and Sidani (1995, p. 78) point out that both methodologies “describe their data, construct explana- tory arguments from their data, and speculate about why the outcomes they observed happened as they did.” Additionally, both sets of researchers incorporate safeguards into their inquiries in order to minimize confirmation bias and other sources of in- validity (or lack of trustworthiness) that have the potential to exist in every research study (Sandelowski, 1986).

Regardless of paradigmatic orientation, all research in the so- cial sciences represents an attempt to provide warranted assertions about human beings (or specific groups of human beings) and the environments in which they live and evolve (Biesta & Burbules, 2003). In the social and behavioral sciences, this goal of under- standing leads to the examination of many different phenomena, including holistic phenomena such as intentions, experiences, at- titudes, and culture, as well as more reductive phenomena such as macromolecules, nerve cells, micro-level homunculi, and bio- chemical computational systems (de Jong, 2003). There is room in ontology for mental and social reality as well as the more micro and more clearly material reality. Although certain methodolo- gies tend to be associated with one particular research tradition, Dzurec and Abraham (1993, p. 75) suggest that “the objectives, scope, and nature of inquiry are consistent across methods and across paradigms.” We contend that researchers and research methodologists need to be asking when each research approach

is most helpful and when and how they should be mixed or com- bined in their research studies.

We contend that epistemological and methodological pluralism should be promoted in educational research so that researchers are informed about epistemological and methodological possibilities and, ultimately, so that we are able to conduct more effective re- search. Today’s research world is becoming increasingly inter- disciplinary, complex, and dynamic; therefore, many researchers need to complement one method with another, and all researchers need a solid understanding of multiple methods used by other scholars to facilitate communication, to promote collaboration, and to provide superior research. Taking a non-purist or com- patibilist or mixed position allows researchers to mix and match design components that offer the best chance of answering their specific research questions. Although many research procedures or methods typically have been linked to certain paradigms, this linkage between research paradigm and research methods is nei- ther sacrosanct nor necessary (Howe, 1988, 1992). For example, qualitative researchers should be free to use quantitative meth- ods, and quantitative researchers should be free to use qualitative methods. Also, research in a content domain that is dominated by one method often can be better informed by the use of mul- tiple methods (e.g., to give a read on methods-induced bias, for corroboration, for complimentarity, for expansion; see Greene et al., 1989). We contend that epistemological and paradigmatic ecumenicalism is within reach in the research paradigm of mixed methods research.

Philosophical Issues Debates

As noted by Onwuegbuzie and Teddlie (2003), some individuals who engage in the qualitative versus quantitative paradigm debate appear to confuse the logic of justification with research methods. That is, there is a tendency among some researchers to treat epistemology and method as being synonymous (Bryman, 1984; Howe, 1992). This is far from being the case because the logic of justification (an important aspect of epistemology) does not dic- tate what specific data collection and data analytical methods re- searchers must use. There is rarely entailment from epistemology to methodology (Johnson, Meeker, Loomis, & Onwuegbuzie, 2004; Phillips, 2004). For example, differences in epistemologi- cal beliefs (such as a difference in beliefs about the appropriate logic of justification) should not prevent a qualitative researcher from utilizing data collection methods more typically associated with quantitative research, and vice versa.

There are several interesting myths that appear to be held by some purists. For example, on the “positivist” side of the fence, the barriers that quantitative educational researchers have built arise from a narrow definition of the concept of “science.” 6 As noted by Onwuegbuzie (2002), modern day “positivists” claim that science involves confirmation and falsification, and that these methods and procedures are to be carried out objectively. However, they disregard the fact that many human (i.e., subjec- tive) decisions are made throughout the research process and that researchers are members of various social groups. A few examples of subjectivism and intersubjectivism in quantitative research in- clude deciding what to study (i.e., what are the important prob- lems?), developing instruments that are believed to measure what the researcher views as being the target construct, choosing the




specific tests and items for measurement, making score interpre- tations, selecting alpha levels (e.g., .05), drawing conclusions and interpretations based on the collected data, deciding what ele- ments of the data to emphasize or publish, and deciding what findings are practically significant. Obviously, the conduct of fully objective and value-free research is a myth, even though the regulatory ideal of objectivity can be a useful one.

Qualitative researchers also are not immune from constructive criticism. Some qualitative purists (e.g., Guba, 1990) openly admit that they adopt an unqualified or strong relativism, which is logically self-refuting and (in its strong form) hinders the de- velopment and use of systematic standards for judging research quality (when it comes to research quality, it is not the case that anyone’s opinion about quality is just as good as the next per- son’s, because some people have no training or expertise or even interest in research). We suspect that most researchers are soft rel- ativists (e.g., respecting the opinions and views of different peo- ple and different groups). When dealing with human research, soft relativism simply refers to a respect and interest in under- standing and depicting individual and social group differences (i.e., their different perspectives) and a respect for democratic ap- proaches to group opinion and value selection. Again, however, a strong relativism or strong constructivism runs into problems; for example, it is not a matter of opinion (or individual reality) that one should or can drive on the left-hand side of the road in Great Britain—if one chooses to drive on the right side, he or she will likely have a head-on collision, at some point, and end up in the hospital intensive care unit, or worse (this is a case where sub- jective and objective realities directly meet and clash). The strong ontological relativistic or constructivist claim in qualitative re- search that multiple, contradictory, but equally valid accounts of the same phenomenon are multiple realities also poses some po- tential problems. Generally speaking, subjective states (i.e., cre- ated and experienced realities) that vary from person to person and that are sometimes called “realities” should probably be called (for the purposes of clarity and greater precision) multiple perspectives or opinions or beliefs (depending on the specific phe- nomenon being described) rather than multiple realities (Phillips & Burbules, 2000). If a qualitative researcher insists on using the word reality for subjective states, then for clarity we would rec- ommend that the word subjective be placed in front of the word reality (i.e., as in subjective reality or in many cases intersubjec- tive reality) to direct the reader to the focus of the statement. We agree with qualitative researchers that value stances are often needed in research; however, it also is important that research is more than simply one researcher’s highly idiosyncratic opinions written into a report. Fortunately, many strategies are recognized and regularly used in qualitative research (such as member check- ing, triangulation, negative case sampling, pattern matching, ex- ternal audits) to help overcome this potential problem and produce high-quality and rigorous qualitative research. Finally, qualitative researchers sometimes do not pay due attention to providing an adequate rationale for interpretations of their data (Onwuegbuzie, 2000), and qualitative methods of analyses too “often remain private and unavailable for public inspection” (Constas, 1992, p. 254). Without public inspection and ade- quate standards, how is one to decide whether what is claimed is trustworthy or defensible?

Fortunately, many (or most?) qualitative researchers and quan- titative researchers (i.e., postpositivists) have now reached basic agreement on several major points of earlier philosophical dis- agreement (e.g., Phillips & Burbules, 2000; Reichardt & Cook, 1979; Reichardt & Rallis, 1994). Basic agreement has been reached on each of the following issues: (a) the relativity of the “light of reason” (i.e., what appears reasonable can vary across per- sons); (b) theory-laden perception or the theory-ladenness of facts (i.e., what we notice and observe is affected by our background knowledge, theories, and experiences; in short, observation is not a perfect and direct window into “reality”); (c) underdeter- mination of theory by evidence (i.e., it is possible for more than one theory to fit a single set of empirical data); (d) the Duhem- Quine thesis or idea of auxiliary assumptions (i.e., a hypothesis cannot be fully tested in isolation because to make the test we also must make various assumptions; the hypothesis is embedded in a holistic network of beliefs; and alternative explanations will continue to exist); (e) the problem of induction (i.e., the recogni- tion that we only obtain probabilistic evidence, not final proof in empirical research; in short, we agree that the future may not re- semble the past); (f) the social nature of the research enterprise (i.e., researchers are embedded in communities and they clearly have and are affected by their attitudes, values, and beliefs); and (g) the value-ladenness of inquiry (this is similar to the last point but specifically points out that human beings can never be com- pletely value free, and that values affect what we choose to in- vestigate, what we see, and how we interpret what we see).

Pragmatism as the Philosophical Partner for Mixed Methods Research

We do not aim to solve the metaphysical, epistemological, axio- logical (e.g., ethical, normative), and methodological differences between the purist positions. And we do not believe that mixed methods research is currently in a position to provide perfect so- lutions. Mixed methods research should, instead (at this time), use a method and philosophy that attempt to fit together the in- sights provided by qualitative and quantitative research into a workable solution. Along these lines, we advocate consideration of the pragmatic method of the classical pragmatists (e.g., Charles Sanders Peirce, William James, and John Dewey) as a way for researchers to think about the traditional dualisms that have been debated by the purists. Taking a pragmatic and bal- anced or pluralist position will help improve communication among researchers from different paradigms as they attempt to advance knowledge (Maxcy, 2003; Watson, 1990). Pragmatism also helps to shed light on how research approaches can be mixed fruitfully (Hoshmand, 2003); the bottom line is that research ap- proaches should be mixed in ways that offer the best opportuni- ties for answering important research questions.

The pragmatic rule or maxim or method states that the current meaning or instrumental or provisional truth value (which James [1995, 1907 original] would term “cash value”) of an expression (e.g., “all reality has a material base” or “qualitative research is su- perior for uncovering humanistic research findings”) is to be de- termined by the experiences or practical consequences of belief in or use of the expression in the world (Murphy, 1990). One can apply this sensible effects- or outcome-oriented rule through thinking (thinking about what will happen if you do X), practi-



17OCTOBER 2004

cal experiences (observing what happens in your experience when you do X), or experiments (formally or informally trying a rule and observing the consequences or outcomes).

In the words of Charles Sanders Peirce (1878), the pragmatic method or maxim (which is used to determine the meaning of words, concepts, statements, ideas, beliefs) implies that we should “consider what effects, that might conceivably have practical bearings, we conceive the object of our conception to have. Then our conception of these effects is the whole of our conception of the object” (this quote is found at the end of Section II in How to Make Our Ideas Clear). Building on Peirce’s lead, James (1995, 1907 original) argued that “The pragmatic method is primarily a method of settling metaphysical disputes that otherwise might be interminable. . . . The pragmatic method in such cases is to try to interpret each notion by tracing its respective practical con- sequences” (p. 18). Extending the works of Peirce and James, Dewey spent his career applying pragmatic principles in devel- oping his philosophy and in the practice of educating children (e.g., the Experimental School of Chicago). Dewey (1948, 1920 original) stated that “in order to discover the meaning of the idea [we must] ask for its consequences” (p. 132). In short, when judging ideas we should consider their empirical and practical consequences. Peirce, James, and Dewey were all interested in ex- amining practical consequences and empirical findings to help in understanding the import of philosophical positions and, im- portantly, to help in deciding which action to take next as one attempts to better understand real-world phenomena (including psychological, social, and educational phenomena).

If two ontological positions about the mind/body problem (e.g., monism versus dualism), for example, do not make a dif- ference in how we conduct our research then the distinction is, for practical purposes, not very meaningful. We suspect that some philosophical differences may lead to important practical consequences while many others may not.7 The full sets of be- liefs characterizing the qualitative and quantitative approaches or paradigms have resulted in different practices, and, based on our observation and study, we believe it is clear that both qualitative and quantitative research have many benefits and many costs. In some situations the qualitative approach will be more appropriate; in other situations the quantitative approach will be more appro- priate. In many situations, researchers can put together insights and procedures from both approaches to produce a superior prod- uct (i.e., often mixed methods research provides a more workable solution and produces a superior product). We are advocating a needs-based or contingency approach to research method and concept selection.

Philosophical debates will not end as a result of pragmatism, and certainly they should not end. Nonetheless, we agree with others in the mixed methods research movement that consider- ation and discussion of pragmatism by research methodologists and empirical researchers will be productive because it offers an immediate and useful middle position philosophically and meth- odologically; it offers a practical and outcome-oriented method of inquiry that is based on action and leads, iteratively, to further action and the elimination of doubt; and it offers a method for selecting methodological mixes that can help researchers better answer many of their research questions. Pragmatically inclined philosophers and researchers also would suggest that we can reach

some agreement about the importance of many (culturally de- rived) values and desired ends, such as, for example, preventing the dropping out of school by adolescents, reducing the use of il- licit drugs by children and adolescents, finding effective teaching techniques for different kinds of students, educating children and adults (i.e., increasing their knowledge), helping to reduce discrimination in society, and attempting to eliminate or reduce mental, learning, and other disabilities. In other words, pragma- tism takes an explicitly value-oriented approach to research.

We reject an incompatibilist, either/or approach to paradigm selection and we recommend a more pluralistic or compatibilist approach. Beyond the basic pragmatic method or maxim (i.e., translated in mixed methods research as “choose the combina- tion or mixture of methods and procedures that works best for answering your research questions”) there also is a full philo- sophical system of pragmatism which was systematically devel- oped by the classical pragmatists (Peirce, James, Dewey) and has been refined in newer directions by latter-day neo-pragmatists (e.g., Davidson, Rescher, Rorty, Putnam) (see Menand, 1997; Murphy, 1990; Rescher, 2000; Rorty, 2000). To provide the reader with a better understanding of the full philosophy of prag- matism (for consideration), we have outlined, in Table 1, what we believe are classical pragmatism’s most general and important characteristics.

Although we endorse pragmatism as a philosophy that can help to build bridges between conflicting philosophies, pragma- tism, like all current philosophies, has some shortcomings. In Table 2 we present some of these. Researchers who are interested in applying pragmatism in their works should consider the short- comings, which also need to be addressed by philosophically in- clined methodologists as they work on the project of developing a fully working philosophy for mixed methods research. Practic- ing researchers should be reflexive and strategic in avoiding the potential consequences of these weaknesses in their works.

Comparing Qualitative, Quantitative, and Mixed Methods Research

Mixed methods research is formally defined here as the class of re- search where the researcher mixes or combines quantitative and qual- itative research techniques, methods, approaches, concepts or language into a single study. Philosophically, it is the “third wave” or third research movement, a movement that moves past the paradigm wars by offering a logical and practical alternative. Philosophically, mixed research makes use of the pragmatic method and system of philosophy. Its logic of inquiry includes the use of induction (or discovery of patterns), deduction (testing of theories and hypotheses), and abduction (uncovering and relying on the best of a set of explanations for understanding one’s results) (e.g., de Waal, 2001).

Mixed methods research also is an attempt to legitimate the use of multiple approaches in answering research questions, rather than restricting or constraining researchers’ choices (i.e., it rejects dogmatism). It is an expansive and creative form of research, not a limiting form of research. It is inclusive, pluralistic, and com- plementary, and it suggests that researchers take an eclectic ap- proach to method selection and the thinking about and conduct of research. What is most fundamental is the research question— research methods should follow research questions in a way that




offers the best chance to obtain useful answers. Many research questions and combinations of questions are best and most fully answered through mixed research solutions.

In order to mix research in an effective manner, researchers first need to consider all of the relevant characteristics of quantitative and qualitative research. For example, the major characteristics of traditional quantitative research are a focus on deduction, confir- mation, theory/hypothesis testing, explanation, prediction, stan- dardized data collection, and statistical analysis (see Table 3 for a more complete list). The major characteristics of traditional qualitative research are induction, discovery, exploration, theory/ hypothesis generation, the researcher as the primary “instrument” of data collection, and qualitative analysis (see Table 4 for a more complete list).

Gaining an understanding of the strengths and weaknesses of quantitative and qualitative research puts a researcher in a posi- tion to mix or combine strategies and to use what Johnson and Turner (2003) call the fundamental principle of mixed research. According to this principle, researchers should collect multiple data using different strategies, approaches, and methods in such a way that the resulting mixture or combination is likely to re- sult in complementary strengths and nonoverlapping weaknesses (also see Brewer & Hunter, 1989). Effective use of this principle is a major source of justification for mixed methods research be- cause the product will be superior to monomethod studies. For example, adding qualitative interviews to experiments as a ma- nipulation check and perhaps as a way to discuss directly the is- sues under investigation and tap into participants’ perspectives

Table 1 General Characteristics of Pragmatism

• The project of pragmatism has been to find a middle ground between philosophical dogmatisms and skepticism and to find a workable solution (sometimes including outright rejection) to many longstanding philosophical dualisms about which agreement has not been historically forthcoming.

• Rejects traditional dualisms (e.g., rationalism vs. empiricism, realism vs. antirealism, free will vs. determinism, Platonic ap- pearance vs. reality, facts vs. values, subjectivism vs. objec- tivism) and generally prefers more moderate and commonsense versions of philosophical dualisms based on how well they work in solving problems.

• Recognizes the existence and importance of the natural or physical world as well as the emergent social and psycholog- ical world that includes language, culture, human institutions, and subjective thoughts.

• Places high regard for the reality of and influence of the inner world of human

"Is this question part of your assignment? We can help"