Running head: CRITIQUES OF MIXED METHOD STUDIES 1

Critiques of Mixed Methods Studies

Dionne Woolford-Hudgins

Morgan State University

EDHE 627

February 10, 2019

CRITIQUES OF MIXED METHODS STUDIES 2

It is important that scholars recognize which types of research studies are appropriate for

the use of a mixed methodology approach (Creswell & Plano Clark, 2018). Researchers should

provide their listening audience with sound justification regarding the decision to use a mixed

methods approach to answer their research questions, as not all research is suitable for this

methodology. The purpose of this paper is to examine and critique two journal articles and two

dissertation studies.

According to Creswell and Plano Clark (2018), titles in mixed methods studies should

include four major components: major topic, study participants, site, and general methods

approach. After the title is framed, the statement of the problem should follow to introduce the

study and the researcher’s justification for utilizing a mixed methods approach. The purpose

statement should follow and serves to identify the overarching aim or objectives of the study.

Specific research questions should be designed to narrow the purpose statement and predictions

to be investigated in the study.

In an article entitled, “An Investigation of High-Achieving African-American Students

Attending Community Colleges: A Mixed Methods Research Study”, authors Gipson, Mitchell,

and McLean (2017), explored non-cognitive factors that influenced high academic achievement

amongst African-American students in four different community colleges. The title of this study

is well written as it clearly states and includes the words “mixed methods”.

The problem and purpose of this research are explained within the first three paragraphs

of the article and reads as follows: “Thus, the purpose of this study was to investigate academic

success and why students decided to remain enrolled at community colleges, rather than focusing

on deficit and dropout.” The purpose statement is appropriately and immediately followed by

the research question, “What influences personal and academic success for high-achieving
CRITIQUES OF MIXED METHODS STUDIES 3

African-American students?” A deficit is clearly identified regarding the focus of prior research

studies.

Within the review of literature the researchers uses Tinto’s original theory as a theoretical

framework on which to build the philosophical paradigm. The researchers point out that though

Tinto’s theory was embedded in the experiences of students at four year institutions that further

research has proven theoretical applicability to community colleges. The review of literature

includes the types of research that had been conducted regarding African-American student

retention and involvement, and identified that there was a gap in the literature that focused on

factors that motivated students to persists and complete, as opposed to starting from a deficit

point of view or lens.

The research design was clearly stated by the scholars as mixed methods. The authors in

the methodology section of the article explained that a concurrent triangulation study was chosen

as the design approach. It was further explained why this approach was chosen, how data was

collected and then merged for interpretation. Justification for the use of conducting a mixed

methods studies was clearly explained, as the authors stated that few studies examined the

experiences of high-achieving African-American students attending community colleges, with

most coming from a deficit point of view.

The methodology was very clearly delineated. Data were collected and interpreted using

a concurrent triangulation strategy, over two separate phases and were then integrated for

interpretation to allow the researchers to examine the convergence or divergence of the data (As

cited by Gipson et al., 2017). It discussed the participants and setting. There were 243 students

in the quantitative portion and eight in the qualitative portion of the study. The scholars spelled

out in detail how data was collected and analyzed both quantitatively and qualitatively. Two of
CRITIQUES OF MIXED METHODS STUDIES 4

the scholars were responsible for collecting and analyzing the data. When concepts were

encountered that one had and the other didn’t it was explained that the two engaged in critical

conversations to either reject or include the concepts.

Results were appropriately separated by quantitative and qualitative findings which were

both summarized in clear to understand text and chart exhibits. The discussion section of the

article showed how the data were integrated in support of the mixed-methodology approach.

The scholars re-directed the reader back to the purpose of the mixed methods approach and

reminded the reader of the research question being answered in the study. The scholars revisited

what the current literature stated and how the study either confirmed or rejected previous

findings in support of African-American student success.

Though there was no section indicated as the “conclusion”, the discussion, limitation,

implications for future research and practice included all the necessary components of a

conclusion. Value was definitely obtained from utilizing a mixed-methods approach to conduct

this study as it expanded the knowledge of and created new implications as it relates to the

success of African-Americans in community college settings.

CRITIQUES OF MIXED METHODS STUDIES 5

QUANTITATIVE QUALITATIVE

Data Collection Data Collection

Data Analysis Data Analysis

Results compared, integrated & interpreted

Mixed-Methods Concurrent Triangulation Design

CRITIQUES OF MIXED METHODS STUDIES 6

In the journal article entitled, “Student Persistence in a Distributed Doctoral Program

in Educational Leadership in Higher Education: A Mixed Methods Study,” scholars

Ivankova and Stick (2007), sought to identify factors contributing to students’ persistence in the

University of Nebraska-Lincoln Distributed Doctoral Program in Educational Leadership in

Higher Education. As in the previously critiqued study, the title is well written and includes the

words “mixed methods”. An abstract was included with this article and the first sentence gave

the purpose of the study, and informed the audience that a mixed methods sequential explanatory

design approach was being used.

The introductory section of the article included the problem, review of literature,

theoretical framework, and provided background and demographical data on the doctoral

program. The framework included three major theories of student persistence from Tinto’s

Student Integration Theory, Bean’s Student Attrition Model, and Kember’s Model of Dropout

from Distance Education Courses. Unlike the previous article, the purpose statement in this

article is not found until page three but appropriately reads, “The purpose of this mixed methods

sequential explanatory study was to identify factors contributing to students’ persistence in the

ELHE program by obtaining quantitative results from a survey of 278 current and former

students and then following up with four purposefully selected individuals to explore those

results in more depth through a qualitative case study analysis.” Though scholars referenced the

term “research question,” the research question for this study was not explicitly stated but was

implied.

The methods section of this article was very detailed and included study design and target

population. The reading audience was reminded that a sequential exploratory design was being

employed, and further went on to define what the design entailed. Scholars also included within
CRITIQUES OF MIXED METHODS STUDIES 7

the article a diagram that clearly explained the process for this type of design, which was an

added bonus. The article then broke down the data collection and data analysis for the

quantitative and qualitative phases of the study. The results section was very detailed, and again

was separated by quantitative and qualitative information and analysis. Scholars used charts to

depict what was explained in the text and included rich descriptions from study participants in

the qualitative result reports.

Similar to the previous article, there was no section titled “conclusion”. However,

scholars thoroughly discussed the study’s findings and implications and recommendations for

doctoral programs that are delivered through distance education. In the closing section scholars

pointed to the relevancy of the study and revisited the rationale for conducting a mixed

methodology approach that addressed a gap in the literature. Contrary to the previously critiqued

study there was no section labeled “limitations”, though study limitations were discussed in a

very brief and concise paragraph at the end of the study. The study added value to the body of

knowledge regarding student persistence in doctoral programs, with a focus on distance learning.
CRITIQUES OF MIXED METHODS STUDIES 8

Quantitative Data Collection

Quantitative Data Analysis

Case Selection
Interview Protocol
Development
CC

QUALITATIVE
Data Collection

QUALITATIVE
Data Analysis

Integration of
Quantitative &
Qualitative Results

Explanatory Sequential Design

CRITIQUES OF MIXED METHODS STUDIES 9

A dissertation entitled, “Impact of Professional Nursing Tutoring on National Council

Licensure Examination Success” (Mondeik, 2014) examined the impact of professional

nursing tutors on National Council Licensure Examination (NCLEX) success. Unfortunately the

title of the study did not allow the writer to know that a mixed methodology approach was being

utilized. According to Creswell and Plano Clark (2018), a good mixed methods title includes the

words “mixed methods”.

The problem statement clearly identified the ongoing nursing shortage as an issue that is

indirectly affected by NCLEX pass rates. Those that are unsuccessful on the licensure exam,

though they’ve completed necessary course work are unable to enter the workforce and close the

shortage gap. Mondeik (2014) provided adequate data to support the problem statement. The

purpose of the study was clearly identified and read, “The purpose of this mixed-methods study

was to examine the impact of professional nursing tutors on NCLEX pass rates.” To further

support the purpose of the study a section entitled, “Contextual Orientation” addressed issues

colleges were facing that may hinder NCLEX success rates due to restricted funding revenues.

This writer obtained a clear understanding of Mondeik’s statement of the problem and the

purpose of conducting the study.

Following the purpose statement were two overarching research questions that each

contained sub-questions and were written as follows:

1. Did the students who met the professional nursing tutor have a higher NCLEX pass

rate that the students who did not meet with the professional nursing tutor?

a. Is there a relationship between the number of hours spent with the professional

tutor and having a higher success rate on the NCLEX?

CRITIQUES OF MIXED METHODS STUDIES 10

b. Is there a relationship in NCLEX pass rates between students who participated in

group tutoring versus students who selected individual tutoring?

2. What are the perceptions of students related to the effectiveness of the professional

nursing tutor on their NCLEX success?

a. What strategies did the professional nursing tutor use that students perceived

beneficial to their NCLEX success?

b. What suggestions did students share to improve the tutoring experience?

c. What reasons did students share for not attending tutoring?

The first research question and sub-questions are looking at correlations, which make them

quantitative questions. The second research question and sub-questions are qualitative. These

research questions were selected in an effort to better understand the impact of professionally

hired tutors on NCLEX success rates for a college which had the resources to implement this

strategy.

Within the introduction section of this dissertation was the theoretical framework, which

was based on Maki’s (2010) effective assessment. According to Maki, as cited by Mondeik

(2014), there must be continuous systematic assessment to evaluate the effectiveness of

programs and implemented strategies. The four steps of Maki’s assessment cycle (2010) were

included and discussed in detail so that the reader gained an adequate understanding of the

framework

Chapter two of the dissertation was the review of literature. It consisted of approximately

20 pages which were dedicated to showing the connection between critical program assessment

and the national nursing shortage. Literature specific to the importance of passing NCLEX was

highlighted as this is a challenge faced by nursing programs across the country. The review of
CRITIQUES OF MIXED METHODS STUDIES 11

literature also clearly identified a gap in the literature, and the need to validate the benefits of

new teaching strategies and best practices for NCLEX success.

Chapter three of the dissertation was the scholar’s methodology section. The researcher

did not specify the type of mixed-methodology being utilized, and plainly stated that the

investigation employed a quantitative and qualitative design. Mondeik (2014), revisited the

purpose of the study, the two primary research questions, background of the study, research

design, instrumentation, data sources, data collection procedures, data analysis, human subjects,

and the strengths and limitations of the design. Unlike the previous studies, Mondeik (2014) did

not separate out the quantitative data collection and analysis procedures from the qualitative

portions. There was limited information given in this section of the dissertation. The reading

audience would have benefited from more details regarding data collection and analysis from

each portion of the study (qualitative and quantitative) and how the data was merged.

Chapter four of the dissertation was the results of the study. This chapter included a

reminder of the research questions being investigated. There was also a section entitled, “review

of method”, which again failed to give the specific type of mixed-methodology being employed.

A section entitled, “descriptive data” discussed the study sample as being 38 nursing graduates

from the Associate Degree Nursing Program at Northcentral Technical College in Wisconsin,

and spoke about the demographical data obtained. Results regarding variables was discussed in

detail with some of the information being depicted in text and tables. A brief section entitled,

“key findings” spoke to the significant findings in the study. Of note, some of the information

that was not included in the chapter on methodology was found in the results section, such as

some of the data collection and analysis processes that were utilized (i.e. use of Fisher’s exact

test.
CRITIQUES OF MIXED METHODS STUDIES 12

Unlike the journal articles previously critiqued, chapter five was the conclusion chapter.

It included an overview of the study, key findings, conclusions, recommendations and a

summary which highlighted key points of the study. This dissertation has great value, as it

answered questions in which there was a gap in the literature. The study has implications for

future studies that can be replicated in similar or different environments to further increase the

body of knowledge as it relates to NCLEX success. Though the study did not specify which type

of mixed-methodology was utilized, it is concluded that a sequential explanatory design was

utilized. The author cited Creswell (2014) to justify the mixed-methodology approach, stating

that the qualitative data obtained would further explain the quantitative data initially obtained. A

diagram for this type of study can be located at the end of the previous journal article, as both

employed the same mixed-methodology.

The final study critiqued was a dissertation entitled, “The Nursing Student Experience:

Student Perceptions Concerning Factors Leading to Success in an Associate of Science in

Nursing Program at Montana State University-Northern” (Pappas, 2006). As in two of the

three previous studies critiqued, this study did not use the term “mixed methods” in the title,

making it impossible to know what type of methodology was being utilized at a glance. Within

the abstract it stated, “The purpose of this mixed methods study was to discover and understand

the program and personal factors that current students, graduates, and previous nonpersistent

students reported as contributing to leaving the associate of science in nursing program.” The

abstract also informed this writer as to how many participants were in the study, the years in

which data was obtained from, major themes identified from the study, and provided

recommendations as a result of the study’s findings.

CRITIQUES OF MIXED METHODS STUDIES 13

Within the introduction the problem statement was clearly explained. Students were

enrolling in nursing programs but not matriculating. This reader gained a sense of urgency

regarding the need to increase the graduation rate amongst nursing students in an effort to lessen

the negative impact of an impending and current nursing shortage in Montana and across the

country. The purpose statement was clearly written and was previously mentioned in the

abstract portion of the dissertation. Research questions immediately followed the purpose

statement and were written as follows:

1. Which program factors reported by students, graduates, and faculty have helped

students be successful in the ASN program at MSU-Northern?

2. Which personal factors, reported by students, graduates, and faculty, have helped

students be successful in the ASN program at MSU-Northern?

3. What program factors have led nursing students to perceive the program negatively?

4. What personal factors have contributed to nursing students leaving the ASN program

at MSU-Northern?

5. What recommendations do current students and graduates have for increasing

students’ chances of success in the ASN program at MSU-Northern?

6. What recommendations do previous nonpersistent students in the nursing program

have for increasing students’ chances of success in the ASN program at MSU-

Northern? did the author develop qual, quant or mixed methods questions? How did they report Commented [1]: did the author develop qual, quant or
mixed methods questions? How did they report both
both qual and quant data/findings to answer their research questions? qual and quant data/findings to answer their research
questions?

Qualitative and quantitative questions were used to obtain and report information.

Though the overarching research questions appear to be qualitative, the researcher did

utilize a likert scale to survey students regarding 24 students that spoke to the participants
CRITIQUES OF MIXED METHODS STUDIES 14

perceptions. An example of a statement is as follows: “my interactions with the nursing

faculty have (had) a positive effect on my professional growth.” The researcher in this

study clearly defined within the methodology the phases of the study. Initially,

quantitative data was obtained from a questionnaire. The purpose of the survey

questionnaire was to obtain information concerning characteristics of students who

completed the nursing program and compare that data to characteristics of students who

did not complete the program.

The questionnaire also served as a method of retrieving information concerning the

research questions and a means of asking for volunteers who had graduated, who had left

the program, and who were currently enrolled in the program to participate in the

qualitative phase of the study by expressing willingness to be interviewed. Upon

completion and return of the questionnaire, a qualitative approach was utilized to obtain

richer, thicker information concerning the characteristics of students who completed the

nursing program and those who did not complete the program. Interviews were

conducted to give the researcher a better understanding of student experiences.

The conceptual framework for this dissertation was very detailed and more complex

than the previously critiqued studies. This dissertation study was based on several

theories related to student retention. Theorists included Tinto’s Theory of Student

Integration, Bean’s Theory of Undergraduate Student Persistence, Bean and Metzner’s

Model of Student Attrition, and Bean and Eaton’s Psychological Model of Student

Retention. The researcher included a visual depiction of the conceptual framework

which gave this writer a better understanding as to how the various theorists factored into

this particular dissertation study, which was an added bonus in this writer’s mind.
CRITIQUES OF MIXED METHODS STUDIES 15

The review of literature (chapter 2) was very thorough and included a variety of

variables related to student attrition in higher education, some of which included:

external student responsibilities, support of friends and family members, academic

preparation, psychosocial factors, financial resources, goal commitment, residential

status, and academic and social integration. The methods chapter included the rationale

for the mixed methods approach. It also broke down the quantitative and qualitative

phases of the study which included participants, design and procedure, instrumentation,

trustworthiness, and data collection and analysis. Unlike the previous dissertation study,

this separation of quantitative from qualitative made the study easier to read and

understand. As in the previous methodology, the specific type of methodology was not

specified.

The results chapter was very detailed and broke down each variable with findings.

Again, the quantitative and qualitative results were reported separately. The results were

then appropriately integrated. The quantitative and qualitative recommendations were

finalized in a summary that closed out the chapter. The last chapter, conclusions

reviewed the introduction and gave an overview of the methods and data collection and

data analysis. Each of the research questions were answered. A comparison to literature

and recommendations with suggestions for further research were included.

Though this study failed to mention the types of mixed methodology utilized it

thoroughly described the process, letting the reader know that there were two distinct

phases, with the quantitative phase being completed first, then the qualitative phase was

described. As in the previous dissertation study, this study also employed an explanatory

mixed-methods design. This study was of great value as it not only added to the existing
CRITIQUES OF MIXED METHODS STUDIES 16

body of knowledge, but contributed unique findings and recommendations regarding

factors that lead to success in a nursing program.

In conclusion, it was interesting to critique the journal articles and dissertations.

Though most of the studies included most of the properties of a mixed methods approach

only one used the term mixed methods in the title, and only one of the studies explicitly

stated which type of mixed methods design was utilized. Though all had a lot of the

qualities needed for a mixed methods study, only one had all of the components as listed

in the mixed methods survey provided.

CRITIQUES OF MIXED METHODS STUDIES 17

References
Creswell, J. W., Plano Clark, V. L. (2018). Designing and conducting mixed methods

research (3rd ed.). Thousand Oaks, CA: Sage Publications.

Gipson, J., Mitchell, D., & McLean, C. (2017). An investigation of high-achieving African-

american students attending community colleges: A mixed methods research study.

Community College Journal of Research and Practice. Retrieved from

http://dx.doi.org/10.1080/10668926.2017.1299652

Ivankova, N. V., & Stick, S. L. (2007). Students’ persistence in a distributed doctoral program in

educational leadership in higher education: A mixed methods study. Research in Higher

Education, 48(1), 93-135.

Mondeik, S. L. (2014). Impact of professional nursing tutors on national council licensure

examination success. (Doctoral dissertation). Retrieved from

https://search-proquest-com.proxy-

ms.researchport.umd.edu/pqdtglobal/docview/1512620146/fulltextPDF/4C7E588FE2954

D5CPQ/1?accountid=12557
CRITIQUES OF MIXED METHODS STUDIES 18

Pappas, M. W. (2006). The nursing student experience: Student perceptions concerning factors

leading to success in an associate of science in nursing program at Montana state

university-northern. (Doctoral dissertation). Retrieved from

https://search-proquest-com.proxy-

ms.researchport.umd.edu/pqdtglobal/docview/305264250/fulltextPDF/7F38343C74F04F

6CPQ/1?accountid=12557