Author: Alec Fearon

Posted on

Unit of analysis

An earlier posting identified four basic types of case study design. Two of these have embedded units of analysis: what does this mean? Yin (2009, p.50) explains with the example of a public programme (the single case) which funds a large number of projects (the embedded units). See also his more comprehensive guidance (pp.29-33) about defining what the case is and choosing the unit of analysis.

Yin (2010, pp.82-87) offers another viewpoint in a discussion about ‘data collection units’ and their relationship to the main topic of a study. He notes that the data collection unit is usually the unit of analysis, though there can be complicated situations when it is not. He suggests that “most qualitative studies have more than one level of data collection unit” and that these “are likely to fall within a nested arrangement: a broader level (eg a field setting) that contains or embeds a narrower level (eg a participant in the setting)”.His Exhibit 4.1 lists examples of a topic and two levels of data collection unit.

Reference:
YIN, R. K. 2009. Case study research: Design and methods, Sage publications, INC.
YIN, R. K. 2010. Qualitative research from start to finish, The Guilford Press.

Posted on

Case study designs

Robert Yin identifies four basic types of case study design, illustrated below.  In all designs it is necessary to consider and perhaps analyse the context of the case.

case_study_designsBasic types of design for case studies (Yin, 2009, p.46)

Reference:
YIN, R. K. 2009. Case study research: Design and methods, Sage publications, INC.

Posted on

Evaluation Coding

Johnny Saldana (2009, p.3) suggests that a code in qualitative research “is most often a word or short phrase that … assigns a summative, salient, essence-capturing … attribute for a portion of language-based or visual data. The data can consist of interview transcripts, … field notes, journals, documents, literature, artifacts, photographs, video, websites, email correspondence and so on. The portion of data to be coded … can range in magnitude from a single word to … an entire page of text …“.

In the model below, Saldana (2009, Fig 1.1) shows that codes can be categorised and that categories can be mapped into themes or concepts and fitted within theory. As drawn by Saldana, the model shows an inductive process where theory is developed from code. A deductive process is equally valid, whereby code is developed from theory.

codes-to-theory-model_saldana-2009_fig-1-1

Johnny Saldana (2009, p.97) draws on Rallis & Rossman to describe evaluation coding as “the application of non-quantitative codes onto qualitative data that assign judgements about the merit and worth of programs or policy (Rallis & Rossman, 2003, p.492)“.

He draws on Patton to say that program evaluation is “the systematic collection of information about the activities, characteristics and outcomes of programs to make judgements about the program, improve program effectiveness and/or inform decisions about future programming. Policies, organisations and personnel can also be evaluated (Patton, 2002, p.10)“.

“To Rallis & Rossman, evaluation data describe, compare and predict. Description focuses on patterned observations or participant responses of attributes and details that assess quality. Comparison explores how the program measures up to a standrd or ideal. Prediction provides recommendations for change, if needed, and how those changes might be implemented.” (Saldana, 2009, p.97) (my emphasis)

Saldana says (p.98) that evaluation coding is “appropriate for policy, critical, action, organizational, and (of course) evaluation studies, particulary across multiple sites and extended periods of time“. He suggests that other coding methods “can be applied to or supplement evaluation coding” and lists magnitude coding, descriptive coding, values coding and grounded theory coding methods for this purpose. He draws on Pitman & Maxwell to observe that evaluation coding “is also customised for specific studies since the coding system must also reflect the questions that initiated and structured the evaluation in the first place (Pitman & Maxwell, 1992, p.765)“.

Finally, Saldana (p.101) quotes Stake to note that “all evaluation studies are case studies (Stake, 1995, p.95)”

References:
PATTON, M. Q. 2002. Qualitative research and evaluation methods, Sage Publications, Inc.
PITMAN, M. A. & MAXWELL, J. A. 1992. Qualitative approaches to evaluation: Models and methods. The handbook of qualitative research in education, 729, 770.
RALLIS, S.E and GROSSMAN, G.B. 2003. Mixed methods in evaluation contexts: a pragmatic framework. Handbook of Mixed methods in social & behavioral context, Eds. A. Tashakori a Ch. Tedllie. London: Sage.
SALDAÑA, J. 2009. The coding manual for qualitative researchers, Sage Publications Ltd.
YIN, R. K. 2009. Case study research: Design and methods, Sage publications, INC.
STAKE, R. E. 1995. The art of case study research, Sage Publications, Inc.

Posted on

Evaluation research

If you want to understand ‘evaluation research’, a good way to begin is by reading Ranjit Kumar (2010, ch.18). His model for the concept of evaluation looks like this:

evaluation_concept1

The concept of evaluation (Kumar, 2010, p.325)

Kumar suggests that evaluation can be approached from two perspectives (p.328): “the focus of the evaluation; and the philosophical base that underpins an evaluation“. The types of evaluation that arise from each perspective are shown in the figure below. He emphasises that the perspectives are not mutually exclusive, explaining this with the example of a study which determines the impact of a programme by asking what its clients perceive its effects to have been on them. This study could be classified as an impact / outcome evaluation from the ‘focus of the evalution’ perspective or, equally, as a client-centred evaluation from the ‘philosophical’ perspective.

evaluation_perspectivesPerspectives in the classification of evaluation studies (Kumar, 2010, p.329)

From the perspective of the focus of the evaluation, there are many designs or methods of evaluation. For example in an impact assessment evaluation Kumar identifies (p.338) the following commonly used designs:

  • after-only design;
  • before-and-after design;
  • experimental control design;
  • comparative study design;
  • reflexive control design;
  • interrupted time-series design (see model below);
  • replicated cross-sectional design

interrupted_time_series_designInterrupted time-series design (Kumar, 2010, p.340)

From a philosophical perspective, Kumar argues (p.342) that “there are no specific models or methods of evaluation. You use the same methods and models [as for the other perspective] but the required information is gathered from different people or aspects …”

Kumar provides an informative overview. For greater detail and a better appreciation of these perspectives he recommends Stufflebeam & Shinkfield (1985).

Reference:
KUMAR, R. 2010. Research methodology: A step-by-step guide for beginners, Sage Publications Ltd.
STUFFLEBEAM, D. L. & SHINKFIELD, A. J. 1985. Systematic evaluation: A self-instructional guide to theory and practice, Kluwer-Nijhoff Boston.

Posted on

Writing your dissertation – the Research Method

The rationale below is based on the circumstances of a particular research project but I hope it illustrates an approach that can be extended to other circumstances. It is in the form of guidance to an MSc student.

You might consider basing your Research Method chapter on ideas explained by Kumar (2010), Gray (2009) and Yin (2009).

Kumar (2010, p9) suggests a good way to begin describing your type of research. He says that research can be looked at from three perspectives: (1) application of the findings of the research study; (2) objectives of the study; and (3) mode of enquiry used in conducting the enquiry. Gray (2009, p35) says that “exploratory studies seek to explore what is happening and to ask questions about it“. He suggests that they can be conducted, amongst other things, by a search of the literature and by talking to experts in the field. After citing these references you might therefore say that your research was:

  1. Applied research in that “the research techniques, procedures and methods …. are applied to the collection of information about various aspects of a situation, issue, problem or phenomenon so that the information gathered can be used in other ways – such as for …. the enhancement of understanding of a phenomenon“.
  2. Exploratory in that it explores the proposition that ideas from lean thinking can enhance the organisation’s approach to programme and project management.
  3. Based on a qualitative mode of enquiry.

Gray (pp14,15) clarifies the differences between deductive and inductive reasoning. Your strategy of enquiry was based on inductive rather than deductive reasoning, that is you used the data to develop a theory rather than starting with a defined theory and seeking specific data to test it.

You might now take advice from Yin (2009, pp 5-10) who identifies five research methods: experiment, survey, archival analysis, history and case study. He suggests that three conditions determine which method is most appropriate:

  • the type of research question posed (how, why, who, what, where, how many, how much);
  • the extent of control an investigator has over actual behavioural events;
  • the degree of focus on contemporary as opposed to historical events.

In your research I believe it is fair to say that the form of research question is ‘how?’ or ‘why?’, you had no control of behavioural events and the focus was on contemporary events. In this set of conditions Yin suggests that a case study is appropriate. He goes on to define a case study (Yin, 2009, p18) as “an empirical enquiry that:

  • investigates a contemporary phenomenon in depth and within its real-life context, especially when
  • the boundaries between phenomenon and context are not clearly evident.

This definition seems to fit your circumstances. You should therefore follow his advice (Yin, 2009, p27) that, for case studies, “five components of a research design are especially important:

  1. a study’s questions;
  2. its propositions, if any;
  3. its unit(s) of analysis;
  4. the logic linking the data to the propositions; and
  5. the criteria for interpreting the findings.“

1. Your research question is “How can ideas from lean thinking …. international development programmes”.

2. Yin (2009, p28) says that an exploratory study does not need to have any propositions. But it should have a purpose, and this should be stated.

3. The unit of analysis for your case study appears to be the organisation’s programme and project management methodology.

4. Yin (2009, p35) identifies five analytic techniques as ways of linking data to propositions or purpose: pattern matching, explanation building, time-series analysis. logic models, and cross-case analysis. It seems to me that you have used explanation building.

5. Yin (2009, pp 160-161) proposes a number of prlnciples underlying good research, including that your analysis should:

  • show that you attended to all the evidence;
  • address all major rival interpretations;
  • address the most significant aspect of your case study.

You now need to explain your choice of case study design. Yin (2009, p46) explains the options. Did you adopt a single case design? If so, the case would be the organisation as a whole embracing its policy, process and practice for programme and project management and the evaluations at organisational level of their fitness for purpose.

Finally you should outline your data collection methods. I believe these were:

  • To use unobtrusive data from primary sources in the form of contemporaneous documents and the content of official websites and to supplement this with data from secondary sources. Many of the documents were in the public domain, some were obtained from the organisation’s intranet through the agency of your informant.
  • An in-depth interview. This was primarily to clarify and expand the scope of your understanding of the data and to a lesser extent to provide you with some primary data. I suggest the person you interviewed was an ‘informant’ rather than an ‘interviewee’. See this posting for an explanation of what this means.

References

GRAY, D. E. 2009. Doing research in the real world, Sage Publications Ltd.

KUMAR, R. 2010. Research methodology: A step-by-step guide for beginners, Sage Publications Ltd.

YIN, R. K. 2009. Case study research: Design and methods, Sage publications, INC.

Posted on

In-depth interview

Yin (2009, p107) offers useful insight about the use of an in-depth interview during a case study.

“You can ask key  respondents about the facts of a matter as well as their opinions about events.  In some situations, you may even ask the interviewee to propose her or his own  insights into certain occurrences and may use such propositions as the basis  for further inquiry. The “interview” may therefore take place over an extended  period of time, not just a single sitting. The interviewee also can suggest other  persons for you to interview, as well as other sources of evidence.The more that an interviewee assists in this manner, the more that the role  may be considered one of an “informant” rather than a respondent. Key informants are often critical to the success of a case study. Such persons provide the  case study investigator with insights into a matter and also can initiate access to  corroboratory or contrary sources of evidence. …… Of course, you need to be cautious about becoming overly dependent on a key informant, especially because of the interpersonal influence – frequently subtle – that the informant may have over you. A reasonable way of dealing with this pitfall again is to rely on other sources of evidence to corroborate any insight by such informants and to search for contrary evidence as carefully as possible.”

Reference:
YIN, R. K. 2009. Case study research: Design and methods, Sage publications, INC.