Methodology Help and Advice

Dear Dr. Phoebe,

I conducted my first interview today – and it went badly. My project assesses the 2008 NHS reform and involves interviewing many of the key decision-makers from that period. I started with former Secretary of State for Health, Alistair Carrington, who agreed to meet me in the House of Commons. The interview didn’t go as expected, and now that the word has gotten out, no one wants to talk to me. Maybe I took the wrong approach. I’ve attached an excerpt from the transcript – maybe you could look at and tell me where it all went wrong?

Desperate in Docklands,

Sam Bookantiqua, E16

# Interview Transcript
Sam Bookantiqua: SB
Alistair Carrington: AC

SB: Very nice to meet you Mr. Carrington. I’d like to start the interview with a couple of demographic questions. Is that ok?
AC: Yes.
SB: Alistair Carrington, what is your name?
AC: Alistair Carrington.
SB: What’s your gender?
AC: I would have hoped this was obvious but… why don’t you try to guess.

Continue reading →

Dear Dr. Phoebe,

I have hired two supposed “expert” coders to read a sample of newspaper articles and code each paragraph according to a coding scheme that I have carefully developed as part of my project in media studies. But the coder interreliability (reported by NVivo) is only .15. Should I be concerned? 

Vernon Szarhazy, SW12

Dear Vernon,

I’m afraid you have a problem with the reliability of your coders, your coding scheme, or both. What NVivo has reported is a kappa score, ranging from 0 (no agreement) to 1.0 (perfect agreement). Kappa is similar to a correlation or percentage agreement, but also takes into account chance agreement (which will small anyway if you have lots of possible codes). Landis and Koch (1977, “The measurement of observer agreement for categorical data”, Biometrics 33:159–174) have a much overused table interpreting 0.15 as slight agreement, but let’s just say if that were your grade for MY429 Qualitative Content Analysis, you’d be resitting the course. You need to take a hard look at your coding scheme and ask whether it can be simplified. Training your coders better, or choosing more consistent coders, would also be a good idea.

Sincerely,

Dr. Phoebe

Dear Dr. Phoebe,

I’ve just had an epiphany after a month of wracking my brains for a research project that uses the cool ethnographic methods I’ve learned from MY426. The idea came after my new girlfriend forced me to watch a night of Johnny Depp movies, starting with something about a drunken possibly gay pirate that looks a bit like Michael Jackson. But that was followed by a Donnie Brasco, the film that gave me my research idea using participant observation methods. In the film – based on a true story – Depp plays an FBI undercover agent who infiltrates the Bonanno crime Family in New York. He finds the gate-keepers, he records lots of information, he becomes one of the most trusted hit-men of the family and, eventually, he helps the FBI to arrest all the Mafia big bosses. The movie is great because it shows how Donnie is very close to ‘turn native’ at some point (like to become a true gangster); it also shows that gaining access, building trust, and taking notes undercover is not easy of course, but still feasible.

So my idea is to go undercover, like Donnie Brasco, to investigate how hierarchical relationships work out within the Sicilian Mafia. For fieldwork, I’d go to Palermo and take up work in a restaurant frequented by the Mafia. There I will become familiar with the crime bosses, and also secretly record all that they say using microphones hidden under my shirt. Building upon that, I will try to infiltrate higher strata of the Mafia and will study their inner power relationships. Like Brasco, I’ll deeply penetrate their organization, even if this means having participate and not simply observe their activities. But my girlfriend is worried sick, saying I might have to commit crimes or sleep with some mafioso’s daughter to gain their trust. I haven’t told my supervisor about any of this, since he’d probably just worry. I think I’ll be fine, since two summers ago I went on a backpacking tour through Sicilily and just loved it. The data I get will be original and incredible, and the project really seems exciting too. Is this a sound and valid ethnographic research design?

Nick Picklewaite-Busby, SW19

Dear Nick,

Fuggedaboudit.

Continue reading →