Tag Archives: phd thesis

When coding doesn’t work, or doesn’t make sense: Synoptic units in qualitative data analysis

You can download a full pdf of this blog post including the three examples here. Please feel free to share with others, though preferably direct them to this page to download it!

 

How do you analyse qualitative data? You code it, right? Not always. And even if you do, chances are coding has only taken you a few steps in the long journey to your most important analytical insights.

I’m not dismissing coding altogether. I’ve done it many times and blogged about it, and expect I will code again. But there are times when coding doesn’t work, or when it doesn’t make sense to code at all. Problems with coding are increasingly being recognised (see this paper by St Pierre and Jackson 2014).

I am often asked: if not coding, then what? This blog post offers a concrete answer to that in terms of a logic and principles, and the full pdf gives examples from three studies.

Whatever you do in qualitative analysis is fine, as long as you’re finding it helpful. I’m far more worried about reaching new insights, seeing new possible meanings, making new connections, exploring new juxtapositions, hearing silences I’d missed in the noise of busy-work etc than I am about following rules or procedures, or methodological dogma.

I’m not the only one saying this. Pat Thomson wrote beautifully about how we can feel compelled into ‘technique-led’ analysis, avoiding anything that might feel ‘dodgy’. Her advocacy for ‘data play’ brings us into the deliciously messy and murky realms where standard techniques might go out of the window: she suggests random associations, redactions, scatter gun, and side by side approaches.

 

An approach where you are a strength not a hazard

The best qualitative analyses are the ones where the unique qualities, interests, insights, hunches, understandings, and creativity of the analyst come to the fore. Yes, that’s right: it’s all about what humans can do and what a robot or algorithm can’t. And yes, it’s about what you can do that perhaps no-one else can.

Sound extreme? I’m not throwing all ideas of rigour out of the window. In fact, the first example below shows how the approach I’m advocating can work really well in a team scenario where we seek confirmation among analysts (akin to inter-rater reliability). I’m not saying ‘anything goes’. I am saying: let’s seek the analysis where the best of us shines through, and where the output isn’t just what is in the data, but reflects an interaction between us and the data – where that ‘us’ is a very human, subjective, insightful one. Otherwise we are not analysing, we are just reporting. My video on ‘the, any or an analysis’ says more about this.

You can also check out an #openaccess paper I wrote with Prachi Srivastava that highlights reflexivity in analysis by asking: (1) What are the data telling me? (2) What do I want to know? And (3) What is the changing relationship between 1 and 2? [There is a video about this paper too]

The process I am about to describe is one in which the analysts is not cast out in the search for objectivity. We work with ‘things’ that increasingly reflect interaction between data and the analyst, not the data itself.

 

An alternative to coding

The approach I’ve ended up using many times is outlined below. I don’t call it a technique because it can’t be mechanically applied from one study to another. It is more a logic that follows a series of principles and implies a progressive flow in analysis.

The essence is this:

  1. Get into the data – systematically and playfully (in the way that Pat Thomson means).
  2. Systematically construct synoptic units – extractive summaries of how certain bits of data relate to something you’re interested in. These are not selections of bits of data, but written in your own words. (You can keep track of juicy quotations or vignettes you might want to use later, but the point is this is your writing here).
  3. Work with the synoptic units. Now instead of being faced with all the raw data, you’ve got these lovely new blocks to work and play seriously with. You could:
    1. Look for patterns – commonalities, contrasts, connections
    2. Juxtapose what seems to be odd, different, uncomfortable
    3. Look again for silences
    4. Look for a prior concepts or theoretical ideas
    5. Use a priori concepts or theoretical ideas to see similarity where on the surface things look different, to see difference where on the surface things look the same, or to see significance where on the surface things seem unimportant
    6. Ask ‘What do these units tell me? What do I want to know?’
    7. Make a mess and defamiliarize yourself by looking again in a different order, with a different question in mind etc.
  4. Do more data play and keep producing artefacts as you go. This might be
    1. Freewriting after a session with the synoptic units
    2. Concept mapping key points and their relationships
    3. An outline view of an argument (eg. using PowerPoint)
    4. Anything that you find helpful!

 

In some cases you might create another layer of synoptic units to work at a greater analytical distance from the data. One of the examples below illustrates this.

The key is that we enable ourselves to reach new insights not by letting go of the data completely, but by creating things to work with that reflect both the data and our insights, determinations of relevance etc. We can be systematic as we go through all the data in producing the synoptic units. We remain rigourous in our ‘intellectual hygiene’ (confronting what doesn’t fit, what is less clear, our analytical doubts etc) . We do not close off on opportunities for serious data play – rather we expand them.

If you’d like to read more, including three examples from real, published research, download the full pdf.

Advertisements

How to make sure people care about your research

No-one cares about your research. Particularly if it’s your PhD (or any other kind of doctorate). In fact if someone knows it’s the latter, or you mention it, they probably care less, or at least have alarm bells ringing that you’re about to launch into a prolonged account of your scholarship woes, the fact your supervisor hasn’t replied to any emails for 17 hours now, the horrible ethics committee, and the impossibility of writing only 100,000 words when it’s taken you 7 years and you’ve just got so much to say…

Even more of concern is the journal reviewer, or assessor of your grant proposal who is put off and frustrated before they’ve finished reading the first paragraph.

Fear not, for help is at hand! Fortunately, there is a really easy and effective way to avoid all these problems. Admittedly, this assumes your research does actually matter in some way, in the sense that it connects with something wider and non-trivial.

My solution will cost you nothing: no hard currency, no bitcoins, and no sleepless nights. Probably not even any extra words. In fact you may end up telling and selling the story of your research in fewer words than before! All it takes is a bit of trust, and a few minutes of your time.

My solution is this: when introducing your research, use a sequence that follows a ‘so’ logic rather than a ‘why?’ logic. This may well involve reversing the order of your ideas and sentences. If so, rejoice! – because this means you’ve already had all the right ideas, made all the right connections. You just need to turn it all upside down.

So what on earth is a ‘so’ logic, or a ‘why?’ logic, and why do these matter?

A ‘why’ logic is based on a sequence of sentences where each sentence is followed by one that explains the first. Example:

My research is about improving generic skills of university graduates.

This is important because employers increasingly look for generic skills in recruiting new staff, and repeatedly report shortcomings among graduates.

This matters because generic skills are known to be crucial to successful business innovation.

This looks great, right? It’s clear, follows a nice logical order, and explains to the reader why your research is important. I’ll admit, it’s not bad. Just I think it could be better. What’s really going on in the sequence above is an unwritten conversation with the reader. Let’s look at it again, this time with the silent responses inserted:

My research is about improving generic skills of university graduates.

[So what?]

This is important because employers increasingly look for generic skills in recruiting new staff, and repeatedly report shortcomings among graduates.

[Yeah. And? Why should I care about that?]

This matters because generic skills are known to be crucial to successful business innovation.

[Oh! Now I get it!]

Look at it from the reader’s point of view. You first sentence left them unconvinced, and probably rang all the alarm bells of dread, foreboding the terrors I outlined at the beginning of this post. Only after pushing you twice for more information, are they rewarded with something that they actually ‘get’, and might even care about. To them your research, in only three sentences, has been an uphill slog, full of doubt, experienced as some kind of puzzle that leaves them guessing. After each sentence they are left asking themselves: “why?”. This is the reason I call this a ‘why?’ logic.

But it doesn’t have to be this way. We can swap ‘why?’ for ‘so’. And we barely have to change a word. In fact we delete quite a few!

Generic skills are known to be crucial to successful business innovation.

Employers increasingly look for generic skills in recruiting new staff, and repeatedly report shortcomings among graduates.

My research is about improving generic skills of university graduates.

In this logic, you start with the idea that the reader really ‘got’ in the first scenario. The thing that matters most universally, directly and immediately to your readers. The kind of thing that they will accept as obvious, perhaps even unquestionable. There’s nothing wrong with showing a reader that you are both on the same wavelength. Take a shared assumption about something that you know to be a common concern. Something you don’t have to convince them to care about. Exploit what’s already there between you!

Then simply follow up with a sentence that leads from that towards your research, in a gradually narrowing down. What’s happening this time, is something more like this:

Generic skills are known to be crucial to successful business innovation.

[Absolutely! You sound like a sensible sort of person who knows what I care about. I’m curious. Tell me more].

Employers increasingly look for generic skills in recruiting new staff, and repeatedly report shortcomings among graduates.

[Yes. That makes sense.]

My research is about improving generic skills of university graduates.

[Seriously?! Wow! That’s wonderful! It’s just what we need. And it sounds very focused too. Tell me all about it in intricate detail!]

At each step you carry the reader with you, and one sentence follows on from the next exploiting this. Sentence 1 [brilliant!] so…. sentence 2 [amazeballs!] so… sentence 3 [no way! Where’s that Novel prize nomination form?]

That’s it. It may take you more than 3 sentences (hopefully not too many more, though).

Give it a try. I dare you. What have you got to lose?

Acknowledgement

I would like to acknowledge the influence of Martyn Hammersley’s framework for reading ethnographic research (see my video and podcast), Pat Thomson and Barbara Kamler’s miraculous ‘tiny texts’ approach to writing abstracts, the group of UTS Doctor of Education students based in Hong Kong, and Lee Williamson from UTS’ Research Office. Without you all this would never have come to fruition.come to fruition.

Say goodbye to your Nobel Prize, and get a doctorate instead (or: frame your doctoral study around doctoralness)

This post highlights a few key points from a great paper (freely available) by Mullins and Kiley. The title sets the tone: ‘It’s a PhD, not a Nobel Prize’: how experienced examiners assess research theses. (Studies in Higher Education 27(4), 369-386, doi 10.1080/0307507022000011507)

I’m putting this post up now because it confirms some elements of what I’ve already written about parsimony, keeping your reader/examiner on track, structuring a thesis etc. And it will lay out some terms and foundations for posts that are coming soon – about literature reviewing, and qualitative research design.

Tip for readers of this post: try putting the text into a wordcloud and see what comes up!

First up: avoid irritating your reader

Sounds obvious, but examiners report experiencing this – eg. through typos, poor layout, unclear structure, lack of signposting, indulgence / overly verbose text. Johnston (1997) in Mullins and Kiley, writes:

Examiners require all of the normal forms of assistance which should be provided to any reader. They appreciate work which is logically presented, focused, succinct, summarised and in which signposts are used to help readers to understand the path they are taking through the work … One of the problems with work that is poorly presented is that the examiner tends to lose confidence in the candidate and can become suspicious that there are deeper problems of inadequate and rushed conceptualisation. (p. 345) 

‘Only’ a doctorate

A prior study by Winter et al found that doctorates were viewed by examiners as needing to:

· be a report of work which others would want to read;
· tell a compelling story articulately whilst pre-empting inevitable critiques;
· carry the reader into complex realms, and inform and educate him/her;
· be sufficiently speculative or original to command respectful peer attention (p. 36).

Note there is nothing here about telling the examiners all the ups and downs, emotional traumas etc that you went through. By being report of work which others want to read I would argue what is needed is an account of a successful piece of research. Issues of anticipating critique, being clear in complexity, and adopting a sufficient voice are important too, but wasted if they’re not part of telling a story about, and selling, something that gives us something new.

Put yourself in your examiners’ shoes

Your examiners are already very busy people, likely with family, personal and other things they might rather be doing of an evening than reading your thesis. But, like with peer review, they do this because they know it’s important (and examiners did it for them, right?). Mullins and Kiley list these as questions that examiners have in mind, and considering these yourself, regularly, throughout your candidature might not be a bad idea:

· How would they have tackled the problem set out in the abstract and the title?

· What questions would they like answers to?

· Do the conclusions follow on from the introduction?

· How well does the candidate explain what he/she is doing

· Is the bibliography up to date and substantial enough? · Are the results worthwhile?

· How much work has actually been done?

· What is the intellectual depth and rigour of the thesis?

· Is this actually ‘research’—is there an argument?

The ‘how much work has actually been done’ is an important one (as they all are). I myself was (rightly) pulled up in an assessment 3/4 of the way through my doctorate: I had utterly undersold my efforts in the field and analysis. Don’t be shy to leave your examiners in no doubt whatsoever how much time you spent generating data, how much data was generated, and how meticulous, iterative and thorough your analysis was. Don’t forget this concrete stuff because you’re living in the clouds of high-brow intellectual argument. They aren’t separate. But don’t forget Mullins & Kiley’s title either: it’s not a Nobel Prize here, only a doctorate.

Key tips:

Manage first impressions – don’t annoy your examiner, or set low expectations. What do you want your examiner to think after reading the abstract, first chapter, lit review:

“Ooh this one doesn’t look great, she’s going to have to pull something incredible out of the bag to get over the line”

Or:

“Wow. This is clear, crisp, easy to understand, clearly offering something new. This is a strong student, and unless there are some serious mishaps, this seems like it will sail through”.

Bear in mind, if you’re going for the latter, set up reasonable expectations – don’t promise Nobel-winning scope and then fail to deliver. As was noted above, make sure you live up to  your promises, and that the intro and conclusions line up.

 

Poor thesis characteristics:

Again, taking the list from Mullins & Kiley:

· lack of coherence;

· lack of understanding of the theory;

· lack of confidence;

· researching the wrong problem;

· mixed or confused theoretical and methodological perspectives;

· work that is not original;

· not being able to explain at the end of the thesis what had actually been argued in the thesis.

Think about what is not listed here. No thesis was failed for being too clear, for making the ultimate arguments really crisp and easy to understand, for adopting clean methodological and theoretical approaches, for being built around a consistent thread / argument across the thesis…

Brilliance in theses:

Mullins and Kiley list what participants in their study said made for an outstanding PhD:

·  an artistic endeavour where the student is designing the work and there is elegance of design, of the synthesis, and executions;

·  creativity;

·  design—where it all fits together;

·  elegant;

·  a well-sculpted piece of work.

I love the craft and artistry in these dimensions. I argue they all support my own view of “parsimony rules!”. But I love how this says brilliance comes not from pure technical rigour, nor from massive complexity, huge scale / volume of work. But from things like sparkle, excitement etc.

 

Dr… or not?

Mullins and Kiley offer a summary of what the final judgements depend on:

· the student’s confidence and independence;

· a creative view of the topic;

· the structure of the argument;

· the coherence of theoretical and methodological perspectives; and

· evidence of critical self-assessment by the student.

What’s going on here? All the stuff we’ve seen before, about coherence, structure, critique etc. What I’d like to point out here is the reference, again, to confidence. As a doctoral candidate you have to write/speak with confidence and authority. Being timid, underselling yourself, ducking your own view and resting exclusively on the writing of others – all potentially catastrophic in terms of nicely convincing your examiners that you don’t deserve the title ‘Dr’. You may just have to trust yourself, and accept that you are in a position to argue something, to tell the world something new and important. If you’re plagued by anxiety that your’e not good enough and the intellectual fraud syndrome, you’re not alone, but you’re not setting yourself up for a positive reading of your thesis either.

Not a Nobel prize, just a doctoral thesis

Did we mention that it’s not a Nobel Prize? Just a thesis! Only a thesis! This is not to trivialise what is a significant piece of work and time in someone’s life. But it is to say “be realistic!”, let parsimony rule. Instrumentalism or being just good enough need not follow from an approach that frames doctoral work around doctoralness.

Finally, I’d like to share a phrase offered by David Boud to a doctoral student. We were working on a one-page summary of the thesis, as a prelude to writing full chapters. “Getting this right is really tough, but important,” Dave said, “the rest… is mere detail”.