Doctor Diagnosis

INTERVIEW: CRITICAL APPRAISAL

Author: Dr Zaha Kamran Siddiqui

Academic Foundation Doctor at Sheffield University Hospitals NHS Trust. 

 

CRITICAL APPRAISAL

How can I possibly appraise a clinical paper during the interview? Is there even enough time? Where do I even start? Surely, it’s impossible!

If like me, these were your initial thoughts when you found out about this station, then you’re in the right place for some guidance and reassurance!

Critically appraising research is a key skill that comes with practice. It can be daunting at first but the more you do it the easier it becomes – trust me, I was in your shoes once.

It's essential to practice:

  1. Appraising an abstract / paper you haven't seen before. 

  2. Presenting an appraisal of a paper you liked/ have recently read. 

These are core skills that are essential to any academic interview. You can never be too certain what they might ask you and the stakes are too high to take this risk. Following is just a guide to help you understand the process of critical appraisal. You will need to do some of your own reading around certain topics.

 

Top tip !

The best way to practice is to appraise clinical papers of your choosing from any research database – it can be something you’re interested in or not. Choosing papers with different study designs and methodologies is not only good for practice but will also allow you to develop an understanding of strengths and limitations of different study designs.

 

GREAT, SO YOU’VE CHOSEN AN ABSTRACT TO APPRAISE. WHAT NEXT?

You might already have an approach to this, or you might not. Everyone works differently and there is no one right way to appraise research. I’ll guide you through the essential steps of how to perform a critical appraisal and hopefully you can take from it what suits you and develop your own method.

1. IDENTIFYING CORE DETAILS OF THE STUDY:

Whilst reading your abstract identify the aim of the study and the paper’s study design according to the hierarchy of evidence as shown in the pyramid above. Then ask yourself:

Is the study design appropriate for the research question and will it help in achieving the aim of the study?

To answer the above you need to have some basic understanding of different types of study designs so make a pros and cons list of each study design.

Next use the acronym ‘PICO’ to identify core details of the study.

P – Population: recruitment method, sample size, inclusion, and exclusion criteria

I – intervention: this could range from a drug being delivered to a programme being implanted

C – control: this is an optional criterion and mostly used in randomised control trials or case-controlled studies

O – outcome: what was measured? Any 2o Outcomes? Follow up time?

2. HIGHLIGHT THE MAIN FINDINGS OF THE STUDY:

What did the study conclude? Were the results statistically significant?

This depends on what the outcome measure is. Know how to interpret means, percentages, and ratios along with their confidence intervals. When testing a hypothesis, you will need to interpret a p-value. There are plenty of resources online with good, worked examples.


Risk ratios are commonly associated with cohort studies and odds ratios are commonly associated with case-control studies.

Confidence interval: is an estimate of precision. It provides a range within which the true value of whatever you’re measuring in a population will lie.

For the result to be statistically significant the confidence interval cannot cross 0 when mean difference is the outcome measure or it cannot cross 1 if the outcome measure is a risk or odd ratio.

 P<0.05 indicates that an outcome between groups is statistically significant.

P>0.05 indicates there is not a statistically significant difference between groups

3. IDENTIFY STRENGTHS & LIMITATIONS

These are usually difficult to identify without reading the full paper. However, if you’re only given an abstract you can still draw upon pros and cons for the study design. For instance, a cohort study is good to find out about aetiology and prognosis, especially if the exposure being investigated is a rare condition as it allows for direct estimation of incidence of the disease. However, the outcome can take a long time to develop after the exposure therefore making cohort studies time-consuming and resource intensive.

Every research is unique in its design and hence will have its own limitations. PICO is your best friend – by critiquing PICO you can identify whether the population is generalisable and whether the intervention was appropriate. Here are just a few other things to think about when reading a paper.

Have they carried out an intention-to-treat analysis during an RCT? Have they used allocation concealment?

Have they sampled appropriately? Small or large sample size?

Are the results reliable? Multicentre trial?

Based on the limitations, you can make a judgement on how to improve the study as well. For instance, if the paper is a cohort case study you can say that to improve reliability you could possibly repeat the study using an RCT design. Another example for an improving a study is, if the sample size is small, you could improve the precision of treatment effect by recruiting more participants. A larger sample size in turn would also improve the power of the study. There are some key concepts in these examples that require further reading if you are not already aware of them.

4. EXTERNAL VALIDITY

External validity is essentially the extent to which the results of the study can be applied to other situations, outside the given context of the study. Assess whether the population and sample size selected are generalisable to the wider population.

Can you see this study changing clinical practice – why?

5. ETHICAL CONSIDERATIONS

You might not always be asked about this. However, if you do get questioned then apply the four pillars of medical ethics to the study you are appraising. Think about the following:

  • Researcher – Is there any bias? Funding? Associated links?

  • Populations – was informed consent gained? –

  • Study design – was it ethical to give less effective treatment or placebo in RCT? –

  • Generalisability – if the treatment was proved to be effective would it be readily available? 

 

Top tip No. 2!

Every deanery runs stations differently. You need to be able to answer questions beyond just critical appraisal, and possibly the wider context of the paper. However, this may not happen and simply presenting a critical appraisal as per your template or as described above will be ok!

 

PRESENTING A PAPER YOU'VE READ

I would suggest discussing research that aligns with your interests or a topic associated with a project that you have done. This is prime opportunity for you to shine. Why you ask? Well for one you would have already prepared a paper to discuss, showcasing your skills to critically appraise and formulate an opinion and two, discussing a paper that aligns with your research interest will show your enthusiasm towards wanting to do an AFP. However, remember the focus is to discuss a research paper so don’t go off in a tangent.

1.  Prepare

To prepare for this, I would suggest critically appraising more than one study beforehand. Randomised control trials are good to discuss as you can comment on the process of randomisation, blinding, concealment, and ethics. However, you can choose anything you want.

 2. Consolidate

Use the method discussed above and have key points in your head such as the title of the paper, when was it published, the aim and study design and PICO. Just by remembering the aim and the study design you will be able to discuss the strengths and limitations of the study. A crucial part of this station is to discuss the relevance of your research to current practice, how it might benefit the population and how you could improve the study.

3. Present

 Regardless of whether you have been asked to appraise research provided to you in the station or discuss a paper you have read; you should have a structure to present your finding back to the interviewer. The following is just an example:

Summarise core details: The study (mention title) published by (name of Author) on (date) in (name of journal) aims to (include aim). This study is a (mention study design such as RCT, cohort, case-controlled etc.) which is (high or low-level study?) according to the oxford hierarchy of evidence – comment on whether this is appropriate? The study recruited x patients at y duration (days, months etc) of whom z completed follow up – comment on whether appropriate sample size, demographics, and sampling method? The primary outcome of the study was…. Further comment on whether this aligns with the aim of the study. The study concluded… These results were – comment on statistics and explain why you think it is significant or not.

Further discussion:  Here you can discuss the strengths and limitations of the study, generalisability, critique the inclusion / exclusion criteria, any confounding factors, and improvements to the study.

 

FINAL TOP TIPS

- PRACTICE MAKES APPRAISING EASIER - SO START EARLY!

- DURING YOUR PRACTICE APPRAISE PAPERS WITH DIFFERENT STUDY DESIGNS

- HAVE A STRUCTURE AND COVER THE KEY AREAS AS MENTIONED ABOVE

- WORK WITH A FRIEND – GIVE EACH OTHER ABSTRACTS TO APPRAISE AND PRESENT YOUR FINDINGS TO EACH OTHER. THROUGH DISCUSSION YOU WILL LEARN MORE.

- KNOW YOUR STUDY DESIGNS: WHY AND WHEN THEY ARE USED, PROS AND CONS

- HOW TO INTERPRET NUMBER NEEDED TO TREAT, RATIOS, CONFIDENCE INTERVALS AND P-VALUES