Critical appraisal of qualitative data is highly contested. Much of the critique that can be levelled at qualitative data is subjective and sensitive to context, intent and circumstances. However, critical appraisal is a crucial step in most synthesis (the most notable exceptions being scoping reviews and frequently, rapid reviews) and therefore, decisions must be made in selecting some level of criteria against which your chosen studies will be evaluated. Of note: evaluation does not necessarily dictate potential exclusion. Many qualitative tools allow for the inclusion of studies that do not meet all standards, rather the conclusions from lower ranked studies are given less credence in the resulting analysis and review findings.
There are a myriad of tools available for critical appraisal; one review found over 100 different approaches to the task (Munthe-Kass et. al. 2019)! Using a validated and commonly used tool is recommended by the Cochrane Collaboration. The two most commonly used tools are the Critical Appraisal Skills Programme (CASP) tool and the JBI Critical Appraisal Checklist for Qualitative Research.
It is important to emphasize that using a checklist for this work can present challenges. Getting caught up in scores and inter-rater reliability can cloud the ultimate goals of a QES. Researchers are encouraged to focus on methodological strengths and limitations rather than the quality of reporting. For example, one study may score poorly if they did not adequately report the finer details of the methods used and another study may have very rigourously reported their methods, but failed to obtain ethics approval to do the research. The research team must ultimately reconcile the appraisal scores through discussion and consensus building.
Munthe-Kaas, H.M., Glenton, C., Booth, A. et al. (2019). Systematic mapping of existing tools to appraise methodological strengths and limitations of qualitative research: first stage in the development of the CAMELOT tool. BMC Med Res Methodol 19, 113 . https://doi.org/10.1186/s12874-019-0728-6
Carroll, C., & Booth, A. (2015). Quality assessment of qualitative evidence for systematic review and synthesis: Is it meaningful, and if so, how should it be performed? Research Synthesis Methods, 6(2), 149–154. https://doi.org/10.1002/jrsm.1128
Long, H. A., French, D. P., & Brooks, J. M. (2020). Optimising the value of the critical appraisal skills programme (CASP) tool for quality appraisal in qualitative evidence synthesis. Research Methods in Medicine & Health Sciences, 1(1), 31-42. https://journals.sagepub.com/doi/full/10.1177/2632084320947559
Munthe-Kaas, H., Booth, A., & Noyes, J. (2023). Chapter 7. Assessing methodological strengths and limitations of qualitative studies. In Cochrane-Campbell Handbook for Qualitative Evidence Synthesis (1.0). Cochrane Collaboration. https://training.cochrane.org/cochrane-campbell-handbook-qualitative-evidence-synthesis