Qualitative Evidence Synthesis (QES)

The Cochrane Guidance documentation provides the general advice that when formulating a search strategy to find qualitative evidence, the searcher should "generally privilege specificity (retrieval of only relevant items) over sensitivity (retrieval of all potential items) in recognition that qualitative research is far less prevalent than quantitative research and so subject searches that run without methodological filters will contain a higher proportion of irrelevant hits." (Harris et. al., 2018). That being said, one must be very thoughtful in developing the search strategy.

Barnett-Page and Thomas expand on the methodological and conceptual links between various approaches to QES in their 2009 study, emphasizing two broad categories: idealist vs realist. The budding QES researcher should read this particular article to gain a more nuanced understanding of the differences and similarities in QES methods before selecting which is most appropriate for their needs. 

Consider the following:

• Is your goal to be comprehensive in your search? Is it crucial to locate every single study related to your topic? Put in different terms, are you aiming for an aggregate approach or an interpretive approach?
  • Aggregate: this will require significant refinement of your research question in order to not be overwhelmed by irrelevant results. It will be an important step to consult with an information specialist or subject expert librarian in order to create a search strategy that will be exhaustive. An aggregative approach is more appropriate for a descriptive study, rather than an interpretive one. This approach to QES is not without its critics: see Thorne (2017) for an elaboration on the proliferation of what she calls a "technical and theoretically superficial form of scholarly enterprise". Unlike quantitative effectiveness reviews, the impact of missing a qualitative study in a QES does not have the same effect, particularly if searches are yielding studies with "more of the same" type results. 
  • Interpretive: a less complex search strategy that is relatively straightforward will likely suffice. In order to assist in selecting which studies to focus on, consider a purposive sampling technique (as opposed to random or probabilistic); this method is generally accepted within qualitative research for its emphasis on a depth of interpretive understanding and saturation (Palinkas et.al., 2015).
• Will you be considering or focusing on theory? If so, it is highly recommended that you consider the guidance in the following citation: Paterson, B. L., Thorne, S. E., Canam, C., & Jillings, C. (2001). Meta-theory. In Meta-Study of Qualitative Health Research (pp. 91-108). SAGE Publications, Inc., https://doi.org/10.4135/9781412985017 

Sources

Barnett-Page, E., & Thomas, J. (2009). Methods for the synthesis of qualitative research: A critical review. BMC Medical Research Methodology, 9(1), 59. 

Harris, J. L., Booth, A., Cargo, M., Hannes, K., Harden, A., Flemming, K., Garside, R., Pantoja, T., Thomas, J., & Noyes, J. (2018). Cochrane Qualitative and Implementation Methods Group guidance series—paper 2: Methods for question formulation, searching, and protocol development for qualitative evidence synthesis. Journal of Clinical Epidemiology, 97, 39–48. https://doi.org/10.1016/j.jclinepi.2017.10.023

Palinkas, L. A., Horwitz, S. M., Green, C. A., Wisdom, J. P., Duan, N., & Hoagwood, K. (2015). Purposeful sampling for qualitative data collection and analysis in mixed method implementation research. Administration and Policy in Mental Health, 42(5), 533–544. https://doi.org/10.1007/s10488-013-0528-y

Thorne S. (2017). Metasynthetic Madness: What Kind of Monster Have We Created?. Qualitative health research, 27(1), 3–12. https://doi.org/10.1177/1049732316679370

Research has indicated that searching for qualitative studies is more challenging than searching for quantitative studies. Reasons for this include poor indexing, non-descriptive titles, inconsistent use of terminology and missing abstracts. For this reason, while database searching is a crucial first step, supplementary search methods are required in QES in order to capture more relevant studies (Frandsen and Eriksen, 2023). This will be covered in the next section. 

The first step in any evidence synthesis project is to identify a "seed paper" or "pearl citation": that is, a paper that explores the foundational concepts that are key to your research question. It will be a crucial tool in the execution of your search, and will very likely be one of the first things a librarian will ask about. You will use this exemplar paper to identify key concepts, subject headings, keywords, relevant journals and databases, and highly cited authors in this subject area. 

Database selection will always depend on the subject you are studying. Consulting with a subject expert librarian is recommended. At the very least, you should consider using the databases that are recommended in the research guide most pertinent to your subject area. 

Currently, there is no consensus on the number of databases that should be searched when doing QES. Studies in the health sciences have sought to empirically demonstrate the ideal combination of databases to maximize recall and relevance in search results, invariably pointing to a combination of CINAHL, PubMed/Medline, Embase and PsycINFO as the researchers "best bets". For more detail on these studies and for further advice on the selection of sources, see this resource. 

Sources

Frandsen, T. F., & Eriksen, M. B. (2023). Supplementary strategies identified additional eligible studies in qualitative systematic reviews. Journal of Clinical Epidemiology, 159, 85–91. https://doi.org/10.1016/j.jclinepi.2023.04.017

Thus far, it has been clearly noted that database searches for qualitative information has significant deficiencies, and therefore, any QES search strategy cannot rely on databases alone. Supplementary search strategies are highly recommended. There is diffuse terminology to describe "supplementary" search strategies: hand-searching, forward and backward citation searching, non-database searches, citation chasing, etc. By any name, this method is tried and true (Frandsen & Eriksen, 2023). 

The CLUSTER method is recognized as a rigorous approach to supplementary searching (Tsang & Maden, 2021). Using CLUSTER enhances the possibility for replicability in supplementary searching, and has been used with success in QES. It is particularly useful in the context of QES for its capacity to capture thick and rich data that would not necessarily present through traditional search methods. Follow the steps outlined in the graph below to use the CLUSTER method. If you need help interpreting any of this terminology or don't know how to follow the steps, ask us for help. 

Sources

Frandsen, T. F., & Eriksen, M. B. (2023). Supplementary strategies identified additional eligible studies in qualitative systematic reviews. Journal of Clinical Epidemiology, 159, 85–91. https://doi.org/10.1016/j.jclinepi.2023.04.017

Tsang, A., & Maden, M. (2021). CLUSTER searching approach to inform evidence syntheses: A methodological review. Research Synthesis Methods, 12(5), 576–589. https://doi.org/10.1002/jrsm.1502

There is no dominant or generally accepted search filter to capture qualitative research. This is due to a variety of reasons, not the least of which that "qualitative research" as a MeSH term only appeared in 2003, and there was no retrospective effort to index pre-2003 studies to add this new term. In addition, databases that do not use MeSH will very likely have inconsistent indexing for qualitative research.

Studies have shown that the use of a compact search filter such as (qualitative OR interview* OR Interviews/ OR findings) performed comparably to a more sophisticated filter (Flemming & Briggs, 2007; Gorecki et. al. 2010). There are, however, established search filters to capture qualitative data for several databases. For more detail on how to use these filters, see the ISSG Search Filters Resource here. 

Sources

Flemming, K., & Briggs, M. (2007). Electronic searching to locate qualitative research: Evaluation of three strategies. Journal of Advanced Nursing, 57(1), 95–100. https://doi.org/10.1111/j.1365-2648.2006.04083.x

Gorecki, C. A., Brown, J. M., Briggs, M., & Nixon, J. (2010). Evaluation of five search strategies in retrieving qualitative patient-reported electronic data on the impact of pressure ulcers on quality of life. Journal of Advanced Nursing, 66(3), 645–652. https://doi.org/10.1111/j.1365-2648.2009.05192.x

This is the quintessential and enduring question of most researchers - when can I stop? When do I know that I have enough?

In short, there is no answer to this question, but there comes a point when you have to stop searching and start reading! Some questions to consider when trying to decide if you can conclude the search stage of your QES with confidence:

1. Are you starting to encounter the same references over and over? If so, you have likely reached the stage known as "bibliographic sufficiency" (Booth, 2021 [slide 52]). 
2. Are you only finding studies that seem to have similar findings? Are you confident that it is unlikely that you will learn more about your topic by seeking out further studies? If so, you have likely reached the stage known as "theoretical saturation" (Booth, 2021 [slide 52]). 
3. Have you decided whether or not you are privileging sensitivity (as many results as necessary) over specificity (relevancy over exhaustive searching)? 

NB: if you are employing purposive sampling, you should aim to achieve "diversity NOT homogeneity" (Booth, 2021 [slide 38]). In other words, your synthesis will not necessarily be served well by selecting a critical mass of studies with similar findings. This is not to say that you cannot use studies that have similar findings, but you should be looking for diversity of experience and outcome throughout the studies (i.e. not just in the main conclusion). This will enrich your analysis, and result in an intellectual output that is more likely to contribute to the wider breadth of knowledge in your discipline. 

Source: Booth, A. Searching for qualitative research for inclusion in systematic reviews: a structured methodological review. Syst Rev 5, 74 (2016).