Retracted publications in medical education: systematic review

Abstract

Introduction

Research Integrity is based on fundamental principles, including reliability, honesty, respect and accountability. Practices that threaten these standards are classified as research misconduct and fraud and the resulting publications must lead to retraction. Although research in medical education impacts university policies and influences professionals’ behavior, publication retraction in this field has never been explicitly investigated. The main aim of this study is to examine the characteristics of retracted publications in medical education.

Methodology

An eletronic search was performed during June 2023 in three databases: PubMed, Web of Science and Scopus, to identify all the retracted publications in medical education research. We extracted the characteristics of the authors, publication journals, citations and retraction notices.

Results

A total of 12 publications were included in this systematic review. Fifty percent of the publications were published after 2020, being 42% of the studies from China. The average impact factor of the journals was 3.1. Among all the citations found, 54% happened after retraction date and none of them was used as an example of misconduct or to refer to the retraction process. The most common reasons for retraction were duplicate publication (25%) and systematic manipulation of the publication process (25%), followed by peer review concerns (17%).

Conclusion

This study shows evidence that the number of retracted publications in medical education is increasing. Retraction notices tend to be ambiguous and unclear without enough information regarding the request or the reasons of retraction. All of these findings affect the truthfulness and transparency of medical education research. More efforts need to be done to standardize and improve the retraction notice availability, and researchers, journals, academic institutions and funders have to be more aware and publicise this growing problem, playing a key role to prevent the dissemination of misconduct and fraud in medical education research among the academic and scientific communities.

Introduction

Medical education is at a crucial moment for potential change as many challenges were identified after the COVID pandemic (Frenk et al. 2022; Dimassi et al. 2023).

Contrary to medical practice, despite the efforts put in place over the last decades, medical education is far from being evidence-based and many medical educators are largely unaware of the medical education literature.

According to Atluru et al. (2015) research in medical education aims to address questions and contemporary issues in medical education as well as design, support and evaluate curricular innovations (Atluru et al. 2015). Although research in medical education contributes to university policies and influences the behaviour of professionals (Dimassi et al. 2023), the results of useless or even potentially harmful change-inducing research are still poorly identified (Hope et al. 2021).

Research Integrity refers to the “principles and standards that have the purpose to ensure validity and trustworthiness of research” (WCRI 2023) and is based on fundamental principles, including reliability, honesty, respect, and accountability (ALLEA 2023). Another core elements for research integrity are rigor; transparency and open communication in declaring conflicts of interest; and care and respect for all participants in and subjects of research (Commons 2018). Practices that might threaten these standards are classified as research misconduct (Stavale et al. 2049) and can compromise the validity and reliability of research results (Wager et al. 2009).

Problems with research can lead to retraction which is a “mechanism for correcting the literature and alerting readers to articles that contain seriously flawed or erroneous content or data that their findings and conclusions cannot be relied upon” (COPE 2009).

Reasons for retraction include several types of scientific misconduct and errors as plagiarism, data fabrication or falsification, unethical author conduct, redundant publications, peer review concerns, data without authorization, copyright infringement, failure to disclose a major competing interest, scientific mistakes, duplication of publication and journal errors (COPE 2009; Bolland et al. 2022; Claxton 2005; Nair et al. 2020; Samp et al. 2012).

The costs of research misconduct and retracted publications can be both for individuals and economics, implying financial costs to funding sources and harm to the careers of those committing misconduct (Stern et al. 2014). There is as well an “Epistemic cost” defined by literature as “the extent to which the retraction distorts scientific knowledge” which magnitude is not estimated yet (Fanelli et al. 2022). On the other side, many retracted publications are cited and included in meta-analysis, which challenges the purpose of retractions (Fanelli et al. 2022).

The literature suggests that the number of retracted publications has been increasing, although it is unclear whether this increase represents a decline of science integrity, an increase in the visibility and accessibility of published articles or an increasing effort done by publishers and editors (Nair et al. 2020; Samp et al. 2012).

In 2009, the Committee on Publication Ethics (COPE) published the first Retraction Guidelines that aimed to advise publishers and editors on expected practices when dealing with retraction (COPE 2009). Although these Guidelines have increased awareness of this problematic issue, fraud and misconduct seem to be still underreported (COPE 2009; Claxton 2005; Samp et al. 2012).

All scientists/educators must be aware that scientific publications with compromised integrity must be immediately retracted as to allow trustable science (Bolland et al. 2022; Claxton 2005).

Good practices in research mean that honesty and integrity are expected of every stakeholder and they should be promoted through training and education (COPE 2020). Journals, funders, academic institutions and researchers can have both an educational and monitoring role in preventing research misconduct (Stavale et al. 2019).

Integrating new teaching strategies into medical curricula can be challenging for medical educators (Frenk et al. 2022). Being aware of the existence of inadequate publications (containing errors or misconduct) in this field can avoid wasting time and resources essential for others based on solid literature. Literature shows that although several medical fields have raised retracted publications as a concern, mainly in the clinical domain, this has not been explicitly investigated in medical education. The main aim of this study is to examine the characteristics of retracted publications in medical education.

Methodology

A systematic review of the literature was led guided by the question: What are the characteristics of the retracted publications in medical education research? The review was conducted and reported according to the Preferred Reporting of items for Systematic Reviews and Meta-Analysis (PRISMA) Guidelines (Page 2020).

Search strategy and study selection

The first researcher performed an electronic search overseen by the supervisor (third researcher) and without the use of automation methods during June 2023 in three different databases: PubMed, Web of Science and Scopus, using the following algorithms: [(medical education) OR (medical teaching)) OR (medical learning)] AND (retracted publication), the latter being applied as filter. There was no time or language filter applied.

Inclusion criteria

All the retracted publications in medical education were included in this review.

Publications were deemed eligible according to the criteria defined by PICO tool:

. P (Population) – Publications in medical education.

. I (Intervention) – Retracted Articles.

. C (Comparison) – N/A.

. O (Outcomes) – The characteristics of the retracted publications.

Exclusion criteria

Were excluded from this review all the publications outside the medical education field, systematic reviews, meta-analysis, letters, opinion articles and conference communications based either on the title or the abstract (screening phase).

Data extraction

After title and abstract screening, performed by the first and second researchers and supervised by the third researcher, the full text of all eligible publications was obtained. Disagreements at the screening stage were resolved by consensus. For all eligible studies were extracted details of each retracted publication: year of publication, number of authors per publication and 1st and last authors affiliation and number of retracted publications. The characteristics of publication journals such as name, quartile, and impact factor were also extracted (Clarivate 2023). To characterise the publication citations we extracted the number of citations of every publication (from google scholar) and all of them were analyzed to find out if they happened before or after the retraction. For all the citations done after retraction we analysed their context as well as time (in years) between the retraction and the citation. In order to further characterize the process of retraction, we searched the notices of retration.

To distinguish the citations done before and after the retraction date, the ones done in the year of that date were considered “citations before retraction”.

To assess the notices of retraction, we followed the COPE retraction guidelines: Notices of retraction should: Be linked to the retracted article wherever possible (i.e., in all online versions); Clearly identify the retracted article (e.g., by including the title and authors in the retraction heading or citing the retracted article); Be clearly identified as a retraction (i.e., distinct from other types of correction or comment); Be published promptly to minimise harmful effects; Be freely available to all readers (i.e., not behind access barriers or available only to subscribers); State who is retracting the article; State the reason(s) for retraction; Be objective, factual and avoid inflammatory language (Commons 2018).

The reasons of retraction were classified accordingly to the reasons found in the eligible publications: ambiguous claims and conflicting sources, data manipulation, peer review concerns, duplication, systematic manipulation of the publication process and unethical author conduct.

To identify the publications with or without a watermark mentioning the retraction, the publications were searched in different databases and if at least one of them did not have a watermark, these were considered as not having “watermarking”.

All the retracted publications with an associated retracted notice were considered to “Be published promptly to minimize harmful effects”.

The eligible publications were divided into different categories of medical education research according with the studies found: self-regulated learning, medical humanities programs, discipline construction, clinical learning, palliative care teaching, research skills teaching, virtual reality, problem-based learning and individual learning style. Disagreements at any of the described stages were similarly resolved by consensus.

Results

Identified publications

A total of 1005 articles were identified through database screening based on the initial searching. After duplicated publications were removed (n = 128), the remaining 877 were screened in two steps. First 827 publications were excluded based on the title followed by 29 studies excluded based on the abstract. The remaining 21 publications were eligible for full article reading, from which 9 were excluded for not meeting the inclusion criteria. Finally, the remaining 12 publications were included in this systematic review (Fig. 1).

Fig. 1

Search process depicted in a PRISMA flow diagram

Characteristics of retracted publications

The oldest retracted publication included in this review is from 2004 and the most recent is from 2022, being 6 (50%) of the studies published after 2020 (Fig. 2).

Fig. 2

Retracted publications by year

Of all the publications, 6 (50%) have between 3 and 4 authors, 3 (25%) between 5 and 6, 2 (17%) more than 6 and 1 (8%) only 2 authors. The first and last author affiliations from the retracted publications included in this review are the same for all the 12 publications and are originated from six different Countries, being the largest number of retractions from China (42%) (Table 1). Two (17%) of the 1st authors of these publications have at least 2 retracted publications. All the last authors have only 1 retracted publication.

Table 1 Characteristics of the authors

The 12 remaining articles were published in eight different Journals. The Journal of Healthcare Engineer published 3 (25%) of all the publications (Table 2). The average impact factor of the various journals was 3.1, being the maximum 5.5 and the minimum 0.13. Regarding the journal’s quartiles, 5 (42%) are in Q2, 3 (25%) Q3, 2 (17%) Q1 and 1 (8%) Q4 (Table 2).

Table 2 Characteristics of publication journals

Retracted articles have been cited on average 11 times. The most highly cited retracted publication has been cited 54 times and 3 (25%) publications have no citations. Five (42%) retracted publications were cited only before the retraction date, 1 (8%) was cited only after the retraction date and 3 (25%) were cited both before and after that date (Table 3).

Table 3 Characteristics of citations

Among all the citations found (136), 62 were done before and 74 after the retraction date. Regarding the citations done after the retraction date, besides the 15 not able to be analysed, none of them were used as an example of misconduct or to refer to the retraction process. Forty-nine (49%) were used to support the discussion of the results of the articles and 14 (19%) the introduction (Table 3).

From all the citations done after the retraction date, 11 (15%) were cited less than 2 years after that date, 19 (26%) between 2 and 5 years and 44 (59%) more than 5 years after the retraction date (Table 3).

Out of the 12 eligible articles, 10 (83%) have a retraction notice available. Out of the 10 retraction notices available, 10 (100%) comply with six of the COPE retraction guidelines: linked to the retracted article wherever possible; clearly identify the retracted article; be clearly identified as a retraction; be published promptly to minimize harmful effects; be freely available to all readers; and be objective, factual, and avoid inflammatory language. One (10%) notice of retraction did not mention the reason of retraction and 8 (80%) did not state who was retracting the article (Table 4). In Table 4, the details on retraction notices are relative to the number of publications with available retraction notes.

Table 4 Characteristics of Retraction

The most common reasons for retraction were duplicate publication (25%) and systematic manipulation of the publication process (25%), followed by peer review concerns (17%). The remaining publications were retracted for ambiguous claims and conflicting sources (8%), data manipulation (8%) or unethical author conduct (8%). One article did not have description of any specific reason for retraction. Two (67%) of the retractions were requested by the journal/publisher/editor, 1 (8%) by the author(s) and in 9 (75%) this information is not described (Table 4).

Regarding the delay of retraction after publication, in 7 (54%) publications this period lasted between 12 and 23 months, in 4 (30%) less than 12 months and in 1 (8%) more than 36 months. The longest time between publication and retraction was 96 months (Table 4).

Seven (58%) publications have at least one online platform where there is no watermark or any other indication of the retraction.

Out of the 12 retracted publications, 3 (25%) investigate about problem-based learning, 2 (17%) virtual reality, 1(8%) self-regulated learning, 1(8%) medical humanities programs, 1(8%) discipline construction, 1(8%) clinical learning, 1(8%) palliative care teaching, 1(8%) teaching research skills and 1(8%) individual learning styles.

Discussion

Although retractions in medical journals have been studied over the years, even for each medical specialty (Fernandes et al. 2023), data from medical education publications are sparse. Indeed, apart from a few exceptions, such as a recent paper on predatory medical education journals (Tomlinson 2024) there is still little literature on the publication quality in medical education.

In this systematic review, the characteristics of retracted publications in medical education research were examined, focusing on authors details and characteristics of the retractions, journals of publication and citations.

The research suggests that the number of retracted publications has increased in recent years, as 50% (ALLEA 2023) of the publications were retracted after 2020. These findings are similar with previous research in retracted publications that seems to follow the same trend (Chauvin et al. 2019; Chen et al. 2012; Rapani et al. 2020). A recent publication reported that the retraction rate for European biomedical-science papers increased fourfold in the last twenty years, being nearly 67% of them withdrawn owing to misconduct (Freijedo-Farinas et al. 2024). It is still uncertain whether this results from an increase of scientific misconduct and fraud or the adoption of the COPE guidelines that is accelerating the detection of unreliable publications (Nair et al. 2020; Samp et al. 2012; Rapani et al. 2020).

Fifty percent of the eligible studies were performed by 3 to 4 authors, which is roughly lower than the number of authors (4.3) described for medical education research (Maggio et al. 2021). (Li et al. 2018) for example, also identified 4 as the median number of authors of retracted publications, despite suggesting that complexity of research may be reflected by the number of authors, due to the requirement of multi-expertise and collaborative work.

China represents 42% of the findings of 1st author affiliation and according to Bhatt (Bhatt 2021), China is in the top 5 countries in the World with the highest number of retractions.

Out of all the 1st and last authors of the studies included in this systematic review, two 1st authors appear in more than one retracted publication (two), which can indicate that these are isolated cases of retraction rather than organized systems that systematically ignore principles of research’ ethics and integrity (Rapani et al. 2020).

Nagella & Madhugiri (Nagella and Madhugiri 2020) noticed that journals with higher impact factors retract more when comparing with those with lower impact factors. This could be explained by the fact that the former are more read and consequently more examined or high impact journals tend to publish strong positive studies more frequently reflecting data fabrication (Nagella and Madhugiri 2020). Among the identified retracted publications, the highest journal impact factor was 5.5, being the median number 3.1. Fifty nine percent of the publications were published in journals Q1 and Q2.

The number of citations found after retraction (74), also observed by other studies (Chen et al. 2012; Rapani et al. 2020), proves that new research can still use sources originating from retracted publications, which means that retraction per se cannot eliminate the risk of their use by other authors. Chen et al. (Chen et al. 2012) also pointed that the citations’ validity verification is likely to become very challenging as more and more studies are becoming attached to retracted publications and researchers take their validity for granted. Indeed, citation of retracted publications represents a major problem for the scientific community (Rapani et al. 2020). (Moylan and Kowalczuk 2016) argue that the fact that retraction notices are rarely cited themselves suggest that the publications retraction are not known by the readers, which is consistent with the results of this systematic review: none of the citations done after retraction were used either as an example of misconduct or to the retraction process. To overcome this problem, Chauvin et al. (Chauvin et al. 2019) recommend the withdrawal of the retracted publications full texts to be replaced by a complete and comprehensible retraction notice.

Various other retraction-related studies suggest that most retracted publications are due to misconduct and fraud (Chauvin et al. 2019; Rapani et al. 2020; Li et al. 2018; Moylan and Kowalczuk 2016). In this review, 25% (three) of the eligible publications were retracted due to systematic manipulation of the publication process, which is not a clear and detailed description of the reasons of retractions as it is recommended by COPE guidelines (2009). Other studies also mention that it is common to find only generic and unclear retraction notices (Rapani et al. 2020).

Out of the 12 publications, 10 (83%) had a retraction notice associated and available. Although all of them have fulfilled at least 6 out of 8 of the COPE retraction guidelines, 8 out of 10 do not state who requested the retraction. Stavale et al. (2019) highlights that the existence of a reporting retraction standard instrument could promote a consensus both in ethical policies and publication form of retractions. Disciplinary and educational actions are essencial to increase the awareness on publication ethical issues and to prevent malpractices and misconduct.

Considering the time between the publication of the original article and the retraction notice, our findings show a median number of 19 months, which goes along with other studies in retracted publications (Chauvin et al. 2019; Moylan and Kowalczuk 2016). (Moylan and Kowalczuk 2016) demonstrated that articles retracted due to misconduct took longer (approximately 12 months) to retract comparing with articles retracted due to honest error (approximately 6 months).

Even though watermark can be an effective method of easily identify the retracted publications (Nair et al. 2020), the findings of this study show that 58% of our sample can be found online without any kind of mark or any indication that the publication was retracted. This can explain the fact that 74 of the citations were done after retraction.

Our study has a number of limitations. First, both the screening phase and data extraction were initially performed not independently by two authors, double-checked by the third author and ultimately discussed for a consensus by all the authors. Although a considerable number of single-author systematic reviews are indexed in Medline (Pacheco et al. 2023), this can be seen as a limitation. Secondly, we did not contact the authors of included publications for missing data or further details. For instance, temporal changes in retraction reasons may occur due to publisher/journal procedures rather than actual changes in those reasons. Third, the possibility that we missed some retractions despite the use of updated databases, as well as retractions of more recently published articles. Lastly, the studies included in this review were carried out in a few countries, which may hypothesize that in those countries there is a greater number of retractions, and limits the generalization of the results to other geographical contexts.

Conclusion

As a first approach to analyse the characteristics of retracted publications in the medical education field, this study shows evidence that the number of retracted publications is increasing. Similarly to other fields, retracting problematic articles helps to maintain the accuracy and integrity of the medical education literature.

Retraction notices tend to be ambiguous and unclear without enough information regarding the request or the reasons of retraction. Some of the publications remains fully available online with no watermark or any notification that they have been retracted, which can be related with the fact that citations of retracted publications continues after retraction. All of these findings affect the truthfulness and transparency of medical education research and integrity. More efforts need to be done not only to identify problematic publications but also to standardize and improve the retraction notice availability. Researchers, journals, academic institutions and funders have to be more aware and publicise this growing problem, playing a key role to prevent the dissemination of misconduct and fraud in medical education research among the academic and scientific communities.