Effect of editors' implementation of CONSORT guidelines on the reporting of abstracts in high impact medical journals: interrupted time series analysis.
Keywords
Article abstract
OBJECTIVE:
To investigate the effect of the CONSORT for Abstracts guidelines, and different editorial policies used by five leading general medical journals to implement the guidelines, on the reporting quality of abstracts of randomised trials.
DESIGN:
Interrupted time series analysis.
SAMPLE:
We randomly selected up to 60 primary reports of randomised trials per journal per year from five high impact, general medical journals in 2006-09, if indexed in PubMed with an electronic abstract. We excluded reports that did not include an electronic abstract, and any secondary trial publications or economic analyses. We classified journals in three categories: those not mentioning the guidelines in their instructions to authors (JAMA and New England Journal of Medicine), those referring to the guidelines in their instructions to authors but with no specific policy to implement them (BMJ), and those referring to the guidelines in their instructions to authors with an active policy to implement them (Annals of Internal Medicine and Lancet). Two authors extracted data independently using the CONSORT for Abstracts checklist.
MAIN OUTCOME:
Mean number of CONSORT items reported in selected abstracts, among nine items reported in fewer than 50% of the abstracts published across the five journals in 2006.
RESULTS:
We assessed 955 reports of abstracts of randomised trials. Journals with an active policy to enforce the guidelines showed an immediate increase in the level of mean number of items reported (increase of 1.50 items; P=0.0037). At 23 months after publication of the guidelines, the mean number of items reported per abstract for the primary outcome was 5.41 of nine items, a 53% increase compared with the expected level estimated on the basis of pre-intervention trends. The change in level or trend did not increase in journals with no policy to enforce the guidelines (BMJ, JAMA, and New England Journal of Medicine).
CONCLUSION:
Active implementation of the CONSORT for Abstracts guidelines by journals can lead to improvements in the reporting of abstracts of randomised trials.
Article content
Introduction
Clear, transparent, and sufficiently detailed abstracts of journal articles reporting randomised trials are important, because readers often base their initial assessments of a trial on the content of the abstract. In some cases, health practitioners will have access only to the abstract, and could, therefore, make healthcare decisions based solely on the information in that abstract.1 As such, the journal abstract should be a clear and accurate reflection of what is included in the journal article. However, several studies have highlighted problems in the accuracy and quality of abstracts, including a lack of information about the trial methodology and the robustness of the trial results.2 3 Studies comparing the accuracy of information reported in a journal abstract with that reported in the text of the full publication have found claims that are inconsistent with, or missing from, the body of the full article.4 5 This inadequate reporting could seriously mislead a reader’s interpretation of the trial findings.6
In response to these limitations, an extension to the CONSORT statement (CONsolidated Standards Of Reporting Trials) was published in January 2008, providing a list of essential items to include when reporting the main results of a randomised trial in a journal or conference abstract.7 8 However, the effect of the CONSORT for Abstracts guidelines has not been widely evaluated. In this study, we aimed to investigate the effect of these guidelines, and journals’ editorial policy to implement them, on the reporting quality of abstracts of randomised trials published.
Methods
Design and sample
To assess the effect of publication of the CONSORT for Abstracts guidelines, we planned a segmented regression of interrupted time series analyses, a quasi-experimental design method frequently used and recommended to evaluate the effect of guidelines or other complex interventions.9
We selected all primary reports of randomised trials indexed in PubMed (publication type “Randomized Controlled Trial”) and published in one of five high impact general medical journals (the Annals of Internal Medicine, BMJ, Lancet, JAMA, and New England Journal of Medicine). We excluded reports that did not include an electronic abstract, and any secondary trial publications or economic analyses. Because the number of randomised trials published by journals varies each year, we selected at random (www.random.org) a sample of up to 60 trial reports per journal per year for inclusion in our final sample. If fewer than 60 reports of randomised trials were obtained for a journal in a particular year, we assessed all eligible reports.
Interventions
We aimed to assess the effect of the publication of the CONSORT for Abstracts guidelines and the effect of different editorial policies used to implement these guidelines. Therefore, we checked each journal’s instructions to authors in January 2010 for any reference to the CONSORT for Abstracts guidelines (for example, reference to a publication or link to the relevant section of the CONSORT website). For those journals that mentioned the guidelines in their instructions to authors, we contacted the editor of that journal to ask when the guidance was added, whether the journal enforced the guidelines, and if so, how. We classified journals in three categories: those not mentioning the CONSORT guidelines in their instructions to authors (JAMA and the New England Journal of Medicine); those referring to the guidelines in their instructions to authors, but with no specific policy to implement them (BMJ); and those referring to the guidelines in their instructions to authors, with a policy to implement them (Annals of Internal Medicine and the Lancet). In the final journal group, we defined such an implementation policy as having any action to enforce the adherence to CONSORT—that is, an email was sent to authors to revise the abstract according to the CONSORT for Abstracts guidance at the revision stage of the manuscript, or changes were made by the assistant editors of these journals towards the end of the editorial process.
The study period
For the purpose of the study, we selected abstracts published between January 2006 and December 2009. The CONSORT for Abstracts guidelines were published in the Lancet8 and PloS Medicine7 in January 2008 (26 and 22 January, respectively). Shortly after publication (January 2008), the Annals of Internal Medicine, Lancet, and BMJ added the guidelines to their instructions to authors.
Overall, the study period had 48 monthly intervals: 25 before the publication of the CONSORT for Abstracts guidelines (January 2008), three during a transition period to accommodate a gradual implementation (February 2008 to April 2008), and 20 after the intervention (we considered that the intervention began in May 2008).
Outcomes
The primary outcome was a composite outcome. We decided a priori only to focus on items that were reported in fewer than 50% of the abstracts across the five journals in 2006. By focusing only on items that were poorly reported, we hoped to see the greatest effect of implementation of the guidelines (box, web table 1). Therefore, our primary outcome was the monthly mean number of items reported per abstract on a 0 to 9 scale. For our secondary outcome, we assessed items reported in fewer than 20% of abstracts across the five journals in 2006 (box, web table 1). Consequently, the secondary outcome was the monthly mean number of items reported per abstract on a 0 to 5 scale.
Box: Study outcomes
Primary outcome: items reported in fewer than 50% of abstracts across five journals in 2006
-
Details of the trial design
-
Allocation sequence generation
-
Allocation concealment
-
Detailed blinding status (that is, who was blinded)
-
Number of participants randomised to each group
-
Number of participants analysed in each group
-
For the primary outcome, results for each group and its effect size
-
Harms data
-
Funding source
Secondary outcome: items reported in fewer than 20% of abstracts across five journals in 2006
-
Allocation sequence generation
-
Allocation concealment
-
Detailed blinding status (that is, who was blinded)
-
Number of participants analysed in each group
-
Funding source
Data extraction
For each included abstract, we checked to see whether items included in the CONSORT for Abstracts guidelines were adequately reported or not reported. Single data extraction was carried out by two authors (SH and IB). Abstracts were allocated at random to the two assessors across the five journals and time periods; thus, each assessor reviewed a similar number of abstracts from each journal per year. However, both authors first piloted the data extraction form to ensure consistency in the extraction process. Any uncertainty regarding a particular abstract was checked and resolved by discussion. It was not possible to blind assessors to the journal being reviewed, because abstracts for each journal had their own particular house style.
Data analysis
We plotted outcomes graphically over time (by month blocks) from 2006 to 2009 (we excluded the three month transition period from the statistical models but included it in the visual presentation of figures 1⇓ and 2⇓. This allowed us to assess, for the primary and secondary outcomes, any improvements before and after the intervention. We fitted a time series segmented linear regression and estimated monthly use to evaluate changes after the publication of the CONSORT for Abstracts guidelines. The model included terms to evaluate the following variables: a constant term (for levels before the intervention at baseline), a term for linear trends before the intervention, terms to estimate level and trend changes after the intervention, and a residual term. Web appendix 1 provides more information about the statistical model. The ordinary least square model assumes a linear association between time (before and after intervention) and the outcome. We used the Durbin-Watson test to check for autocorrelation. If a Durbin-Watson test result was significant, we corrected the model for autocorrelated errors. One key assumption of ordinary regression analysis is that the errors are independent of each other. To overcome this problem, we used a stepwise autoregression method that initially fits a high order model with many autoregressive lags and then sequentially removes autoregressive parameters until all remaining autoregressive parameters have significant t tests.
Fig 1 Change in outcomes from January 2006 to December 2009, before and after the introduction of the CONSORT for Abstracts guidelines implemented in January 2008 for all journals. Circles=actual values; straight lines=regression lines traced out by the structural predicted values; jagged lines=full model prediction values formed from both the structural and autoregressive parts of the model; vertical lines=transition phase (February to April 2008)
Fig 2 Change in outcomes from January 2006 to December 2009, before and after introduction of the CONSORT for Abstracts guidelines implemented in January 2008 for each journal category. Circles=actual values; straight lines=regression lines traced out by the structural predicted values; vertical lines=transition phase (February to April 2008)
The models (one per outcome) used to estimate within group changes were built on each of the following samples: all journals, journals not mentioning the CONSORT for Abstracts guidelines in their instructions to authors, journals referring to the guidelines in their instructions to authors but with no policy to implement them, and journals referring to the guidelines in their instructions to authors with a policy to implement them.
Finally, we added a dummy group variable to indicate an active or inactive editorial policy and bivariate interaction terms (that is, a group by pre-intervention trend, a group by change in level, and a group by change in trend) to the initial models (one per outcome), to test for differences between journals with an active policy for implementation (Annals of Internal Medicine, Lancet) and those with no active policy (BMJ, JAMA, New England Journal of Medicine). We used SAS statistical software (release 9.2, SAS Institute) for the statistical analyses, and considered P<0.05 as significant.
Results
We included 955 reports of abstracts of randomised trials in our final sample and included trials from a wide variety of clinical specialties.
All journals
Figure 1 and web table 2 detail the change in the mean number of items reported per abstract for the primary and secondary outcomes before and after the introduction of the CONSORT for Abstracts guidelines for all journals. Overall, for the primary outcome, publication of the CONSORT guidelines did not lead to a significant increase in the level of the mean number of items reported (increase of 0.3035 of nine items, P=0.16) or the trend (increase of 0.0193 items per month, P=0.21). For the secondary outcome, we saw a significant increase in the level of the mean number of items reported after the implementation of the CONSORT guidelines (increase of 0.3882 of five items, P=0.0072) and in trends (increase of 0.0288 items per month, P=0.0025). Web table 3 provides the individual data per checklist item, per journal per year.
Effect of the journal policy
We first focused on journals not mentioning the CONSORT for Abstracts guidelines in their instructions to authors (JAMA, New England Journal of Medicine) and journals referring to the guidelines in their instructions to authors but with no policy of enforcement of them (BMJ). We observed no significant increase in the level change or trend change for the primary and secondary outcomes (fig 2, web table 2).
However, when focusing on journals with an active policy to enforce the CONSORT guidelines (Annals of Internal Medicine, Lancet), the publication and editorial policy led to an immediate increase in the level of mean number of items reported but no further improvement in trend for the primary outcome (fig 2, web table 2). At baseline, in January 2006, the mean number of items reported per abstract was 1.52 of nine items, which increased to 2.56 nine items during the 25 months before the intervention. Implementation of the CONSORT guidelines led to an immediate significant increase of 1.50 immediately after the intervention (P=0.0037). However, the trend did not change (increase of 0.0193 items per month, P=0.6). In December 2009, 23 months after the publication of the guidelines, the mean number of items reported per abstract for the primary outcome in the Annals of Internal Medicine and the Lancet was 5.41 items, which represented a 53% increase compared with the expected level estimated on the basis of pre-intervention trends. Web appendix 2 shows data for each of the nine items, per journal, over time.
For the primary outcome, we used a model to compare journals that had an active policy with those that did not have an active policy. The model showed a significant difference in the change in level (that is, relative to the inactive policy group). The change in level immediately after implementation of the CONSORT for Abstracts guidelines increased significantly (increase of 1.90 items, P=0.0006 for interaction test). However, we saw a non-significant difference in trend change (increase of 0.0316 items per month relative to the inactive policy group, P=0.3915 for interaction test).
For the secondary outcome, when focusing only on the Annals of Internal Medicine and Lancet at baseline in January 2006, the mean number of items reported per abstract was 0.18 of five items (fig 2, web table 2). This level had increased to 0.42 items during the 25 months before the intervention. Implementation of the CONSORT for Abstracts guidelines led to a significant increase in the mean number of items reported per abstract immediately after the intervention (increase of 1.03 items, P=0.0035) and a significant change in trend (increase of 0.0613 items per month, P=0.0154). In December 2009, 23 months after publication of the guidelines, the secondary outcome reached 2.91 points in the Annals of Internal Medicine and Lancet, a 4.5-fold (350%) increase compared with the expected level estimated on the basis of pre-intervention trends.
The model used for comparing journals that had an active policy with those that did not have active policy for the secondary outcome showed a significant difference between groups in the change in level (relative to the inactive policy group, increase of 1.24 items, P<0.0001 for interaction test) and change in trend after implementation of the CONSORT for Abstracts guidelines (increase of 0.0761 items per month, P<0.0001 for interaction test).
Discussion
This study evaluated the effect of journals’ editorial policy to implement the CONSORT extension for abstracts guidelines on the reporting quality of abstracts of randomised trials published in five high impact, general medical journals. After publication of the guidelines in January 2008, we identified a significant increase in the reporting of key items in the two journals (Annals of Internal Medicine, and Lancet) that endorsed the guidelines in their instructions to authors and that had an active editorial policy to implement them. By comparison, we observed no significant difference in the one journal (BMJ) that endorsed the guidelines but did not have an active implementation strategy, and in the two journals (JAMA, New England Journal of Medicine) that did not endorse the guidelines in their instructions to authors.
Comparison with other studies
Several studies have assessed the quality of published reports of abstracts of randomised trials.2 3 4 10 11 12 These studies all highlight problems in the accuracy and quality of abstract reporting, particularly a lack of information about the trial methodology and the robustness of its results.2 11 No studies have evaluated the effect of the CONSORT for Abstracts guidelines over time. Our study used a segmented regression of interrupted time series analyses, a quasi-experimental design method to evaluate the effect of the guidelines.13 In such designs, multiple measurements are taken before and after an intervention are applied. This design is effective if a fairly large number of measurements for the outcome of interest are available and a segmented regression can be used to analyse the data. Segmented regression can identify both sudden change due to an intervention, as well as more gradual changes over time.
For clinical practice guidelines, reporting guidelines aim to improve knowledge (authors are aware of the guidelines), attitudes (authors agree and accept the guidelines as a new standard), behaviour (authors change practice patterns to conform with the guidelines), and outcomes (the guidelines improve reporting).14 For this purpose, a strategy of dissemination and implementation of the guidelines is necessary.15
The CONSORT for Abstract guidelines were disseminated through its publication in two general medical journals with high impact factors.7 8 In October 2008, the International Committee of Medical Journal Editors also recommended that “articles on clinical trials should contain abstracts that include the items that the CONSORT group has identified as essential (www.consort-statement.org)” (www.icmje.org). Furthermore, since publication, the guidelines have also been incorporated into the recently revised main CONSORT statement in March 2010.16 17 Dissemination was also enforced by some journal editors referring to the CONSORT for Abstracts guidelines in their instructions to authors.
However, our results show that the guidelines improved reporting only when actively implemented by a specific editorial policy. This policy implied a relatively simple intervention consisting of an email sent to authors to revise the abstract according to CONSORT guidance at the revision stage of the manuscript and, in some instances, further changes made by the journals’ assistant editors at the end of the editorial process. Our results accord with those of systematic reviews showing that, in various fields, passive dissemination of information was generally ineffective and that the use of specific strategies to implement research based recommendations seems to be necessary to ensure that practices change.18
Limitations
Our study has several limitations. We based our study on abstracts of randomised trials published in five high impact, general medical journals, all of which have considerable resources to support their work. We recognise that the resources and procedures available for these journals might not be the same for all journals; therefore, it is not possible to say that what works in these journals is applicable in other smaller, less well resourced journals. Furthermore, the overall reporting quality of abstracts published in these five journals might be higher than in other less well known journals, which was certainly the case when comparing the reporting of CONSORT items in general journals with reporting in specialty journals.19
Our primary outcome was a composite outcome, which focused on the nine CONSORT items reported in fewer than 50% of abstracts in 2006. This composite outcome assumed that each of the nine items were equally important, which might not always be the case, depending on the perspective of the reader. However, by focusing only on items that were poorly reported, we hoped to see the greatest effect of implementation of the guidelines.
Finally, we allowed a three month transition period in our study, during which we could implement and measure the effect of the CONSORT for Abstracts guidelines. This transition period might have been longer for some articles, since they can take longer than three months to pass through the editorial process.
What is already known on this topic
-
Problems in abstract accuracy and quality include a lack of information about trial methods and the robustness of results; such inadequate reporting could seriously mislead a reader’s interpretation of trial findings
-
The CONSORT for Abstract guidelines were published in 2008, listing essential items to include when reporting the main results of a randomised trial in an abstract
-
The effect of these guidelines has not been widely evaluated
What this study adds
-
Based on results from five high impact medical journals, endorsement and active implementation of the CONSORT for Abstracts guidelines by journal editors can lead to improvements in the reporting of abstracts of randomised trials
-
Although authors bear the main responsibility for implementing the guidelines, journals can, and should, have an important role in implementing them