Reporting randomised clinical trials of analgesics after traumatic or orthopaedic surgery is inadequate: a systematic review
© Montané et al; licensee BioMed Central Ltd. 2010
Received: 1 July 2009
Accepted: 12 January 2010
Published: 12 January 2010
Several randomised clinical trials (RCTs) of analgesics in postoperative pain after traumatic or orthopaedic surgery (TOS) have been published, but no studies have assessed the quality of these reports. We aimed to examine the quality of reporting RCTs on analgesics for postoperative pain after TOS.
Reports of RCTs assessing analgesics in postoperative pain after TOS were systematically searched from electronic databases. The quality of reports was assessed using the CONSORT checklist (scoring range from 0 to 22). The quality was considered poor when scoring was 12 or lesser. The publication year and the impact factor of journals were recorded.
A total of 92 reports of RCTs were identified and 69 (75%) scored 12 or lesser in CONSORT checklist (range 5-17). The mean (SD) CONSORT score of all reports was 10.6 (2.7). Missing CONSORT items included primary and secondary outcome measures (11%), the specific objectives and hypothesis definition (12%), the sample size calculation (12%), the dates defining the periods of recruitment (12%), the discussion of external validity of findings (14%), the allocation sequence generation (24%), and the interpretation of potential bias or imprecision of results (25%). There was a little improvement in CONSORT scores over time (r = 0.62; p < 0.001) and with impact factor of journals (r = 0.30; p < 0.001).
Quality of reporting RCTs on analgesics after TOS is poor. Reporting of those RCTs should be improved according to methodological standard checklists in the next years.
Traumatic and orthopaedic surgery (TOS) is one of the most painful surgical interventions. Evidence on the efficacy of analgesic drugs for pain after TOS is usually obtained from randomised clinical trials (RCTs). However, interpretation of results can be hampered by incomplete data reporting and by low methodological quality of the trial . The quality of the evidence provided by orthopaedic journals about major orthopaedic surgery has been assessed, [2, 3] as well as the quality of the RCTs assessing analgesic interventions . We have assessed characteristics of patients included in RCTs on analgesics drugs for pain after TOS, as well as analgesic drugs, outcomes and observation periods in a previous study . Nevertheless, quality in reporting RCTs of analgesics for postoperative pain after TOS has not been systematically assessed. The aim of this study was to examine the quality of reporting RCTs on the efficacy of analgesic drugs in postoperative pain after TOS.
Data sources and searching
A systematic review assessing quality of RCTs of analgesic drugs in pain after TOS was performed. A search from electronic databases PubMed, EMBASE, and The Cochrane Library, and references from identified papers and hand searches were carried out. The search included combinations of the following MeSH terms: "pain, postoperative", "randomised controlled trials", "analgesics", "anti-inflammatory agents, non steroidal", and "orthopaedics".
RCTs of analgesic drugs for the control of pain after TOS published from January 1966 to June 2006 were included. The language of the reports was restricted to English, French, Italian, German, and Spanish. Reports of clinical trials in adult patients were included if they were parallel group studies, and if patients had been randomly allocated to the various treatment groups (either opioids, paracetamol, or non-steroidal anti-inflammatory drugs [NSAIDs], combinations of them or placebo), exclusively after TOS, and the route of drug administration was either oral, intramuscular, intravenous, subcutaneous, rectal, or transcutaneous. Only full original articles were considered for inclusion. Case reports, abstracts, and letters were excluded. RCTs on anaesthetics, on preoperative or perioperative drug administration, and on spinal drug administration, and those on patients undergoing disc surgery were all excluded.
Data abstraction and study characteristics
The following data were extracted from each report: language, year of publication, geographical area where the study was carried out, medical journal, its impact factor, and its medical area. The quality of reporting of each included study in the reports was assessed independently by three evaluators (EM, AV, CA) with CONSORT checklist . The quality in reporting trials was scored according to the revised CONSORT checklist, which includes 22 items assessing the quality of the contents of the report (Title and Abstract, Introduction, Methods, Results, and Discussion). Discrepancies between the evaluators were discussed and resolved by consensus.
The quality of RCTs was classified in three categories according to CONSORT score: excellent (≥ 20 items), good (between 13 and 19), and poor (≤ 12).
The impact factor, for each journal where the RCTs were published, was determined from the Science Citations Index (http://www.accesowok.fecyt.es/, last accessed March 23, 2009). The RCTs from journals without a citation index were excluded from the impact factor median and correlation analysis. The RCTs were considered to be from high-impact journals if the citation index was above the median of the journals included in the study, low-impact journals if the citation index was below the median and no-impact journals if they didn't have a citation index. These three categories were used for the comparative analysis between quality scales.
Mean (SD), median (range), and frequencies were used as descriptive statistics according to the variable characteristics. Comparison of the mean overall scores for CONSORT checklist according to compared groups in RCTs (with placebo or without placebo group, and with different compared analgesics) was carried out with ANOVA analysis. The Spearman correlation was used for correlations between the overall scores of each trial in the CONSORT checklist and the year of publication, and the impact factor of the journal where the report had been published. Comparisons between overall score in the CONSORT checklist and impact factor of the journals were carried out with ANOVA analysis; p-values were adjusted using the Bonferroni method. Statistical significance was considered when p-values were less than 0.05. Agreement between evaluators was assessed with intraclass correlation coefficient and 95% confidence intervals (ICC and 95% CI). We accepted the definition of a high level of agreement represented by values of ICC from 0.65 . All analyses were conducted using SPSS 12.0 (SPSS Inc, Chicago IL, USA).
A total of 326 reports were selected, and 235 were excluded because the studies were not RCTs, orthopaedic surgery was combined with another type of surgery, non analgesic drugs were also assessed, and for other reasons. Ninety-one publications reporting 92 RCTs met inclusion criteria (one publication reported two RCT). Eighty four (91.3%) were published in English, six (6.5%) in French, and two (2.2%) in Italian. The first report had been published in 1971, and 49 (53.3%) had been published after 1990. The articles described 58 RCTs carried out in Europe (63%), 26 in America (26; 28.3%), 5 in Asia (5.4%), and 3 in Africa (3.3%). Reports were published in 46 medical journals from different medical areas, mainly Pharmacology and Therapeutics 32 (35%), and Anaesthesiology 26 (28%). A total of 76 (82.6%) RCTs were published in 34 journals included in the Science Citation Index. The median impact factor was 1.76 (range 0.31 - 7.53). Thirty-six reports had been published in low-impact journals, and 40 in high-impact journals.
Reporting randomised controlled clinical trials according to different items in CONSORT checklist.
Paper section and Topic
(N = 92)
Title and abstract
Outcomes and estimation
Quality of reporting according to compared groups in randomised clinical trials.
RCT (n = 92)
Mean score (sd)
Placebo control group
Yes (n = 42)
No (n = 50)
p = 0.011
NSAIDs (n = 46)
Opioids (n = 10)
NSAIDs vs. Opioids (n = 19)
NSAIDs + opioids vs. NSAIDs and/or opioids (n = 17)
p = 0.511
Mean (SD) overall scores for CONSORT checklist according to impact factor (IF) of the medical journals.
No IF journals
(n = 16)
(n = 36)
(n = 40)
(N = 92)
The results of the present study illustrate that the overall quality of reporting of RCTs assessing the efficacy of analgesic drugs after TOS was poor. The main deficiencies were lack of information on methods to define the hypothesis and the outcomes, the sample size calculation, and lack of data on the results of estimated effects size and their precision. These methodological flaws limit their validity and the interpretation of the results, and they may lead to biased findings [1, 8]. In fact, few reports adequately interpreted the results of RCTs in discussion section, taking into account sources of potential bias or imprecision, the problems associated with multiplicity of analysis and outcomes or the external validity of findings. Our results are similar to those of other systematic reviews on the methodological quality in orthopaedic-surgery related topics, and on analgesic interventions . However, the results also show that there has been a small and progressive improvement in the quality of reporting of RCTs over time, as described in studies in other therapeutic areas [9–13]. In general, our results indicate that editors and reviewers of journals, as well as readers, should be more aware of the methodological shortcomings in the reports of RCTs evaluating analgesic drugs in postoperative after TOS. We think that most of those methodological deficiencies in reporting RCTs, could be easily corrected and avoided in the future.
Three approaches have been developed to assess the quality of RCTs: one based on specifically relevant individual components (which evaluate selected aspects of trials, such as randomisation or blinding), scales (providing numerical scores of quality), and checklists (that involve lists of items such as CONSORT or Delphi) . The journal impact factor has also been used widely as a quality measure . In our opinion, the CONSORT checklist helps to evaluate quality in a more accurate way because it includes other important methodological items such as the hypothesis description, definition of the primary and secondary outcomes, sample size calculation, and others. On the other hand, we expected that journals with the highest impact factors would have the highest scientific quality . Bath et al.  have reported a significant association between the CONSORT score and the impact factor in stroke. When we classified the medical journals according to the impact factor (none, low, or high), the CONSORT score was higher in high-impact journals than non-impact journals. Since the reporting guidelines such as the CONSORT checklist, or the Standards for Reporting in Trials (SORT) -focused on reporting side effects- were published or endorsed as a requirement for publishing, reports of clinical research have improved [17, 18]. But in fact, in 2003 only 22% of high impact journals had endorsed it, and there are still many journals in anaesthesiology, pain, or orthopaedic topics that have not done so.
Assessment of the methodological quality and reporting has been carried out in other specialities and/or in other conditions [9–13, 20–33]. In general, the quality of reporting of RCTs was low or less than optimal, with more than half of all studies scoring less than half of the scores from the quality scales used with the exception of two studies which were adequately reported [13, 32]. However, these studies have used a variety of instruments based on different criteria (ranging from 1 to more than 80 items). The CONSORT checklist and the Jadad scale were the most frequently used, as well as modified versions of both . Therefore, it is not possible to make a direct comparison with the results of our study. Most of the referred studies have assessed clinical trials published in specific medical journals (basically assessing the quality of those journals), while the present study focused on a specific problem.
We limited the bias in the selection and in scoring the methodological quality of RCTs by conducting process in triplicate and independently . As available scales and checklists are heterogeneous in size and complexity, their overall scores can also diverge considerably depending on the evaluator [36, 37]. However, our results showed high agreement between evaluators in the overall scores of the quality checklist. The CONSORT checklist is not a quality scale but as it gives recommendations on reporting, it provides an unweighted score of reporting compliance, where some items may be clearly more important than others. Moreover, its relevance can vary depending on the condition being assessed (e.g., blinding is crucial in RCTs on the treatment of pain, but less so on those in the treatment of infectious diseases) . Nevertheless, we have used it because of the lack of a better alternative.
It should also be acknowledged that most of the trials included in this review were published several years ago and may no longer reflect current reporting practices. Furthermore, only published trials were retrieved, and authors or pharmaceutical companies were not contacted. Unpublished trials tend to be of a lower methodological quality, and their results tend to be more biased than those of published trials; therefore, the methodological quality of controlled research on analgesics in TOS may be even worse than described in the present report.
The quality of reporting RCTs on analgesic drugs after TOS is inadequate, although a modest improvement has been seen over time. Therefore, editors and reviewers of journals, as well as readers, should be more aware of the shortcomings in reporting RCTs on analgesic drugs in postoperative pain after TOS. The quality of reporting RCTs on analgesia in TOS should be improved according to standard methodological checklists in the future.
Financial support was provided by Boehringer Ingelheim, Spain SA. The study sponsor had no involvement in the study design, collection, analyses, interpretation of data, writing of the report, and in the decision to submit the paper for publication. We thank Angus Paules for English revision.
- Moher D, Pham B, Jones A, Cook DJ, Jadad AR, Moher M, Tugwell P, Klassen TP: Does quality of reports of randomised trials affect estimates of intervention efficacy reported in meta-analyses?. Lancet. 1998, 352: 609-613. 10.1016/S0140-6736(98)01085-X.View ArticlePubMedGoogle Scholar
- Kiter E, Karatosum V, Günal I: Do orthopaedic journals provide high-quality evidence for clinical practice?. Arch Orthop Trauma Surg. 2003, 123: 82-85.PubMedGoogle Scholar
- Bhandari M, Morrow F, Kulkarni AV, Tornetta P: Meta-analyses in orthopaedic surgery. A systematic review of their methodologies. J Bone Joint Surg. 2001, 83: 15-24. 10.1302/0301-620X.83B1.10986.View ArticlePubMedGoogle Scholar
- Jadad AR, McQuay HJ: Meta-analyses to evaluate analgesic interventions: a systematic qualitative review of their methodology. J Clin Epidemiol. 1996, 49: 235-243. 10.1016/0895-4356(95)00062-3.View ArticlePubMedGoogle Scholar
- Montané E, Vallano A, Aguilera C, Vidal X, Laporte J: Analgesics for pain after traumatic or orthopaedic surgery: what is the evidence-a systematic review. Eur J Clin Pharmacol. 2006, 62: 971-988. 10.1007/s00228-006-0185-0.View ArticlePubMedGoogle Scholar
- Moher D, Schulz KF, Altman DG, for the CONSORT Group: The CONSORT statement: revised recommendations for improving the quality of reports of parallel-group randomised trials. Lancet. 2001, 357: 1191-1194. 10.1016/S0140-6736(00)04337-3.View ArticlePubMedGoogle Scholar
- Oxman AD, Guyatt GH, Singer J, Goldsmith CH, Hutchison BG, Milner RA, Streiner DL: Agreement among reviewers of review articles. J Clin Epidemiol. 1991, 44: 91-98. 10.1016/0895-4356(91)90205-N.View ArticlePubMedGoogle Scholar
- Petticrew M: Why certain systematic reviews reach uncertain conclusions. BMJ. 2003, 326: 756-758. 10.1136/bmj.326.7392.756.View ArticlePubMedPubMed CentralGoogle Scholar
- Hill CL, La Valley MP, Felson DT: Secular changes in the quality of published randomized clinical trials in rheumatology. Arthritis Rheum. 2002, 46: 779-784. 10.1002/art.512.View ArticlePubMedGoogle Scholar
- Bath FJ, Owen VE, Bath PMW: Quality of full and final publications reporting acute stroke trials. A systematic review. Stroke. 1998, 29: 2203-2210.View ArticlePubMedGoogle Scholar
- Sánchez-Thorin JC, Cortés MC, Montenegro M, Villate N: The quality of reporting of randomised clinical trials published in ophthalmology. Ophtalmology. 2001, 108: 410-415. 10.1016/S0161-6420(00)00500-5.View ArticleGoogle Scholar
- Latronico N, Botteri M, Minelli C, Zanotti C, Bertolini G, Candiani A: Quality of reporting of randomised controlled trials in the intensive care literature. A systematic analysis of papers published in Intensive Care Medicine over 26 years. Intensive Care Med. 2002, 28: 1316-1323. 10.1007/s00134-002-1339-x.View ArticlePubMedGoogle Scholar
- Ko CY, Sack J, Chang JT, Fink A: Reporting randomized, controlled trials. Where quality of reporting may be improved. Dis Colon Rectum. 2002, 45: 443-447. 10.1007/s10350-004-6217-x.View ArticlePubMedGoogle Scholar
- Moher D, Jadad AR, Tugwell P: Assessing the quality of randomized controlled trials. Current issues and future directions. Int J Technol Assess Health Care. 1996, 12: 195-208. 10.1017/S0266462300009570.View ArticlePubMedGoogle Scholar
- Garfield E: Journal impact factor: a brief review. CMAJ. 1999, 161: 979-980.PubMedPubMed CentralGoogle Scholar
- Lee KP, Schotland M, Bacchetti P, Bero LA: Association of journal quality indicators with methodological quality of clinical research articles. JAMA. 2002, 287: 2805-2808. 10.1001/jama.287.21.2805.View ArticlePubMedGoogle Scholar
- Moher D, Jones A, LePage L, for the CONSORT Group: Use of the CONSORT statement and quality of reports of randomized trials: a comparative before-and-after evaluation. JAMA. 2001, 285: 1992-1995. 10.1001/jama.285.15.1992.View ArticlePubMedGoogle Scholar
- Edwards JE, McQuay HJ, Moore RA, Collins SL: Reporting of adverse effects in clinical trials should be improved: lessons from acute postoperative pain. J Pain Symptom Manage. 1999, 18: 427-437. 10.1016/S0885-3924(99)00093-7.View ArticlePubMedGoogle Scholar
- Altman DG, for the CONSORT Group: Endorsement of the CONSORT statement by high impact medical journals: survey of instructions for authors. BMJ. 2005, 330: 1056-1057. 10.1136/bmj.330.7499.1056.View ArticlePubMedPubMed CentralGoogle Scholar
- Schulz KF, Chalmers I, Grimes DA, Altman DG: Assessing the quality of randomization from reports of controlled trials published in obstetrics and gynecology journals. JAMA. 1994, 272: 125-128. 10.1001/jama.272.2.125.View ArticlePubMedGoogle Scholar
- Ah-See KW, Molony NC: A qualitative assessment of randomized controlled trials in otolaryngology. J Laryngol Otol. 1998, 112: 460-463.PubMedGoogle Scholar
- Huwiler-Müntener K, Jüni P, Junker C, Egger M: Quality of reporting of randomized trials as a measure of methodologic quality. JAMA. 2002, 287: 2801-2804. 10.1001/jama.287.21.2801.View ArticlePubMedGoogle Scholar
- Montenegro R, Needleman I, Moles D, Tonetti M: Quality of RCTs in periodontology. A systematic review. J Dent Res. 2002, 81: 866-870. 10.1177/154405910208101214.View ArticlePubMedGoogle Scholar
- Montori VM, Bhandari M, Devereaux PJ, Manns BJ, Ghali WA, Guyatt GH: In the dark. The reporting of blinding status in randomised controlled trials. J Clin Epidemiol. 2002, 55: 787-790. 10.1016/S0895-4356(02)00446-8.View ArticlePubMedGoogle Scholar
- Bhandari M, Guyatt GH, Lochner H, Sprague S, Tornetta P: Application of the consolidated standards of reporting trials (CONSORT) in the fracture care literature. J Bone Joint Surg. 2002, 84: 485-489. 10.1302/0301-620X.84B7.12532.View ArticlePubMedGoogle Scholar
- Bhandari M, Richards RR, Sprague S, Schemitsch EH: The quality of reporting of randomised trials in the Journal of Bone and Joint Surgery from 1988 through 2000. J Bone Joint Surg. 2002, 84: 388-396. 10.1302/0301-620X.84B7.12532.View ArticlePubMedGoogle Scholar
- Boutron I, Tubach F, Giraudeau B, Ravaud P: Methodological differences in clinical trials evaluating nonpharmacological and pharmacological treatments of hip and knee osteoarthritis. JAMA. 2003, 290: 1062-1070. 10.1001/jama.290.8.1062.View ArticlePubMedGoogle Scholar
- Mills E, Loke YK, Wu P, Montori VM, Perri D, Moher D, Guyatt G: Determining the reporting quality of RCTs in clinical pharmacology. Br J Clin Pharmacol. 2004, 58: 61-65. 10.1111/j.1365-2125.2004.2092.x.View ArticlePubMedPubMed CentralGoogle Scholar
- Piggott M, McGee H, Freuer D: Has CONSORT improved the reporting of randomized controlled trials in the palliative care literature? A systematic review. Palliative Medicine. 2004, 18: 32-38. 10.1191/0269216304pm857oa.View ArticlePubMedGoogle Scholar
- Gummesson C, Atroshi I, Ekdahl C: The quality of reporting and outcome measures in randomized clinical trials related to upper-extremity disorders. J Hand Surg. 2004, 29: 727-734. 10.1016/j.jhsa.2004.04.003.View ArticleGoogle Scholar
- Strippoli GFM, Craig JC, Schena FP: The number, quality, and coverage of randomized controlled trials in nephrology. J Am Soc Nephrol. 2004, 15: 411-419. 10.1097/01.ASN.0000100125.21491.46.View ArticlePubMedGoogle Scholar
- Hewit C, Hahn S, Torgerson DJ, Watson J, Bland JM: Adequacy and reporting of allocation concealment: review of recent trials published in four general medical journals. BMJ. 2005, 330: 1057-1058. 10.1136/bmj.38413.576713.AE.View ArticleGoogle Scholar
- Doig GS, Simpson F, Delaney A: A review of the true methodological quality of nutritional support trials conducted in the critically III: time for improvement. Anesth Analg. 2005, 100: 527-533. 10.1213/01.ANE.0000141676.12552.D0.View ArticlePubMedGoogle Scholar
- Jadad AR, Moore RA, Carroll D, Jenkinson C, Reynolds DJ, Gavaghan DJ, McQuay HJ: Assessing the quality of reports of randomized clinical trials: is blinding necessary?. Controlled Clin Trials. 1996, 17: 1-12. 10.1016/0197-2456(95)00134-4.View ArticlePubMedGoogle Scholar
- Clark HD, Wells GA, Huët C, McAlister FA, Salmi LR, Fergusson D, Laupacis A: Assessing the quality of randomized trials: reliability of the Jadad scale. Control Clin Trials. 1999, 20: 448-452. 10.1016/S0197-2456(99)00026-4.View ArticlePubMedGoogle Scholar
- Moher D, Jadad AJ, Nichol G, Penman M, Tugwell P, Walsh S: Assessing the quality of randomised controlled trials: an annotated bibliography of scales and checklists. Control Clin Trials. 1995, 16: 62-73. 10.1016/0197-2456(94)00031-W.View ArticlePubMedGoogle Scholar
- Jüni P, Witschi A, Bloch R, Egger M: The hazards of scoring the quality of clinical trials for meta-analysis. JAMA. 1999, 282: 1054-1060. 10.1001/jama.282.11.1054.View ArticlePubMedGoogle Scholar
- Egger M, Jüni P, Barlett C, Holenstein F, Sterns J: How important are comprehensive literature searches and the assessment of trial quality in systematic reviews? Empirical study. Health Technol Assess. 2003, 7: 1-76.PubMedGoogle Scholar
- The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1472-6904/10/2/prepub
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.