Original Research


Pediatric Orthopaedic Surgery Research Quality: A Decade of Change at POSNA Annual Meetings from 2011-2020

Mitchell A. Johnson, MD1; Hillary Mulvey, MD2; Andrew Parambath, BA3; Jason Anari, MD4; Apurva S. Shah, MD, MBA4

1Hospital for Special Surgery, New York, NY; 2Vanderbilt University Medical Center, Department of Orthopaedic Surgery; Nashville, TN; 3Perelman School of Medicine at the University of Pennsylvania, Philadelphia, PA; 4Division of Orthopaedic Surgery, The Children’s Hospital of Philadelphia, Philadelphia, PA

Correspondence: Apurva S. Shah, MD, MBA, Assistant Professor of Orthopedic Surgery, Children’s Hospital of Philadelphia, The Hub for Clinical Collaboration, Orthopaedics 4th Floor, 3500 Civic Center Blvd., Philadelphia, PA 19104. E-mail: [email protected]

Received: May 9, 2022; Accepted: September 14, 2022; Published: November 1, 2022

DOI: 10.55275/JPOSNA-2022-499

Volume 4, Number 4, November 2022

Abstract:

Background: The Pediatric Orthopaedic Society of North America (POSNA) is a leading organization focused on disseminating research and promoting high-quality pediatric orthopaedic care. The purpose of our study was to understand how research quality in pediatric orthopaedic surgery has evolved over time by evaluating trends in clinical evidence and study design in podium presentations given at POSNA annual meetings from 2011-2020.

Methods: Podium presentation abstracts for all POSNA meetings from 2011-2020 were independently reviewed for number of study patients, single center versus multicenter, randomized vs. non-randomized design, and abstract focus/topic. The level of evidence (LOE) assigned in the abstract program was also recorded. Chi-squared and Mann-Whitney U tests were used to compare differences in abstract characteristics between 2011-2015 and 2016-2020. Linear regressions were performed to analyze changes in level of evidence and the proportion of multicenter studies over time.

Results: A total of 1589 podium presentations were reviewed including 679 from 2011-2015 and 910 from 2016-2020, representing a 34% increase in the total number of presentations. The median number of patients per abstract was higher in 2016-2020 compared to 2011-2015 (81.5 vs. 49, p<0.001). An increased proportion of presentations from 2016-2020 were multicenter (14.1% vs. 9.7%, p=0.009), with spine abstracts having the highest proportion of multicenter collaboration (23.9%, p=0.002). The subspecialty focus of podium presentations has remained consistent over time with the exception of spine-related abstracts (19.5% vs. 14.9%, p=0.018). Overall, 4.9% of the presentations were categorized as level 1 evidence, 18.5% level 2, 41.8% level 3, and 34.8% level 4. Between 2011-2020, the proportion of LOE 3 studies increased significantly (p=0.039). The proportion of LOE 4 studies decreased, but not significantly (p=0.060). There was no change in the proportion of level 1 and 2 studies (p=0.416).

Conclusions: POSNA podium presentations have increased in number over the past decade, in large part due to the development of subspecialty day programs. This appears to have resulted in greater diversity in the academic program, with a lower proportion of spine abstracts over time. The LOE of POSNA presentations has improved with a higher proportion of level 3 studies compared to earlier years. More recent abstracts have evaluated higher numbers of patients and there has been a significant increase in multicenter study designs, suggesting improved collaboration.

Level of Evidence: IV

Key Concepts

  • POSNA podium presentations have increased in number over the past decade due to subspecialty day programming.
  • A decrease in the proportion of studies focused on spine-related topics has been observed.
  • Improved level of evidence has been observed with a higher proportion of level 3 evidence.
  • More recent podium presentations have evaluated higher numbers of patients and more commonly utilized multicenter study designs, suggesting improved collaboration.
  • Further work focusing on fostering and promoting multicenter collaboration across all pediatric subspecialties should be promoted.

Introduction

The Pediatric Orthopaedic Society of North America (POSNA) is a not-for-profit organization with the goal of advancing musculoskeletal care for children and adolescents by promoting education, research, and quality care.1 One of the ways that it aims to do this is through annual meetings, which serve as valuable conduits for the dissemination of new research and ideas to a large network of pediatric orthopaedic providers. Podium presentations are an important opportunity for authors to receive criticism and input from experts in the field and discuss efforts to promote new research goals. As a result, many presentations at the POSNA annual meeting are ultimately published in prominent orthopaedic journals.2

Level of evidence (LOE), first proposed in 1986, is a frequently used tool for evaluating the quality and potential impact of clinical research.3 In the field of orthopaedic surgery, LOE is traditionally assigned based on guidelines released by the Journal of Bone and Joint Surgery.4 The LOE can assist in the interpretation of scientific studies and the determination of how much weight to assign results.

The most recent analysis of research presented at POSNA showed minimal changes in LOE when comparing the 2001-2003 annual meetings to those in 2007-2008.5 However, no studies have evaluated changes since. In this study, we aimed to assess changes in the quality and focus of research presented at POSNA. We hypothesized that both multicenter collaboration and the LOE of podium presentations has improved over the past decade.

Methods

Every paper abstract presented via an oral presentation between 2011-2020 at the POSNA annual meetings was identified from the official abstract book made available to delegates. Abstracts presented in poster format were excluded. Included presentations were reviewed by three separate reviewers to determine study characteristics including total number of subjects, investigation topic, study design, and level of evidence. The level of evidence assigned at the time of abstract submission was recorded. To confirm the validity of LOE assigned in the POSNA abstract books, an independent evaluation was performed of a random sample of 60 abstracts with the level of evidence blinded. Randomization was performed using a computer-based randomization program. One investigator (MAJ) reviewed each abstract and assigned a LOE per the current guidelines for authors published in the most recent Journal of Bone and Joint Surgery.5

Subspecialty focus was classified into one of the following categories based on review by all three reviewers: QSVI (Quality Safety and Value Initiative), Foot/Ankle/Lower Extremity, Hand/Wrist/Upper Extremity, Hip, Spine, Trauma, Sports, Neuromuscular, and Infection/Tumor.

Comparisons were made between abstracts presented between 2011-2015 and 2016-2020. Categorical variables were compared using Chi-squared tests. The Mann-Whitney U test was used to compare the number of patients across different time periods. Linear regressions were performed to analyze changes in the proportion of multicenter studies and the proportions of studies assigned each level of evidence. These regressions were calculated and reported with a slope (B) and 95% confidence interval (CI). Pearson correlation coefficients were converted to a coefficient of determination (r2). Studies with an unspecified level of evidence (specifically basic science abstracts) were excluded from trend analysis. Interobserver reliability between the independent reviewer and the POSNA assignment of LOE was performed. All statistical analysis was performed using SPSS software for Macintosh, version 23. A significance threshold of p<0.05 was used for all analyses.

Results

A total of 1589 oral presentations were reviewed with 679 (42.7%) from 2011-2015 and 910 (57.3%) from 2016-2020. The lowest number of presentations occurred in 2011 (n=93) and the highest in 2017 (n=232). The proportion of LOE 1 or 2 presentations was similar when comparing the second half of the decade to the first (23.2% vs. 21.4%, p=0.386). There was a significant increase in the presentation of multicenter projects in 2016-2020 (14.1% vs. 9.7%, p=0.009) and no change in the proportion of presentations utilizing nationwide databases (8.2% vs. 8.1%, p=0.919). A significant decrease in the proportion of spine-focused presentations between 2016-2020 (14.9% vs. 19.4%, p=0.018) was noted. The number of patients per abstract was higher during the latter half (median 81.5, IQR 37-205) compared to the first half of the decade (median 49, IQR 24-100, p<0.001) (Table 1).

Table 1. Characteristics of POSNA Abstracts, 2011-2015 vs. 2016-2020

2011-2015 2016-2020 P-Value
Number of Abstracts 679 910 -
Median Sample Size (IQR) 49 (24-100) 81.5 (37-205) <0.001
Level of Evidence
1 or 2
3
4

145 (21.4)
229 (33.7)
262 (38.6)

211 (23.2)
404 (44.4)
265 (29.1)
0.386
Multicenter 66 (9.7) 128 (14.1) 0.009
Randomized Controlled Trial 21 (3.1) 30 (3.3) 0.820
Database 55 (8.1) 75 (8.2) 0.919
Survey 44 (6.5) 44 (4.8) 0.156
Topics
Basic Science/Anatomy/Biomechanics
Opioids/Pain Management
Medical Education
Cost Analysis

86 (12.7)
14 (2.1)
14 (2.1)
15 (2.2)

106 (11.6)
26 (2.9)
3 (0.3)
14 (1.5)

0.538
0.317
0.001
0.323
Focus
QSVI (Quality Safety and Value Initiative)
Foot/Ankle/Lower Extremity
Hand/Wrist/Upper Extremity
Hip
Spine
Trauma
Sports
Neuromuscular
Infections/Tumors

44 (6.5)
81 (11.9)
45 (6.6)
118 (17.4)
132 (19.4)
85 (12.5)
56 (8.2)
55 (8.1)
37 (5.4)

82 (9.0)
113 (12.4)
62 (6.8)
139 (15.3)
136 (14.9)
130 (14.3)
99 (10.9)
71 (7.8)
45 (4.9)

0.065
0.769
0.884
0.260
0.018
0.308
0.080
0.828
0.653

Values reported as number (%) or median (IQR). Statistically significant values are in bold (p<0.05).

The proportion of LOE 3 presentations increased by an average of 2.1% per year (95% CI 0.138% – 4.022%, p=0.039) from 44.2% (n=34) in 2011 to 52.2% (n=84) in 2020 (r^2 = 0.433). The proportion of LOE 4 presentations showed an average decrease of 2.8% per year (95% CI -5.769% – 0.151%, p=0.060) from 40.3% (n=31) in 2011 to 31.7% (n=51) in 2020 (r^2 = 0.374). There was no significant change in the proportion of level 1 and 2 presentations (B = 0.725%, 95% CI -1.222% to 2.671%, p=0.416) (Figure 1). The proportion of multicenter presentations increased by an average of 1.9% per year (95% CI 0.635% – 3.129%, p=0.008) from 1.3% (n=1) in 2011 to 23.0% (n=37) in 2020 (r^2 = 0.602) (Figure 2). The subspecialty focus with the highest proportion of multicenter presentations was spine (23.9%) and the lowest was Sports (3.9%, p<0.001).

Figure 1. Proportion of presentations graded as level I and II, III, and IV at POSNA annual meetings, 2011-2020. Trendlines for level III and IV indicated by dashed lines (r^2 = 0.433 and r^2 = 0.374, respectively).

jposna2022499_fig1.jpg

Figure 2. Proportions of presentations reporting multicenter collaboration at POSNA annual meetings, 2011-2020. Trendline indicated by dashed line (r^2 = 0.602).

jposna2022499_fig2.jpg

The interobserver correlation coefficient between the randomly selected sample of abstracts and the POSNA LOE classification showed near perfect agreement for level of evidence assignment (ICC 0.831).6

The largest number of randomized controlled trials were presented with a focus on spine related topics (n=16, 6.2%). However, when comparing between the four largest specialties, there was no difference in the proportion of randomized controlled trials or prospective studies (p=0.233) (Figure 3).

Figure 3. Proportion of level 1, 2, 3, and 4 evidence studies for the four largest subspecialty sections at POSNA annual meetings, 2011-2020.

jposna2022499_fig3.jpg

Discussion

Our study demonstrates increasing levels of evidence at the Pediatric Orthopaedic Society of North America annual meeting since 2011. This change was predominantly driven by an increase in the proportion of LOE 3 studies. Additionally, more presentations in the latter half of the decade were the result of multicenter collaboration and included a larger number of patients.

Changes in level of evidence have been studied across several orthopaedic conferences with improvements occurring at the International Society of Hip Arthroscopy,7 American Shoulder and Elbow Surgeons,8 European Society of Sports Traumatology, Knee Surgery and Arthroscopy,9 Scoliosis Research Society,10 and the Orthopaedic Trauma Association11 scientific meetings over time. Additionally, a review by Voleti et al. of presentations given at the American Academy of Orthopaedic Surgeons annual meeting demonstrated a 19% increase in the proportion of level 1 and 2 studies between 2001 and 2010.12

Previously documented decreases in level 4 and increases in level 3 studies at both POSNA5 and in pediatric orthopaedic journals13 in the early 2000s were associated with no significant changes in the proportion of level 1 or 2 studies. Our findings represent a continuation of this trend, potentially indicating that authors are developing higher quality retrospective study designs by utilizing control groups to elevate level IV evidence to level III. While we did observe an increase in the total number of prospective and randomized controlled trials, the overall proportion did not increase. This is likely due to the difficulty and time investment needed to conduct level 1 and 2 studies.5

In addition to making quality improvements to clinical care, many orthopaedic surgeons value research contributions given the importance of research productivity to academic progression.1416 While many orthopaedic conferences boast high rates of publication overall,2,1722 higher level of evidence studies tend to have a higher chance of publication, thus incentivizing orthopaedic surgeons to produce higher quality study designs.12 Additionally, a larger number of patients allow for greater power to detect differences between comparison groups23 and may lead to a higher rates of publication.24 Despite increasing availability and the use of large nationwide databases in adult orthopaedic surgery research,25,26 presentations using databases were not demonstrated to be increasing at POSNA. Reasons for these study types not being utilized in pediatric orthopaedic research should be explored.

Our study also observed a significant decrease in the proportion of studies focused on spine-related topics. While the field of pediatric spine research has continued to increase,27 this observed decline likely represents an attempt to include research from subspecialties across pediatrics including smaller subspecialties that continue to have clinical importance for orthopaedic providers.

Several limitations should be noted for this study. Level of evidence is not a perfect method for assessing the quality of research,28,29 with some of the most impactful and frequently cited studies in orthopaedic surgery classified as level 3 and 4.3033 Additionally, basic science, cadaveric studies, and animal studies are unable to be classified using the LOE system but may still provide valuable clinical information for providers. However, it remains a widely instituted system for categorizing the methodology of a clinical manuscript or presentation. While we observed a statistical increase in the level of evidence presented, the actual difference in the relevance of information presented remains in question. By including other quality metrics, such as multicenter collaboration and number of patients studied, we hoped to better assess quality of research. Additionally, our investigation was limited to the POSNA annual meeting and may not accurately reflect trends across the entire field of pediatric orthopaedic surgery.

Despite the increasing number of presentations given at POSNA annual meetings in recent years, there has been a significant improvement in both the level of evidence and the proportion of multicenter collaboration. Further work should be directed towards promoting multicenter collaboration across all pediatric subspecialties, the usage of nationwide databases, and the development of prospective study designs.

Disclaimer

No funding was received for this project. The authors have no conflicts of interest to report.

References

  1. About POSNA. [10/5/2020]. Available at: https://posna.org/Our-Society.
  2. Amirhamzeh D, Moor MA, Baldwin K, et al. Publication rates of abstracts presented at pediatric orthopaedic society of North America meetings between 2002 and 2006. J Pediatr Orthop. 2012;32(2):e6-e10.
  3. Sackett DL. Rules of evidence and clinical recommendations on the use of antithrombotic agents. Chest. 1986;89(2 Suppl):2s-3s.
  4. Marx RG, Wilson SM, Swiontkowski MF. Updating the assignment of levels of evidence. JBJS. 2015;97(1):1–2.
  5. Kelley SP, Cashin MS, Douziech JR, et al. Levels of evidence at the pediatric orthopaedic society of North America annual meetings. J Pediatr Orthop. 2010;30(6):612–616.
  6. Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977;33(1):159-174.
  7. Kay J, de Sa D, Shallow S, et al. Level of clinical evidence presented at the International Society for Hip Arthroscopy Annual Scientific Meeting over 5 years (2010-2014). J Hip Preserv Surg. 2015;2(4):332-338.
  8. Kay J, Memon M, de Sa D, et al. Level of clinical evidence presented at the open and closed American Shoulder and Elbow Surgeons annual meeting over 10 years (2005-2014). BMC Musculoskelet Disord. 2016;17(1):470.
  9. Kay J, Memon M, Rogozinsky J, et al. Level of evidence of free papers presented at the European Society of sports traumatology, knee surgery and arthroscopy congress from 2008 to 2016. Knee Surg Sports Traumatol Arthrosc. 2017;25(2):602-607.
  10. Harshavardhana NS, Dormans JP. Observational analysis of changing trends in level of evidence of Scoliosis research society annual meeting podium presentations in the New Millennium (2001-2013). Spine Deform. 2016;4(3):193-199.
  11. Lerman DM, Cable MG, Thornley P, et al. Has the level of evidence of podium presentations at the musculoskeletal tumor society annual meeting changed over time? Clin Orthop Relat Res. 2017;475(3):853-860.
  12. Voleti PB, Donegan DJ, Baldwin KD, et al. Level of evidence of presentations at American Academy of Orthopaedic Surgeons annual meetings. J Bone Joint Surg Am. 2012;94(8):e50.
  13. Cashin MS, Kelley SP, Douziech JR, et al. The levels of evidence in pediatric orthopaedic journals: where are we now? J Pediatr Orthop. 2011;31(6):721-725.
  14. Lopez J, Susarla SM, Swanson EW, et al. The association of the H-Index and academic rank among full-time academic hand surgeons affiliated with fellowship programs. J Hand Surg. 2015;40(7):1434-1441.
  15. Khan AZ, Kelley BV, Patel AD, et al. Academic productivity among fellowship associated adult total joint reconstruction surgeons. Arthroplasty Today. 2017;3(4):298-302.
  16. Cvetanovich GL, Saltzman BM, Chalmers PN, et al. Research productivity of sports medicine fellowship faculty. Orthop J Sports Med. 2016;4(12):2325967116679393.
  17. Jasko JJ, Wood JH, Schwartz HS. Publication rates of abstracts presented at annual musculoskeletal tumor society meetings. Clin Orthop Relat Res. 2003(415):98-103.
  18. Kinsella SD, Menge TJ, Anderson AF, et al. Publication rates of podium versus poster presentations at the American Orthopaedic Society for Sports Medicine meetings: 2006-2010. Am J Sports Med. 2015;43(5):1255-1259.
  19. Naziri Q, Mixa PJ, Murray DP, et al. Adult reconstruction studies presented at AAOS and AAHKS 2011-2015 annual meetings. Is there a difference in future publication? J Arthroplasty. 2018;33(5):1594-1597.
  20. Potter E, Fernandez I, Fillinger M, et al. Abstracts accepted for presentation at Orthopaedic Trauma or Arthroplasty Conferences: which conference is the best indicator of future publication? J Am Acad Orthop Surg Glob Res Rev. 2019;3(8):e020.
  21. Preston CF, Bhandari M, Fulkerson E, et al. The consistency between scientific papers presented at the Orthopaedic Trauma Association and their subsequent full-text publication. J Orthop Trauma. 2006;20(2):129-133.
  22. Raudenbush B, Frost C, Okafor R, et al. Publication rate of podium presentations from the North American Spine Society Annual Meetings. Global Spine J. 2018;8(3):273-278.
  23. Biau DJ, Kernéis S, Porcher R. Statistics in brief: the importance of sample size in the planning and interpretation of medical research. Clin Orthop Relat Res. 2008;466(9):2282-2288.
  24. Easterbrook PJ, Berlin JA, Gopalan R, et al. Publication bias in clinical research. Lancet (London, England). 1991;337(8746):867-872.
  25. Bohl DD, Singh K, Grauer JN. Nationwide databases in orthopaedic surgery research. J Am Acad Orthop Surg. 2016;24(10):673-682.
  26. Karlson NW, Nezwek TA, Menendez ME, et al. Increased utilization of American Administrative Databases and Large-scale Clinical Registries in Orthopaedic Research, 1996 to 2016. J Am Acad Orthop Surg Glob Res Rev. 2018;2(11):e076.
  27. Hosseinzadeh P, Louer C, Sawyer J, et al. Subspecialty training among graduates of pediatric orthopaedic fellowships: an 11-year analysis of the database of American Board of Orthopaedic Surgery. J Pediatr Orthop. 2018;38(5):293-296.
  28. Poolman RW, Struijs PA, Krips R, et al. Does a “Level I Evidence” rating imply high quality of reporting in orthopaedic randomised controlled trials? BMC Med Res Methodol. 2006;6:44.
  29. Bhandari M, Richards RR, Sprague S, et al. The quality of reporting of randomized trials in the Journal of Bone and Joint Surgery from 1988 through 2000. J Bone Joint Surg Am. 2002;84(3):388-396.
  30. Chaudhry ZS, Fram B, Henn RF, 3rd, et al. The 50 most-cited articles in meniscal allograft transplantation research: a bibliometric analysis. Cartilage. 2019;10(2):196-204.
  31. Namdari S, Baldwin K, Kovatch K, et al. Fifty most cited articles in orthopedic shoulder surgery. J Shoulder Elbow Surg. 2012;21(12):1796-1802.
  32. Huo YQ, Pan XH, Li QB, et al. Fifty top-cited classic papers in orthopedic elbow surgery: a bibliometric analysis. Int J Surg (London, England). 2015;18:28-33.
  33. Kavanagh RG, Kelly JC, Kelly PM, et al. The 100 classic papers of pediatric orthopaedic surgery: a bibliometric analysis. J Bone Joint Surg Am. 2013;95(18):e134.