Operational Benchmarking in Graduate Medical Education
In applying benchmarking to the world of graduate medical education (GME), this clarity quickly disappears. This murkiness occurs because the discussion quickly moves from acknowledging the benefits of benchmarking (where to set the bar to start an ongoing monitoring process of key operational and educational metrics) to identifying the perceived shortfall of any benchmarking process (whether the bar is too high or too low or why do we need to benchmark when we know we are doing a great job). It is foolhardy to view benchmarking in that light. Benchmarks are simply starting points in the process of implementing an ongoing improvement plan tied to the measurement of key operational and educational attributes that are important to a residency program and/or its sponsoring hospital. It is how the benchmarks are used that really determines their effectiveness.
Can benchmarking be used to measure the effectiveness of something as dynamic as GME?
GME programs vary in size and setting and have internal nuisances so numerous that you are more likely to find two identical snowflakes before finding two identical residency experiences. Still the answer is yes, but only if applied correctly… For GME, the benchmarking process must measure the effectiveness of key attributes within a residency training program that allow program directors and faculty to directly identify and improve the educational and operational performance of their residency program simultaneously. These attributes are common to similar residencies in other hospitals. In most cases, the financial impact will be an added positive outcome as a result of improving operational and educational improvements. Overall, the objective of operational benchmarking is a better education for the residents so that they are prepared to be clinically proficient and ready for the practice of medicine upon graduation.
Until now, the relative lack of activity in GME benchmarking has been due to the inherent complexity in measuring several concurrent clinical training activities across multiple sites utilizing different combinations of faculty and residents which is what happens in any residency program. Adding to the complexity are other faculty driven activities such as private practice, research and other formal training activities. A further challenge is that many programs use part time community physicians to perform and support different patient care, coverage and resident supervision activities.
Further, a natural tension and lack of common understanding exists between those who budget and distribute the funds associated with GME (Administration) and those that provide patient care (faculty physicians) who also contribute to the support of the program through their professional fee billings. This has led residency programs to be wary of the transparency of their activity to Administration, fearing at a minimum, any acknowledged areas of operational or financial shortfall would be detrimental to their program and ultimately to their personal role as faculty.
In this era of financial crisis, almost any broad based opportunity that appears to provide a cost savings is being implemented. Residency programs have historically been spared the same kind of across the board cuts that have been applied to most of the other activities and services within hospitals. Residency programs are not facing these types of cuts or even worse - the threat of program termination.
Within this context, residency programs have valid concerns that such cuts will have a significant long term negative impact on the program. Dictated cost cutting measures can force residency programs to change operating and educational practices which could be very disruptive while having little or no positive effect on the hospital’s bottom line. A real world example of this was a large teaching hospital that closed its emergency medicine program to save money but discovered after the fact that multiple resident training rotations into the Emergency Department (ED) required to meet Residency Review Committee (RRC) criteria for several programs would now have to be moved to a different hospital resulting in a loss of GME funding several times larger than the expected savings from closing the ED. This type of cost cutting without a complete understanding of educational, operational and financial implications is beginning to occur at teaching hospitals across the country because there is not a comprehensive way to capture and analyze the multiple events occurring within the residency program as well as provide a reasonable description of "average" performance for a residency program.
A solution to more rational budget validation and cost/financial improvement is now available. It’s called "Operational Benchmarking for GME" and it brings a fact-based methodology to validate the various activities within the program as well as provide the ability to compare key metrics across similar GME programs or within the same program over time. Operational Benchmarking for GME focuses on aligning operational information about the residency program with its educational objectives in order to create a comprehensive view using valid metrics tracking the activities and outcomes required of any successful residency program.
This comprehensive approach allows both residents and faculty to maximize the patient care experience (the underpinnings of clinical training) while still meeting necessary educational objectives. As important, Operational Benchmarking for GME can be used by Administration to determine the relative performance of programs as well as to identify and quantify the benefits of changes in how the programs currently operate. Perhaps more importantly, these changes typically also improve the quality of the educational experience.
A recent benchmarking project identified where significant variability in patient census between multiple inpatient teaching teams was resulting in multiple costs to the hospital including longer length of stay, lost billing and potential accreditation problems related to a high in patient census. Operational Benchmarking for GME not only created a fact-based argument for improved coverage and educational outcomes but also showed how implementation of the improvements identified would not only pay for the investments needed to improve the patient care, operational and educational outcomes but also provide a substantial financial return to the sponsoring hospital.
In another situation, tracking the number of patients seen and treated by residents during continuity clinic sessions revealed a significant variance across residents within the same Post Graduate Year (PGY) level. The minimum and maximum number of patients seen per resident based on PGY level is a program requirement for allopathic programs. Tracking this important operational metric provided real value in assessing the overall quality of the educational experience by the residents. It also showed in some cases, that resident experiences were at or below the minimum requirements for their training level. Although it was a surprising and disappointing result, it allowed faculty leadership to more closely examine clinic operations in order to evaluate the reasons why and determine what steps were needed to reduce variability and help ensure that training met program requirements.