Over the past year, my team has worked to develop the foundations for a new operations model for guideline development at our organization. For organizations like ours that develop guidelines, “operations” consist of all tasks and activities which support the intellectual, technical, and mechanical work of a guideline development program. Operations models are uniquely separate from the methodology practices used in developing guidelines. However, both are equally important for organizations to successfully and consistently apply the standards of their field, such as the 2011 IOM Clinical Practice Guidelines We Can Trust, and are crucial for maintaining the value of their portfolio of guidelines.
Since 2020, like many guideline developing organizations, we’ve maintained a living guideline on the treatment and management of COVID-19. As I’ve shared before, the significant financial resources and heroic volunteer efforts required to develop the COVID-19 guidelines disrupted normal ways of developing guidelines and are not sustainable. Therefore, our goal in designing this new operations model has been to operationalize single-PICO flow, and the lessons we learned during the COVID-19 pandemic, in a way that can be more readily scaled across a large portfolio of guideline topics and clinical questions.
The model that we developed is based on these principles:
Scope projects as small as possible; single PICOs are preferred;
Design our website to meet the needs of front-line clinicians;
Release recommendations onto our website first, and later publish in our journal.
Below, I’ll describe how we went about developing and testing this new model, some of the new ways of thinking that we adopted along the way, and the future directions of our work as we begin implementing the model. I’ll also highlight where this work was informed by the Agile Guidelines Development Framework, in both our successes and our failures throughout the journey.
Starting with Mental Models
Mental models are constructs, either shared or unique to individuals, which describe how we think about the world. These models can be found in all areas of human activity, such as the physical sciences, economics, military strategy, and human nature.(1) However, many mental models are invisible to us and serve as the basis for the assumptions we make in our normal cognitive lives.(2)
Most importantly, mental models also shape how we interpret the world and make decisions when interacting with it. As a result, the mental models we habitually use may limit our capacity for solving problems. Conversely, acquiring new mental models can help us view seemingly-intractable problems from a fresh perspective, facilitating the discovery of new insights and leading us to novel and creative solutions.
The longest-standing mental model of “guideline development” is that of generating an academic publication. When guidelines were first developed, primarily in academic settings, print publication was the only method for wide-spread dissemination. Furthermore, as a blend of both clinical and scientific endeavors, it was natural to publish guidelines in a scientific medical journal so they could be widely distributed and implemented in clinical practice. As organizations began developing guidelines more widely, guideline development programs were built around this mental model, including how guideline projects are scoped and disseminated.
The internet has fundamentally changed how we consume information. As a result, nearly all forms of published media, whether audio, video, or print, have been disrupted and forced to evolve to meet the changing demands of consumers. However, despite some notable exceptions during the COVID-19 pandemic, many guideline dissemination strategies have not changed fundamentally from their original modal; they haven’t capitalized on the internet’s ability to quickly and efficiently disseminate new content to a world-wide audience. Therefore, operationalizing single-PICO flow first required us to rethink our mental models of “guidelines,” transitioning from traditional academic publications to modern digital products. We could then begin to address the question of how to get digital products onto our website more quickly.
Leveraging Design Thinking
Design thinking is the application of tools and techniques to solve problems and discover solutions by utilizing curiosity, empathy, and creativity. The tools and practices of design thinking are often introduced in workshops or seminars as brainstorming or interactive activities. As a relatively young discipline, design thinking is quickly becoming integrated into education, business, and government organizations around the world.(3) One of the tools most frequently used in design thinking is a deceptively simple question: “Who is my customer?”
Once the “customer” is identified, practitioners of design thinking move through iterative phases of empathizing with the customer, defining their needs, ideating solutions, prototyping the best solutions, and testing them.(4) However, the application of design thinking often requires a balance of opposing truths, which are exemplified in the following quotes:
“One of the things that I've always found is that you have to start with the customer experience and work backwards to the technology; you can't start with the technology and try to figure out where you're going to sell it.”
Steve Jobs
“If I had asked people what they wanted, they would have said faster horses.”
Henry Ford
Steve Jobs is renowned for designing functional and beautiful consumer technology. In 2023, it’s hard to remember how revolutionary the first iPhone was upon its’ release in 2007.(5) At a time when the tech industry took an “engineering-first” approach to product development, Jobs’ insights into the customer experience forever changed the world.
Henry Ford is also renowned for designing innovative technologies that forever changed the world. His introduction of the moving assembly line revolutionized manufacturing practices around the world, enabling the mass production of consumer goods that underpin the modern economy. Unlike Jobs, however, Ford believed that consumers don’t always know what they want; this belief led Ford to focus on building automobiles, rather than breeding faster horses.
Therefore, when applying the design thinking framework, it’s important to strike a balance between giving the customer what they ask for and seeking new solutions to solve their identified challenges. This is especially true in guideline development, where the typical “customers” are clinicians and patients who may lack familiarity with guideline development methodologies and struggle with interpreting the evidence synthesis upon which guidelines are founded.
When we applied these design thinking tools and practices to our work, one of the key insights we identified from maintaining the COVID-19 guidelines was that users of our guidelines were more than happy to rely on our website as the “source of truth” for the most up-to-date information; the academic journal publication was less important to busy clinicians looking for our most current recommendations. Moreover, survey data revealed that the largest user group of our website was front-line clinicians, and that these users preferred getting access to up-to-date recommendations as they are completed; waiting to release larger batches of recommendations wasn’t serving our customers' needs particularly well. Therefore, our ideation work began by designing a new website layout that was customized to meet the needs of front-line clinicians, and would allow us to post recommendations as soon as they were completed.
Identifying Our Constraints
As we explored how to streamline the process of adding completed recommendations to our website, we quickly identified a major bottleneck: the process of integrating updates or new recommendations into the larger guideline manuscript. Our existing workflow was based on traditional guideline development originating in the 1980s and 90s, where a “guideline” was an academic publication containing a collection of recommendations. As guideline development standards evolved, modern “guidelines” have become a collection of systematic reviews, which are often lengthy, sometimes measuring hundreds of pages. This was certainly true of most of our organization's guidelines, and was clearly a significant burden to our ability to move quickly and efficiently.
However, when we reframed our thinking to look at guidelines as a “digital” product, we realized our recommendations needed to be modular—to be disintegrated as much as possible—to better facilitate future changes and additions. Rather than conceptualizing a “guideline” as a collection of recommendations (and their systematic reviews) published in an academic journal, we saw a “guideline” as individual units: either a single recommendation or the smallest set of recommendations that are clinically meaningful. We then set about designing a website layout that allowed us to modify these “units” in isolation.
Applying the Agile Guideline Development Framework
Prototyping and Building the Minimum Viable Product
The “minimum viable product” (MVP) is a concept borrowed from software development, in which products are built over time via iterative improvements, rather than attempting to produce the best possible version on the first attempt. To develop our new website layout, we first built a prototype based on existing feedback from our primary customer base and input from our volunteer leadership, front-line clinicians themselves. We then conducted user testing via surveys and focus groups, which allowed us to refine the prototype and construct an MVP; the minimal version of the website that met the needs of our core user group (i.e., front-line clinicians).
The MVP website layout was designed to display recommendations as individual sections of the guideline content, much like the previous version of our website. However, the contents within each recommendation section have been streamlined to show only the core features—clinical, quality, and contextual information—that our core user group identified as critical for decision-making. The rest of the recommendation content (e.g., narrative summaries of the evidence synthesis, rationale for developing this recommendation) was removed from the website and made available via a hyperlink, either to a locally-hosted PDF of newly-released content, or to the final academic publication.
We also recognized that each guideline topic and clinical question would likely require custom layouts; therefore, our goal was to develop a set of core features that front-line clinicians would find most useful. Our plan is to continuously refine and improve the website layout over time. As we learn more about our customers’ needs, this layout allows us to add more features and tools to our website, providing our customers with a more robust product and more quickly adapting to changes in their needs.
Testing: Successes and Failures
After developing the MVP website layout, we conducted two types of testing: A/B testing and usability testing. During the testing, we encountered some failures, but also found success.
We used A/B testing (i.e., the internet’s version of randomized controlled trials) to assess the impact of our MVP website on web traffic for our publisher. Although we had moved away from the mental model of academic publications in our new operations model, we remain committed to publishing our guidelines in the traditional academic format because it remains an important communication channel for our organization, our volunteers, and the larger clinical community. Unfortunately, as this was a new endeavor for us, our initial attempt at A/B testing included unexpected variables, rendering the results untrustworthy. Despite this setback, it was a valuable learning experience for our organization, helping to build a foundation of trust and strengthening our capabilities for future successes.
We also conducted usability testing to assess how users interacted with our MVP website. For this, we hired an external organization to conduct side-by-side comparisons of the original and new versions of our website. Although the feedback came from a small sample of possible users, the consistently positive responses to the new layout reassured us that we were moving in the right direction.
Applying the Agile Guideline Development Framework
Updating the Academic Publishing Model
After designing the MVP website, we then had to rethink our approach to publishing our guidelines. If we continued with the existing approach of publishing large, topic-focused guidelines, we would still be faced with the cumbersome work of adding or removing content; the same challenge we are trying to mitigate when posting content on our website.
To generate the modular format that we needed, we split apart the larger, topic-focused guideline document into smaller, separate documents with individual recommendations (or the smallest possible grouping of recommendations). This new format allowed us to avoid the labor-intensive process of updating references, renumbering tables and figures, and revising call-outs every time a recommendation was added, updated, or removed from the larger topic-focused guideline.
Working with our internal journals team and our publisher, we then developed a new manuscript format that allowed us to publish the recommendation(s) resulting from each PICO question as a separate manuscript. Because this new format would result in multiple publications for a single topic, rather than a single large manuscript, we also developed a new “introductory” paper for each topic. This intro paper would describe the new and updated content being published, provide additional key background and contextual information, and describe how the content fits into the broader guideline topic on our website.
In addition, rather than publishing manuscripts in regular issues of the journal, as we’ve done for decades, we decided to publish our guidelines in a separate “supplement” of the journal. This would allow us to release new content onto our website throughout the year, then scoop up this new content into the supplement at regular intervals.
Applying the Agile Guideline Development Framework
Preparing for Implementation
Implementing our new operations model, with a focus on scoping projects as small as possible, also required us to redesign our infrastructure for guideline development (i.e., the way that we form teams to design, develop, and produce guidelines) and our systems for managing workloads.(6)
“An organization is nothing more than a living embodiment of a strategy. That means its “organizational hardware” (i.e., structures, processes, technologies, and governance) and its “organizational software” (i.e., values, norms, culture, leadership, and employee skills and aspirations) must be designed exclusively in the service of a specific strategy.”
Ron Carucci & Jarrod Shappell
Historically, a guideline “project” was scoped to include a large number of PICO questions. Guideline projects were then designed, developed, and produced by a temporary panel of experts; after the guideline was published, the panel was disbanded so that operational resources could be shifted to a new topic. These experts would occasionally be tasked with reviewing the guideline content to determine whether updates or additions were needed; if the topic was prioritized for updating, the panel was reassembled and a guideline update project would commence.
To facilitate our new operations model, with a focus on scoping projects as small as possible, our panel structures needed to be redesigned to facilitate quickly developing individual recommendations, while also maintaining internal consistency across recommendations and retaining historical knowledge of the guideline topic. Therefore, we redesigned our panels to consist of two components: (1) a standing “steering group” that would help oversee the guideline topic, identify when recommendations needed to be developed, and be involved in the review and approval of all recommendations; and (2) ad hoc “work groups” tasked with designing, developing, and producing an individual recommendation project.
Our new operations model also required us to rethink how to measure and manage our operational capacity. Historically, potential guideline projects underwent a prioritization process every few years for selecting guideline topics to update; panels were formed to work on these prioritized topics, but often without taking into full account the remaining workload of the ongoing guideline projects. Over time, and as guideline development became more complex, this resulted in overloading the existing operational capacity with far more work than it could handle, severely disrupting the flow of work and extending the time and costs of all guideline projects.
To help us determine how best to manage the workload of our teams, we mapped out the “value stream” of our guideline development operations (i.e., each process the organization follows to design, develop, and produce a guideline). This mapping allowed us to visualize our guideline development systems, identify the key team members involved in each major process, and modelize our guideline development timeline. From this work, we were also able to estimate the workload required to complete a PICO, from the initial request to publication. This information, along with the new team structures, will allow us to ensure that workloads do not exceed our capacity, and better prioritize ongoing and future guideline projects.
Applying the Agile Guideline Development Framework
Future Directions: Scaling the New Model
As we implement this model over the next few years, our next major tasks include determining how best to prioritize our backload of guideline projects and identify the right timing to initiate new projects. To help us work through this backload, we will also need to experiment to find the optimal use of our volunteer resources and how best to appropriately size our guideline development teams according to the volume and complexity of the work.
Future Applications of the Agile Guideline Development Framework
We will also be continuously refining our work processes to enhance efficiency and quality. This will include developing and updating Standard Operating Procedures for all key program management activities, including prioritizing our guideline portfolio, forming and maintaining teams, and guideline development. We will also explore and experiment with existing and new software tools, including levering AI, balancing efficiency improvements each step in the development process with the overall flow of work.
Future Applications of the Agile Guideline Development Framework
These are some of the larger necessary changes that we’ve already identified. However, managing guideline development programs is a highly complex activity, and it is likely that we will need to rethink and make adjustments to every aspect of our program. Looking forward, I expect that this work will take the better part of a decade, and that we will continue to be met with both successes and failures.
As I tell my team, with both excitement and trepidation, “The work has only just begun!”
References:
コメント