First published: November 2022
Last updated: June 2023
Advances in the science of guideline development methodology continue raising the bar for what “quality” looks like. These advancements contribute to building a healthier and more equitable world. However, advancements in methodology over the past decade have also increased the inherent complexity of guideline development, resulting in continued challenges to maintaining or improving productivity for guideline developing organizations.
Many guideline developing organizations have turned to methodology specialists, either contracted or brought in-house, to improve their productive capacity. But without accounting for how productivity is impacted, most organizations will be unable to progress towards long-term improvements in their capacity to produce quality products.
Productivity: balancing cost, scope, time & quality
Traditional project management often refers to the “Triple Constraint” of project management: cost, scope & time. This theory argues that productivity is relatively fixed in the short-term; therefore, to maintain a defined level of quality, projects must balance these three constraints. Increases in one constraint require reductions in the other two constraints.(1) While there have been discussions in the traditional project management community about what components belong in this triangle,(2) the concept of “quality” is not explicitly considered as a variable that can be manipulated, it is simply a by-product of the other constraints.
Figure 1 - Project Management Triangle: Quality is a product of cost, scope & time (3)
However, the software development community gives us the concept of the “quadrangle” or “devil’s square” which incorporates quality as an explicit constraint.(4) This theory, developed by Harry Sneed in the mid-80s,(5) demonstrates an inverse relationship between quality/scope and time/costs. Increases in quality criteria (e.g. methodological rigor), without appropriate adjustments to scope, will result in increased cost and/or time, to account for the short-term productivity limits of a team or organization.
This Productivity Quadrangle gives us four “levers” with which to manage how we utilize our productive capacity: Quality, Scope, Time, and Cost. These levers also indicate where we can look for opportunities to increase productive capacity.
For example, as the quality of Clinical Practice Guideline development improved over the past decade, the scope of projects was often left unchanged. This resulted in driving up the time and cost required to complete the project (see Figure 2).
Figure 2 - Productivity Quadrangle: Productivity is constrained by quality, scope, cost, and time. As guideline quality criteria increased without adjusting scope, time and cost requirements increased.
Prioritizing Quality: the burden of inherent and incidental complexity
Increases in quality standards creates a demand among guideline developers for better tools and techniques. As a result, there are a myriad of technological applications available to enable more efficient guideline development.(6) However, while intensive usage of modern tools can improve productivity (7), guideline development is an inherently complex process. Tasks which require us to wrestle with complexity include:
Defining the problem precisely enough that we can determine the types of evidence that would provide a solution.
Weighing the various, and variable, outcomes from the body of evidence that was identified.
Balancing socioeconomic, psychosocial, and biomedical considerations when writing recommendations.
As the science of guideline methodology advances and evolves, and the standards for producing “quality” guidelines continually rise, the work of guideline development becomes increasingly complex. While these improvements in quality can lead to increased confidence when applying these guidelines -particularly in clinically challenging scenarios- the processes and practices used to develop guidelines can introduce additional, incidental complexity.(8) In some cases, incidental complexity is introduced by poorly designed processes for conducting a methodology practice. In other cases, incidental complexity occurs when the methodological practices used to develop a guideline exceed the needs of the clinical scenario and questions that arise from it (aka “overengineering” the methodology). Guideline developers should strive to minimize incidental complexities, as these do not contribute to improving the “quality” of a product; they only create more unnecessary work.
Decreasing incidental complexity often requires removing things, in order to simplify. Simplifying the development process for guidelines, when possible and appropriate, can improve productivity by allowing quality to become an adjustable variable, especially when incidental complexity is a result of overengineering. Once guideline developing organizations are able to adjust cost, quality, scope, and time, they can begin meaningful long-term improvements to their productive capacity.
Building Capacity: managing the “cost” of cognitive load
Building capacity is always possible with larger budgets or more volunteers. But we can also build capacity with the resources we already have, through intentional practice. To maximize our productive capacity, we need to create the systems and structures that facilitate the quantity and quality of repetitions that will drive improvements in productive capacity.
Guidelines are a product of knowledge work. Therefore, one of the primary “costs” of developing guidelines is cognitive load; the amount of working memory required to accurately complete the work involved. As the quality requirements of guidelines increase, the cognitive load also rises. These costs are further compounded when guideline products include a wide scope, answering numerous clinical questions.
Figure 3 - Reducing costs to allow increases in scope.
Fortunately, because guidelines are such complex products to produce, we can actually leverage repetition and learning to build greater capacity. This is born out not only in the neuroscience of learning, but in a wide array of industries, in which repetition drives down the costs of work.(9) In human learning, we know that cognitive load declines with repetition as task activities become “chunked” in long term memory.(10) This is both how chess Grand Masters seem to “know” every possible move in a game, and why you didn’t crash your car the last time you were daydreaming on the highway.
Managing Capacity: leveraging scope & quality
For many guideline developing organizations, especially those with limited resources and/or a largely volunteer workforce, their capacity to produce timely guidelines are primarily impacted by scope and quality. As outlined above, quality criterion directly and indirectly influences costs and time, while scope directly influences time, sometimes having a compounding effect.
Figure 4 - Lowering scope to reduce time requirements.
Therefore, when the quality requirements for a guideline project are high, organizations should minimize the scope as much as possible. However, where quality requirements can be lowered, the scope of a project can be expanded without compromising costs.
Maximizing Your Existing Capacity: simplify and manage workloads, carefully!
Making improvements to the productive capacity of a guideline development program also requires enacting limits to the amount of ongoing work, to ensure that progress on the ongoing work doesn’t significantly slow down or stall completely. In Lean manufacturing, this is referred to as limiting the “Work In Progress”.(11)
When workload exceeds capacity, some of the work waits in a queue until the worker(s) can get to it. This is a universal phenomena, whether physically creating a widget in a factory or conducting the knowledge work that guideline development requires. Manufacturers long ago determined that how long work waits to be completed can be calculated by determining worker utilization rates (i.e., busy vs idle) and variations among the work to be completed.
Put simply, the busier a worker is, and the more variability in the work, the longer the work waits to get done.(12)
Figure 3 - Projecting Waiting Time with the Kingman Formula: the impact of utilization and variation on waiting time
These phenomena are evident in the daily work of guideline developers, particularly among clinical experts who volunteer their time. The larger and more complex a guideline project, the longer and longer they take to complete. This is partially a result of greater inherent complexity in the work, which requires more time and effort to address. However, these longer wait times are also driven by workloads with high variability and nearly complete utilization of capacity. This is a recipe for disaster…and a mistake that most guideline developing organizations continue to repeat.
Therefore, the first step towards making long-term improvements in productive capacity is to ensure that capacity is not fully utilized. This allows for ongoing work to flow through the development process more smoothly, by creating the necessary space for the natural variability in the work.
When working with volunteers to develop guidelines, it’s important to assume their utilization is already extremely high. Their professional and personal lives will (and should!) be their top priority, often leaving little capacity for volunteer guideline work. Following the principles summarized above, we can increase the flow of work by reducing the variability in the work that volunteers do.
Once guideline developing organizations are able to adjust cost, quality, scope, and time, they can begin meaningful long-term improvements to their productive capacity.
References:
https://www.coursera.org/articles/project-management-triangle
https://www.pmi.org/learning/library/triple-constraint-erroneous-useless-value-8024
https://www.villanovau.com/resources/project-management/iron-triangle-project-management/
https://stg-tud.github.io/eise/WS15-SE-03-Software_Project_Management.pdf
Gibbs, K. D., Loveless, J., & Crane, S. (2022). A guide to using technological applications to facilitate systematic reviews. Worldviews on Evidence-Based Nursing, 00, 1– 8.
https://pressupinc.com/blog/2014/05/root-causes-software-complexity/
Stalk, G., Hout, T. M. Competing Against Time: How Time-Based Competition is Reshaping Global Markets. Free Press. 2003.
Brown, P., Roediger, H. L., McDaniel, M. A. Make It Stick: The Science of Successful Learning. Belknap Press. 2014.
Comments