Applying Lean principles to create a high throughput mass COVID-19 vaccination site

A high throughput COVID-19 vaccination site was created using Lean principles and tools. Mass-vaccination sites can achieve high output by creating a standard physical design for workspaces and standardised work protocols, and by timing each step in the vaccination process to create a value stream map that can identify and remove all wasteful steps. Reliability of the vaccination process can be assured by creating a visual checklist that monitors the individual steps as well as by building in second checks by downstream personnel. Finally, productivity can be closely monitored by recording the start and completion time for each vaccination and plotting run charts. With 78 personnel working efficiently and effectively together, a maximum throughput of 5024 injections over 10 hours was achieved. As compared with other published COVID-19 mass-vaccination sites, our site attained threefold–fourfold higher productivity. We share our approach to encourage others to reproduce our vaccination system.


PROBLEM
The COVID-19 pandemic continues in many regions of the world because of low vaccination rates. As a consequence, new variants continue to be generated that can partially escape the vaccine and have prevented countries from achieving and maintaining herd immunity. The highly contagious delta variant originated in India at a time when the percentage of the population vaccinated was only 10% and more recently the even more contagious omicron variant emerged in South Africa at a time when less than 30% of the population was vaccinated. Can Lean manufacturing principles be applied to improve the efficiency and effectiveness of mass-vaccination sites to more rapidly achieve vaccine-induced worldwide herd immunity and to reduce the emergence of SARS-CoV-2 variants?

BACKGROUND
The speed of COVID-19 vaccination is a critical parameter for achieving herd immunity 1 and mass-vaccinations sites were recently highlighted as effective approach for achieving this goal. 2 One small country with a unified system of healthcare delivery has achieved high vaccination rates 3 ; however, detailed descriptions of their vaccination procedures have not been published. Government websites also emphasised the importance of mass-vaccination sites and have provided suggestions for their structure. 4 For example, the United States Centers for Disease Control and Prevention (CDC) recommends conducting 'frequent timemotion studies and staff utilisation reviews to maximise staff roles' 5 ; however, further details are not provided. A UK government website recommends drawing on the logistical expertise of the armed forces, and quotes a Lieutenant Lambert, a Logistics Sea Trainer: 'As advisor to the vaccination operation, (I have engaged) various stakeholders to enable information flow, understanding and early identification of emerging issues. This has included helping design an operational view of the programme by identifying key performance indicators to aid us in spotting potential problems and providing solutions in a timely fashion'. 6 While some are able to do so, many local vaccination sites are unlikely to be able to recruit military logistics or manufacturing experts to continually maximise efficiency and effectiveness.
Given the limited resources and healthcare personnel in many regions of the world, we strongly recommend that Lean principles 7 and continual learning be applied to create high throughput vaccination sites. These approaches can be conveniently learnt through massive open online courses (MOOCs) that describe how Lean principles can be applied to healthcare and are available through Coursera and the English National Health Service. 8 9 We describe in detail a realworld example of how we applied Lean principles and tools to create a mass-vaccination site in the hopes that others will emulate our Open access approach as mass vaccinations are conducted around the world.

MEASUREMENT
The time of arrival at precheck-in and the time that vaccine injection was completed were recorded for all recipients and the lead time calculated for each vaccinee (the interval between arrival at precheck-in and vaccination completion) and representative values from each station were periodically entered into a cloud-based spreadsheet. The number of vaccinations for each 10-hour session was tabulated.

DESIGN: CREATION OF A MASS-VACCINATION SITE
On 5 April 2021, the State of Florida released COVID-19 vaccines to all residents 16 years and older. The sudden eligibility of over 52 000 University of Florida students represented a major operational challenge. To create a mass-vaccination site, we identified a 32 000 square foot space in our football stadium, the Evans Champions Club, designed a vaccinee flow pattern with 8 vaccination stations (online supplemental figure S1) and created a standardised work sheet for each individual work station (figure 1) that defined the movement of each vaccinee and the 5 individual steps or work cycles in the vaccination process: (1) precheck-in, (2) online check-in, (3) fill out the consent form, (4) verify the consent form and (5) vaccinate. To further clarify our procedures, a process map was created that illustrated how these steps were combined to achieve timely vaccinations (online supplemental figure S2).

STRATEGY
We next conducted a series of plan-do-study-adjust (PDSA) cycles to maximise individual productivity, reduce unnecessary motion and eliminate time delays. These improvement cycles were performed on days 1-6 and the resulting process improvements were implemented on day 8. To save time, PDSA cycles were conducted simultaneously by those conducting each work cycle in the vaccination process. Online supplemental table S1 lists the individual PDSA cycles conducted during this period. For example, initially, we had two precheck-in people for each station; however, based on feedback following our early sessions, we learnt that one person was idle half the time. Therefore, we eliminated one precheck-in person per station and found that there were no time delays for the vaccinees. On creation of our value stream map (see above), we realised that the critical step for assuring efficient flow was the vaccination step. Our frontline observations revealed that some of our vaccinators were carrying on prolonged conversations (3-5 min) with their patients creating unnecessary bottlenecks. For our intervention, we created a 60-second script for all vaccinators that included a question about allergies to past vaccinations, the arm preferred for their injection and a summary of the potential short-term side effects of the vaccine (local pain, influenza-like syndrome with possible muscle aches, fatigue, headache, chills and fever) followed by advice to seek medical attention if these side effects persisted for over 72 hours. The script reduced excessive conversations reducing vaccination times. We also conducted multiple PDSAs to arrange the locations of each work cycle within the vaccination station. We discovered that keeping distances to the minimum dictated by infection control practices (6 feet) minimised walk times reducing wasted motion. On day 8, we implemented an improvement bundle that included all of changes derived from our PDSA learning cycles (see the Results section).
Because of the short time frame for the development of our mass-vaccination site (1 week), involvement of patients in the development of our site was not possible. During registration and while being observed following vaccination, patient comments were shared with two of our authors (MNF and MPW) and patient suggestions for improvement were incorporated into our PDSA cycles.
Time observation sheets recorded the five individual work cycles six times for each workstation (online supplemental figure S3). An observer used a stopwatch to time six individual subjects going through a vaccine station. We measured walk times and the time spent to complete (1) precheck-in, (2) online check-in, (3) filling out the consent form, (4) verification of the form and (5) vaccination. All times were entered on an Excel spreadsheet and the mean time for each step or work cycle was calculated. Using these mean times, a value stream map was constructed documenting a realistic ideal state for the process (figure 2). The time required to fill out the consent form was shortened by 2-3 min for those who had filled out the online form before arrival (approximately Open access one-fourth of vaccinees). Because the 5 areas were 6 feet apart, walk times were only 2-3 s. Reconstituting the vaccine was performed by a pharmacist team as previously described following the manufacturer's instructions 10 and represented a separate process that was conducted in a site adjacent to the vaccine stations (online supplemental figure S1).
Vaccinations began on 5 April and occurred each Monday, Wednesday, Friday and Saturday for 10 hours. For the first week, all vaccinations were prescheduled by online appointments, allowing an accurate estimate of the vaccine demand for each day. As the appointments began to decrease, we also accepted walk-ins, 150-200/ day. Our highest vaccine demand occurred during the 1st week and consisted of 5000 appointments/day (online supplemental table S2). To meet the demand of 5000 vaccinations over 10 hours, we calculated that the maximum time for each vaccination needed to be 7.

Open access
To encourage all volunteers to follow a standardised work protocol, each job was completely described during morning orientation. Online supplemental table S3 provides complete descriptions of the four key jobs (precheck-in, online check-in, consent verifier and vaccinator). The precheck-in consisted of filling out a laminated card that served as a visual control to monitor each vaccinee's progress (online supplemental figure S4). In addition to the person's last name and birthdate, the time they arrived at the precheck-in table and time of vaccination were transcribed on the laminated sheet, as was the type of vaccine P for Pfizer, M for Moderna, 1 for first shot and 2 for second shot. A box was checked when online registration was completed and once the consent form was verified. The verifier then directed the individual to one of the two vaccinators who reviewed the allergy history, described possible early side effects from the vaccine, prepared the designated right or left deltoid with an alcohol wipe, injected 0.3 ccs of vaccine and placed an adhesive bandage over the injection site. Individuals with a history of past allergic reactions were then observed for 30 min and all others observed for 15 min as recommended by the US CDC. After completing the injection, the vaccinator entered the time the vaccine was administered, and the time the observation period would end on the laminated card, and finally, signed the consent form.
The laminated cards were collected by an individual designated as the 'runner' who entered the time of precheck-in and the time of vaccination in a cloud-based spreadsheet and then returned the cards back to the precheck-in desk where the card was erased and reused. The cloud-based shared spreadsheet was monitored real time by an administrator. If one station was consistently demonstrating delays, the administrator went to the site, identified the sources of delay and solved the problem with the team.
The runner was also responsible for transporting the signed consent forms to the end-of-the-line inspection station (online supplemental figure S1). At this site, forms were inspected to assure that all parts of the form were completed before delivery to the county health department. Finally, the runners were responsible for continually monitoring vaccine supply and transporting prefilled syringes from the pharmacy, replenishing the stock when the number of syringes dropped below 5. One runner was assigned to each station.
Other support personnel included ushers who directed the vaccinees, that included one person in a golf cart who transported those who had difficulty walking from the adjacent parking lot to the vaccine site. The total personnel required for each session was 78: 20 vaccinators, 20 check-in personnel, 10 consent form verifiers, 8 runners, 6 ushers, 2 end-of-the line inspectors, 8 pharmacists and 4 administrators. With the exception of the administrators, all others were volunteers from the health centre and/or the community.

RESULTS
The average time from precheck-in to receipt of vaccine (lead time) varied from session to session. Following the implementation of our set of improvements based on multiple PDSA cycles, we observed a significant reduction in vaccination times as documented by a shift in the run chart median vaccination lead time from 13.2 min to 9.2 min (figure 3). For days 26-40, we experienced some difficulties recruiting volunteers and the presence of new less experienced recruits during this time may explain the wider variation of lead times. Additional PDSA cycles combined with paid permanent personnel could help reduce the amount of variation in the process and over time achieve an ideal waste-free state.
By creating vaccine stations that minimised motion, standardising job descriptions and using a visual control (the laminated card) to monitor performance, we were able to achieve a maximum daily output of 5046 vaccinations (online supplemental table S2). From 5 April to 14 May, we vaccinated a total of 35 453 individuals. Although our county represents only 1.2% of Florida's population, our vaccination site alone was able to perform 2% of the total vaccinations in the state during this time period, attesting to the effectiveness of our mass-vaccination site.

LESSONS AND LIMITATIONS
Multiple mass COVID-19 vaccination sites have been described in the medical literature 4 ; however, few peerreviewed reports have included detailed logistics or delivery performance measures. One modest-sized site used a large conference room and 5 vaccinators to perform 250-300 vaccinations/day. Vaccinations were administered for 14 hours each day, therefore, each vaccinator performed 4-5 injections/hour or less than one quarter the output per vaccinator of our system. 10 A Open access mass-vaccination site in San Diego took place in a 280 000 square foot parking lot and used 300 as compared with our 78 personnel to deliver 5000 vaccines/day, 11 our approach achieving 3.4-fold higher productivity. Another parking lot vaccination programme in Florida applied PDSA cycles, but did not use value stream mapping to achieve an average output of 8.6 vaccinations/vaccinator/hour. 12 In Italy, vaccine stations were established in a large open parking lot and achieved a similar output of 8 vaccinations/vaccinator/hour. 13 A college in Montana created a vaccination site for students and faculty and conducted two 1-day sessions that vaccinated 807 and 776 individuals. This site achieved a maximum productivity of 8.2 vaccinations/vaccinator/hour. 14 These last three vaccination programmes achieved comparable productivity levels that were one-third of the productivity of our vaccine programme.
We used the work sheets downloaded at no cost from the Coursera MOOC entitled 'Fixing Healthcare Delivery 2.0 Advanced Lean' to design and improve our massvaccination site. 8 The stardardised work sheet was used to design the physical layout of our vaccination station (figure 1), the time observation sheet to record the times to complete the individual work cycles and walk times (online supplemental figure S3), and a value stream map to assess the value of each process and identify opportunities for improvement 8 15 (figure 2). For example, our analysis revealed that filling out the consent form prior to arrival at the vaccination site had the potential to shorten the lead time by 2-3 min, which would increase the number of vaccinations per hour by 20%-30%.
To assure that registration, consent forms and allergy histories were accurate, error proofing was designed into our system: precheck-in was verified by the online check-in person, the vaccinees consent form was reviewed by the verifier, allergy history confirmed by the vaccinator and finally the end-of-the-line inspection for all consent forms was conducted at the inspection desk prior to these forms being sent to the health department.
Mass-vaccination sites are particularly suitable for the application of Lean principles and the US Naval Academy applied Lean principles to improve the efficiency of annual vaccinations for 1200 new naval cadets reducing vaccination lead times (time of check-in to receipt of the vaccine) from 55 min to 11 min. 16 Very recently, a team from the NHS applied the lessons taught in their Lean MOOC to improve vaccination lead times in a small beta test involving 16 individuals. 9 More recently, this same team applied Lean principles to create a massvaccination site that enabled 8 vaccinators to inject 1500 individuals/12-hour shift achieving an output of 15.6 vaccinations/vaccinator/hour. 17 We have now applied this same approach to deliver up to 5046 vaccines over 10 hours using 20 vaccinators achieving an individual output of 25 injections/vaccinator/hour. Like the NHS team, we have provided detailed descriptions of each improvement step so that others inexperienced in Lean principles can emulate our example.
A limitation of our project was the application of our Lean improvement tools to a single vaccination site rather than multiple sites. Second, we depended on volunteers that often worked for only three-four sessions, a condition that could compromise the consistent application of standardised work. However, the first 30 min of each session consisted of a detailed orientation that included breakout sessions for each individual job. The orientation instructors were a stable team of paid healthcentre employees who were present for all the vaccine sessions, a condition that maintained continuity and encouraged standardised work. The impact of a single bundle of improvements rather than multiple sequential improvement steps was documented by a formal run chart. Because the semester ended in mid-May, the time window was narrow for vaccinating a significant percentage of student body and this time limitation required us to accelerate change by combining multiple rapid PDSA cycles into a single bundle of improvement interventions. Finally, because times were entered by the runners, it is remotely possible that they favoured entering shorter times; however, this bias would be expected to have been present throughout the study and, therefore, relative differences between early, mid and late sessions should be equally impacted. One of our authors (MG) is a renowned Lean expert who did not participate in the mass-vaccination site design and served as an objective referee to curb positive bias.

CONCLUSIONS
Our experience reveals that Lean tools can be applied to mass-vaccination sites to achieve high output by creating a standard physical design for workspaces, timing each step in the vaccination process and creating a value stream map to identify and remove all wasteful steps. When Lean principles are fully implemented, experts estimate that productivity can be increased by fourfold and compared with other published COVID-19 vaccination sites who did not employ Lean principles, our output per individual was threefold-fourfold higher. Reliability can also be assured by creating visual checklists that monitor the successful completion of each individual step. 18 These principles can and should be applied throughout our healthcare systems to reduce waste and improve reliability.
To effectively achieve herd immunity in each region of the world, we encourage other mass-vaccination sites to follow our example of applying Lean principles to maximise efficiency and productivity to accelerate achievement of the 'last mile'." 19 Twitter Frederick S Southwick @FS_Southwick