Fear of stigmatization, self-stigmatization, and insufficient information can lead to secrecy, reduced help-seeking, lower self-esteem, and lower self-efficacy among people affected by suicidality or suicide. Therefore, we developed an online suicide prevention program aiming to improve knowledge about suicidality and suicide stigma.The Ripple Effect, a German team comprising people with lived experience of suicide, researchers, and clinicians was established for developing an online suicide prevention program. Therefore, we oriented on guidelines for evidence-based health information, for reporting on suicide, and on dealing with suicidality. The lived experience team discussed and developed concept, structure, and content of the program. This manuscript presents summaries of protocols from 16 team meetings and 3 written text reviews to outline the program development process. A summative evaluation 3 years after program development began was qualitatively analyzed based on thematic analysis.Inspired by the Australian program Between 2018 und 2021, the lived experience team (n = 10) discussed possibilities of support in suicidal crises, attitudes towards suicide, content, and design of the online program. In a structured process, six members of the lived experience team reviewed the content. Eight persons shared their lived experience of suicide in video reports by focusing on constructive ways of dealing with suicidality or a loss by suicide, conveying hope and encouraging people to continue living. Team members recommended greater public and patient involvement from the application stage, as well as more financial and personnel resources.Through contributions to discussions and text reviews, the lived experience team shaped decisions in the program development process. While involving persons with lived experiences of suicide, it is important to consider that suicidality is 1. emotionally challenging, 2. a stigmatized issue, and 3. that the aspect of safety must be a priority. A distinction must be made between the duty of care based on actual risk and inappropriate overprotection. Hereby, transparency, autonomy, and a clear structure appeared to be helpful. For further research, we recommend a structured formative review process of the development of the program. Additionally, we recommend discussing the purpose and the specific design of the evaluation with a lived experience team in advance.Trial registration German Clinical Trial RegisterDRKS00015071 on August 6, 2018.The online version contains supplementary material available at 10.1186/s40900-021-00307-9. A team of persons with a lived experience of suicide and researchers jointly developed an online suicide prevention program. This online program addresses the taboo and stigmatized topic of suicidality from the perspective of people with lived experience. In our team are people who tried to take their own lives, thought about suicide, and people who lost a close person by suicide. For the program we wrote texts on suicidality and suicide on a scientific basis. In videos the team shared their experiences with suicide. The team told what helped them to continue living and gave them hope. Online program participants can read these texts and watch these videos. Participants can learn about suicidality and are encouraged to seek help if needed.During the development of the program, the researchers created a working atmosphere characterized by respect, empathy, transparency, and openness. Dealing with suicidality can be emotionally stressful. The team felt comfortable and safe. The team was proud of the developed online program.We describe strengths and weaknesses of the program development. We did not evaluate the involvement during the program development. If one wants to do this, the purpose should be explained to the team. The team should have a say in the design, such as what questions are asked.Our program gained enormously from involvement of people with a lived experience of suicide. Involvement is possible and necessary even with complex and sensitive topics such as suicidality. Antistigma work should involve those affected. Involvement is important to create a credible program.The online version contains supplementary material available at 10.1186/s40900-021-00307-9. Suicidality is a complex phenomenon that can be an expression and/or symptom of a mental illness, a consequence of a  illness, or a consequence of stigmatization –3. Most The use of online interventions can help individuals to inform themselves about mental health despite the stigma and taboo of a topic, to find local treatment options, and to prepare for contact with health care professionals –22. Howewww.psychenet.de [8 Lives-Lived experience reports and facts on suicide is accessible via https://8leben.psychenet.de. The program aims to contribute to this e-mental-health portal byThe German e-mental-health portal henet.de , 29, 30 (1) improving knowledge about suicidality by educating, raising awareness, and presenting professional help services,(2) reducing self-stigma and/or perceived suicide stigma of persons with a lived experience of suicide to enable them to open up to others about their condition and to seek help, and(3) increasing self-efficacy expectations of being able to cope with psychologically demanding situations.8 Lives program does not focus on symptom reduction, such as Internet-based cognitive behavioral interventions aimed at reducing suicidal ideation [The ideation . We clas8 Lives has been developed based on the Australian online program The Ripple Effect [8 Lives was developed by researchers and mental health experts in collaboration with persons affected by suicidality as well as relatives of affected persons. The content of the program was translated into German, broadened to a general population , and adapted to the German cultural context. From our perspective, by including people with a lived experience from the same cultural context as the program's target group, it is possible to design the program in a culturally sensitive way and to consider attitudes toward suicide in the cultural context. In addition, before the program 8 Lives was developed, the project team conducted a representative population survey in Germany to determine attitudes and knowledge about suicidality [The Ripple Effect to obtain firsthand background information and experiences made. The rough outline of The Ripple Effect was adopted, while there were content and conceptual changes in 8 Lives.The unguided online program e Effect –34 whichcidality , 36. FurTo our knowledge, there are no other online programs involving people with a lived experience of suicide that aim to reduce suicide stigma and increase knowledge on an individual level for five different target groups .Since people should have influence on research that affects them, the involvement of patients and the public (PPI) has become a prerequisite for applied health research . This alStigma of Suicide Research Team includes people with lived experience, suicide prevention advocates, mental health providers, loss survivor therapists, and researchers.In suicide stigma research, a community-based participatory research approach has been adopted to identify stereotypes, prejudices, and discriminations against people who attempt or die by suicide  and to dInvolving mental health service users can add substantial value in the development of mobile applications for mental health , e.g. byLittle is known about involving people with a lived experience of suicide in clinical research. This may be attributed to the fact that suicidality is an extremely sensitive topic and is therefore generally often defined as a study exclusion criterion (for both persons involved in research and for participants), and that suicidality as a complex phenomenon can be defined less clearly on the basis of a categorial diagnosis such as a mental illness . LittlewThe aim of this publication is to share our insights into the involvement of people with a lived experience of suicide in the development of an online suicide prevention program. Following the Guidance for Reporting Involvement of Patients and the Public  is described elsewhere [The online program lsewhere . QualitaThe project team comprised expertise from different backgrounds: 1. scientists specialized in medicine, psychology, and social science, 2. clinicians working as psychologists, psychotherapists, or physicians, and 3. persons with a lived experience of suicide (“lived experience team”). The lived experience team was involved in all phases of the development of the online suicide prevention program except the grant application.Irre menschlich Hamburg e.V. [Irre menschlich organizes information, encounters and prevention projects on all aspects of mental health. One of its goals is to promote greater tolerance in dealing with others and sensitivity towards oneself. Twelve persons from Irre menschlich were interested in participating in the project 8 Lives. Criteria for lived experience team membership were.The lived experience team was established using a purposive sampling strategy. At first, the project idea was presented twice to members of the trialogical association urg e.V. , 51. Thi(1) having a lived experience of suicide i.e., suicide attempt, suicidal thoughts, loss of a close person by suicide, or caring for a close person who is suicidal,(2) being currently in a stable situation, i.e., unlikely to get into a crisis because of the project involvement, and(3) being at least 18 years old.Potential team members were given a participant information form and provided written informed consent to participate. Additionally, a licensed psychotherapist assessed the individual's stability as sufficient for project participation. Lived experience team members decided the extent of their participation and had the right to revoke their participation at any time. This also included recalling the provided video and text material.There were four options of participating in the program development that could be flexibly combined. The lived experience team members could be involved.(1) in the development of concept, structure, content, and design of the program,(2) in the review of the text material,(3) in personal video or written experience reports, and(4) in creating short “digital postcard messages” that can be read by online program participants.Team members were reimbursed for their participation; 80€ per text package that was reviewed, 80€ per person for a written experience report, and 800€ per person for a video interview. One member of the lived experience team had a scientific background and was also part of the project team (JB). Her work comprised 10% of a full-time-position and was reimbursed according to regular wages of public service.aims, concept , content, language, design, and structure of the online program.Discussions of the lived experience team meetings and their results were recorded by written protocols. These were drawn up after the meeting and were sent to the lived experience team by e-mail; team members could add content or make amendments to the protocol. Based on the protocols, we summarized discussion points , age (range = 19–73 years), working status  and profession . Most team members reported personal suicide thoughts, suicidal behavior, or suicide attempts (n = 7). Three members were close persons of affected people; one person was close to a person affected by suicidality, and two persons lost a parent through suicide. Four persons were affected by suicidality or suicide in more than one way of the above named. During the two-year development period of the program, one person of the lived experience team decided to be admitted to a psychiatric hospital because of suicidality. This person did not relate his/her crisis to the program development and stated in retrospect that in the long run his/her active involvement led to lower suicidality. Another person described depressed mood and lack of drive within a limited period but did not associate this with the program development either.After the initial meeting, two persons decided against further participation due to lack of time. The final lived experience team (n = 10) differed regarding gender  was sent by e-mail to all active members of the lived experience team 3 years after the start of program development, approximately one year after the program went online, to reflect on the development process of the online program. To give examples of summative evaluation questions we used on topic 1: “From your point of view, what went well/badly in the collaboration within the 8 lives team for the development of the "8 Lives" program? What was helpful/less helpful/not helpful at all?”; topic 2: “What would you recommend to another group if they wanted to develop an online program on suicidality / breaking the taboo? What would you do differently if we started again with the development of "8 Lives"?”, topic 3: "What are your thoughts/opinions on formative evaluation during program development?; “From your point of view, what speaks for/against an evaluation during the development of the program?" One of the original ten lived experience team members died of a somatic condition. Two team members did not participate in the project after completing their tasks due to time constraints. The written free text responses from the lived experience team (N = 7) were qualitatively analyzed based on thematic analysis by Clarke & Brown [A questionnaire with 16 open-ended questions on three topics  or a psychotherapist (SL) led the meetings. One permanent contact person (MD) was defined for any requests and coordination. At the end of each meeting, pending tasks like text reviews, preparation of digital postcard messages, or interview questions were distributed. Finally, an outlook for the next meeting was agreed upon. The lived experience team received a protocol of the meeting by e-mail. A few days before the next meeting, a reminder was sent out with the planned meeting content. The lived experience team members could contact the project team  by e-mail and telephone between group meetings. There was also the possibility to make personal appointments with a project team member , which were mainly used for the preparation of the videos. A formative evaluation of the meeting, e.g., a structured feedback questionnaire after each meeting, did not take place.In the following, results from (1) the lived experience team discussions on the program development, (2) the text reviews, (3) video reports on lived experience of suicide, (4) the content of the program, and (5) the results of the summative evaluation three years after starting the project are presented. Please note, that only an excerpt of the data can be presented below. Therefore, we highly recommend reading the supplementary files , which present the discussion topics that arose in the team meetings on program development (A2), the results of the program text review (A3), questions prepared for video experience reports (A4), the entire content of the online suicide prevention program (A5), and the reflection on the development process (A6) in detail.Several topics regarding the online program development were discussed during the lived experience team meetings . Despite these experiences, we agreed to promote in the online program that it is worthwhile to seek help.The team decided to create shorter video sequences and show video sequences on various topics of all eight lived experience team members in all five variants of the program, with the purpose of approaching the topics from different angles. Participants of the online program who e.g. indicated in the beginning to have lost a close person by suicide can decide to watch videos of people affected by suicidal thoughts and vice versa. However, we decided to make several distinctions based on the experience with suicidality a person has stated—because we considered that some statements may be more or less appropriate/helpful for some target groups.One suggestion during the lived experience team meetings was to address the general attitude towards dealing with death and dying in society in the program. In the joint consensus, we decided against this idea because it would have gone beyond the scope of the program.Shame or fear of shame can lead to withdrawal behavior of persons affected by suicidality. The lived experience team did not find it helpful to "push" to disclosure and decided to leave the decision to the individual (autonomy). Therefore, respect for non-disclosure resonated in texts and video messages while at the same time the importance of support in a suicidal crisis and help offers were described.The use of humor in the online program was controversially discussed. Some stated that humor helps in dealing with suicidality. In the online program texts, we have dispensed with the element of humor, since we do not know how the anonymous participants feel about these messages. We have agreed that the digital postcard messages may contain personal statements with black humor. Also, in the lived experience video reports humor as a strategy can be explained.The wish of the lived experience team was that only people who are seriously interested in the topic should have access to the program. Also, for this reason we implemented a login with email address.Information texts on suicidality, help options, and stigmatization were created and discussed in a first draft . A total of three text packages (between 9 and 15 letter size pages) were sent to the lived experience team by e-mail or mail. Five to six persons reviewed each text regarding comprehensibility, complexity, brevity, conciseness, quality, and completeness. All written comments were sent to the coordinator (MD) by a set deadline. Furthermore, it was possible to make oral comments by phone. Feedback was discussed in the next lived experience team meeting with the whole group and incorporated in the program texts. Overall, texts were described as comprehensive and concise. Some parts had to be adapted for the different target groups. In addition, simpler wording was suggested and incorporated. A summary of the written feedback of the lived experience team during the text review is provided in Additional File After the first team meeting, two persons decided against disclosing their lived experience of suicide in a video report. One person decided to write an anonymous experience report. The decision not to disclose the lived experience in a video was made because it would have indirectly revealed the experience of the close suicidal person. The other person decided not to speak publicly about his/ her experience. Eight persons had decided to disclose a part of their lived experience of suicide in a video report. During team meetings, they discussed the concept and formulated possible questions for the videos  to have enough time for reflection before and after the videos. We agreed on shooting the experience report in one-take. The participants could then add or re-record an answer. There were no time limitations concerning the video length. The edited video sequences  were counterchecked and approved by the lived experience team member.8 Lives – Lived experience reports and facts on suicide contains eight chapters composed of video reports about lived experience of suicide, fact sheets on suicidality, exercises based on cognitive-behavioral models, and worksheets .The lived experience team tested the final technical implementation of the online program 1: evaluation of involvement: enabling and hindering factors, 2: lessons learned from program development, 3: opinions on the lack of evaluation during program development) are presented in additional file All seven members of the lived experience team who were still actively involved in the project completed the summative evaluation questionnaire. The summative evaluation took place in March 2021, three years after the initial team meeting. The entire results of the summative evaluation  as empathetic and caring. The possibility of contact with project members  in potential  crises, beyond meetings on different communication channels  created a safety feeling among the team. The team described the continuous information about the team meetings and the status of the project as positive as well as the clear focus of the team meetings. The time between the team meetings was described as important, also because there was an opportunity to prepare and follow up on the different topics. The team members positively emphasized the varied possibilities of involvement, the extent of one's own involvement being flexible as well as the financial reimbursement.As a result of the project, the team members described various personal changes, which were evaluated as positive: Dealing more intensively with the topic of suicidality and suicide through the project, becoming aware of one’s own self-efficacy and ways of dealing with suicidality, developing a greater understanding of the function which suicidality has for each and every one individually, and having grown through the project.Some team members reported ruminating on issues after team meetings, as well as having unpleasant feelings or thoughts activated in team meetings. Team members described this in part as emotionally taxing, in the long run, more as an internal coming into motion. The team members described that the strain could be well absorbed by the space in the project and team meeting framework. Being involved in the project and team where not much explanation was needed, was especially helpful as well as one-on-one conversations with lived experience team members or project members . To process the feelings and thoughts, some team members could additionally use psychotherapy with a psychotherapist independent of the project.The time required for the project was mentioned as a hindrance: One team member mentioned that the project took too long. Two other team members thought that there was sometimes too little time in the team meetings or that there were too few team meetings. Two team members would have liked to have additional meetings that were not project related. One team member described difficulties with the dual role of a project team member and sometimes felt excluded, rejected, powerless, and not taken seriously enough in team meetings. The program was offline for the time of the evaluation and revision of the program. All team members reported back that the continuation of the program was not discussed early enough, which led to frustration.Lessons learned from the perspective of the lived experience team (N = 7) three years after starting the project are presented in detail in additional information 6.2 . A formative evaluation was also seen as a possible relief for the researchers moderating the group . As a possible disadvantage of a formal evaluation the time factor was mentioned. Moreover, some team member stated as a disadvantage for a formal evaluation the possible interruptions of the  workflow. The team may have focused on teamwork instead of the issue of suicide/suicidality and program development . A reluctance to use questionnaires for evaluation, especially tick-box ones, was also described. Formative evaluation was further described as a kind of "pseudo" feedback that could have exerted pressure, as well as creating a feeling like a "guinea pig". In the post evaluation discussion of the summative evaluation results, it became clear that it would be important to first explain the purpose of a formative evaluation (e.g. improvement of the work process and/or research) and to decide together as a team for or against the evaluation. If the team had decided to implement a formative evaluation, the team would have liked to discuss and determine the concrete form of the evaluation together . The majority of the team members spoke out in favor of an oral evaluation with jointly defined questions that should be formulated in an open manner. According to the team, the evaluation should have not taken place too frequently . The evaluation results should have then been discussed orally in the team.Additional to the summative evaluation of the lived experience team, we compiled reflections from a researcher’s perspective (N = 3) on this in detail in Additional File The lived experience team and project team agreed that in case suicidality increases, the project team should be informed in addition to the person’s health care professional. We found it helpful to discuss the possibility of a deterioration of a team member’s condition at the first meeting. One person could not attend one lived experience team meeting due to an inpatient clinic stay . We discussed with the person and the team how we should deal with the specific situation and decided for a team meeting without the person. Retrospectively, we should have discussed how to deal with a possible inpatient clinic stay of a team member at the beginning when the lived experience team was formed because non-attendance due to an inpatient stay has an impact on the group. We established team rules, e.g. not to send potentially ambiguous e-mails regarding suicidal ideations or behavior to the project team, as it will be treated like an emergency which happened one time during the program development.From the researcher’s perspective, a good working atmosphere, trust due to the dual role of a scientist with a lived experience, good group cohesion, a clear structure, autonomy, transparency and continuous contact, accessibility of the project team, appreciation of opinions and ideas and shared consensus, giving feedback on text reviews and status updates, and being informed about possible risks of participation were helpful in working together with the lived experience team to develop an online suicide prevention program. It was very important to avoid peer pressure, e.g., on disclosure.8 Lives, the video reports of the lived experience team tell “ups and downs” regarding dealing with suicidality or suicide while focusing on hope and overcoming a crisis. The lived experience team members could have a role model function and be beneficial for online program participants since they show how to talk about suicidality or suicide and offer ways of dealing with it in a constructive way [Ten people with a lived experience of suicide were involved in the development of an online suicide prevention program aiming to enhance knowledge and to reduce stigma. We aimed at bringing participants of the program indirectly into video-based contact with people affected by suicidality (either self-affected or affected as a close person). In tive way . Since aOur project showed that involving people with lived experience in research projects is possible and enriching, even for complex and sensitive topics such as suicidality. For the developed program, video reports of eight different team members with lived experiences of suicide as well as one written experience report created different possibilities of identification for online program participants. In addition to the contribution in the video reports, the input of the lived experience team essentially shaped the decisions in the development of the program at various levels: from overarching conceptual decisions such as the selection and level of depth of topics or how to present videos to participants, to detailed decisions such as wording of sentences.While international guidelines for public patient involvement exist , there iIn summary, a trustworthy, friendly working atmosphere was established both from the perspective of the lived experience team and from the perspective of the researchers. The lived experience team described working with each other as equals and always felt safe during program development—despite the emotionally taxing topic of suicidality.One issue that should generally be considered when involving people with lived experience  is safetDifferent memories related to suicide and suicidality came up in the team meetings and the team members needed to talk about these memories. With the topic of suicidality, this can also be emotionally challenging for the researchers who moderate the team meetings. We found it important to give space to these memories and personal feedback and then also to find the way back so that the team could continue to work together on the program development.The lived experience team did not feel sufficiently involved in the discussion on the continuation of the online program. The entire team wanted the online program to be available for users for an unlimited period. This continuation was not covered in the grant. In hindsight, the researchers would have involved the lived experience team earlier in the discussion on continuation of the program. After identifying the misunderstanding between the researchers and the lived experience team , we held several team meetings and benefited from the lived experience team's opinions for further development, e.g., on participants’ feedback, while gaining more understanding on the researcher team's side about importance of data evaluation. The project team is implementing the continuation without reimbursement of efforts or wages. If possible, we recommend that a possible continuation of a project is already considered when applying for funding . We recommend separating program development and program evaluation in terms of personnel.Some lived experience team members described that they were proud of their involvement in the project. Moreover, personal changes took place because of the project, such as becoming aware of one’s own self-efficacy and ways of dealing with suicidality/suicide, and a better understanding of the function which suicidality serves for each and every one individually. The researchers initially had not been aware of how important, empowering, and destigmatizing the project could be for the lived experience team itself (since it focused on stigma reduction among program participants). Our hypothesis is that through co-creation and/or involvement in such a suicide prevention project, in addition to initiating reflective processes, internalized suicide stigma of team members might be reduced. Potentially, this could be accomplished by sharing the lived experience in a safe setting among peers with a focus on empowerment and recovery, as well as the shared aim of helping others affected by suicidality.The lived experience team consisted mainly of persons who have had suicide thoughts, have exhibited suicidal behavior, or have had suicide attempts. Persons who have lost a close person by suicide were less represented as well as the perspective of persons who are worried about a suicidal person. Although all perspectives were represented in our team by at least one member and some team members have been affected in more than one way, the diversity of experience portrayed in our program could have been further increased. Since the relationship to the person who died by suicide plays a role in processing a suicide, a possible expansion of our program would be to add experiences from people with a different relationship status to the deceased person . The perspective of persons who are worried about a close suicidal person could only be represented to a limited extent. From our point of view, it could be reasonable to present both perspectives, i.e., of the person affected by suicidality as well as of the close person. Persons who have a different cultural background and may have a different view of suicidality were given little consideration in the program.Irre menschlich Hamburg e.V., some team members had previous experience in antistigma work and knew each other in advance. The latter enabled good group cohesion, trust, and support to be established quite fast. On the other hand, people within the same association might have similar views on certain topics, which could limit perspectives. To reduce this potential bias, people independent of an association, from other associations, or people affected by the topic for the first time could be included in further projects. Since the program also addresses people who are generally interested in the topic, but are not specifically affected, it would have been reasonable to include them in the development as well.Through participating in the trialogical association The entire project focused on the evaluation of the online program in a pre-post design, not on evaluating the participatory development process. The evaluation of program development could have been more standardized, planned further in advance, and more operationalized –64. Dist8 Lives involving a lived experience team by summarizing topics discussed in the lived experience team meetings, showing results of the text review on information on suicidality, and describing the creation of lived experience of suicide video reports. As a result of a summative evaluation, we recommend a formative evaluation of the development process, of which the purpose and design is co-determined by the people involved. Involving people with lived experience of suicide is possible and enriching for research projects and can also be empowering for the people involved. From our perspective, for the development of antistigma programs the involvement of people with lived experience is essential to create a credible program.We described the development process of the online suicide prevention program Additional file 1. GRIPP2-SF.Additional file 2. Discussion of the lived experience team on program development.Additional file 3. Results from text review.Additional file 4. Questions for lived experience reports.Additional file 5. Content of the online suicide prevention program.Additional file 6. Reflection on the development process. 6.1 Evaluation of involvement: enabling and hindering factors in program development. 6.2 Lessons learned from program development. 6.3 Opinions on the lack of evaluation during program development. 6.4 Reflections on involving a lived suicide experience team in an online suicide prevention program. Development from researchers’ perspective.


The establishment of genomics in health care systems has been occurring for the past decade. It is recognised that implementing genomics within a health service is challenging without a system-wide approach. Globally, as clinical genomics implementation programs have matured there is a growing body of information around program design and outcomes. Program structures vary depending on local ecosystems including the health system, politics and funding availability, however, lessons from other programs are important to the design of programs in different jurisdictions. Here we describe an adaptive approach to the implementation of genomics into a publicly funded health care system servicing a population of 5.1 million people. The adaptive approach enabled flexibility to facilitate substantial changes during the program in response to learnings and external factors. We report the benefits and challenges experienced by the program, particularly in relation to the engagement of people and services, and the design of both individual projects and the program as a whole. The ability to translate genomics into clinical practice is complex and requires tailored system-wide changes to health organisations3, and may require a re-assessment of how genomic services and value in healthcare are evaluated.Information from genome sequencing can be used to inform patient care and is at a point where its mainstream use in various medical specialities is possible7. The level of interrelated system-wide changes required for the incorporation of genomics into routine care likely results in fragmented and costly adoption of genomic services8.To realise the potential of genomics in healthcare, the challenges it poses need to be addressed systematically. Challenges include: increasing patients and community awareness of genomics and familial implications; training a clinical workforce to apply genomics in clinical practice; establishing infrastructure for genomic testing, analysis and information systems for genomic data management, sharing, integration and long-term data usage; providing evidence on the value of genomics to the healthcare system and patient care; and addressing concerns about ethical, legal and social issues raised by the application of genomic medicine13. Each program has set out to address different issues related to the uptake and management of genomics in clinical settings, fundamental research, and translation of genomic research. These approaches have varied, reflecting local requirements, context and end goals.There has been a global investment in the implementation of genomics into healthcare systems, with multiple countries or jurisdictions establishing programs9 as well as individual state/territory-based programs13. These programs have had significant success in creating political and health systems support for clinical genomics in all Australian public health systems. However, the Australian context of a federated healthcare system has resulted in implementation at varying levels of maturity, reflecting each jurisdiction’s healthcare system and research environment.Public healthcare in Australia is a joint responsibility of the Federal Government and the seven states and territories. Each state manages its own public health services. In the last five years investment in genomics has occurred through a single national research program 15 utilising a budget of $AU15.6 billion (2019–20). This publicly funded system provides healthcare for a population of 5.1 million people over a geographical area seven times the size of the United Kingdom.This paper describes the genomics program in one state, Queensland, where health services are governed and delivered through 16 Hospital and Health Services as statutory bodies16 with the stated objective “to demonstrate the value of genomic medicine in everyday Queensland healthcare”. From this investment, the Queensland Genomics Health Alliance (Queensland Genomics) program was established. Queensland Genomics was a 5-year program (2016–2021) established to bring together the health system, research, academia and consumers  if the original program structure was continued, it was unlikely to affect sustained uptake of genomics into the health system once the program ended, (2) the first round of projects had insufficient engagement with the health system as the program was geared heavily towards research rather than healthcare transformation, and (3) the original plan modelled Queensland Genomics to other genomics alliances, but this assumed that these programs operated in the same context and with the same funding structure as Queensland. Therefore continuing this model would not meet the needs of the Queensland health system.In response, the program was restructured prior to the start of the Strategy Round to have an emphasis on health implementation with a focus on Queensland Health ownership of outcomes, and sponsorship of projects and initiatives. The updated aim of the program became to “accelerate the translation of genomics into clinical practice”. The restructure included changes to: governance structure and responsibilities; project selection processes; Queensland Genomics involvement in the management of projects and strategic interactions with Queensland Health; and requirements for projects to engage with health services. This new process was designed by the Queensland Genomics business team in consultation with Queensland Health Executive leadership and approved by the Governance Oversight Group.The restructure resulted in Queensland Genomics moving from its initial research, demonstration and ‘discovery’ orientation to a focus on; (1) sustainable clinical genomics implementation and incubation, (2) clinical innovation with potential to match future needs, and 3) targeted health system capability building. This is reflected in the main outcomes of the program . This enabled the program’s operations to function independently with a governance oversight group that reported to Queensland’s Department of Health. The main roles of the Governance Oversight Group were approving Queensland Genomics processes and endorsing decisions. The Queensland Genomics business team were not genomics experts or clinicians and were recruited to provide leadership in health transformation, business and technology strategy, communication and engagement, and innovation. They supported the individual projects through project design and delivery, contract administration, and establishing connections between projects and Queensland Health. As needed, specialist expertise was drawn upon using ad hoc advisory panels and the program’s community advisory group Fig. .Fig. 3QuFollowing the governance review, Clinical projects were led by a practising clinician and, where possible, Clinical and Capability projects were sponsored and led by Queensland Health or Hospital and Health Services Tables  and 2. EThe Queensland Genomics program had an overarching emphasis that projects should be patient-orientated. The Queensland Genomics Community Advisory Group (CAG) was established at the start of the program Fig.  and cons22.In the Discovery Round, four Clinical project grants were awarded in areas of cancer, common chronic diseases, and infectious diseases Table . Five Ca26. Structuring projects into portfolios enabled sharing of resources, processes and learning, enabling sustainable implementation. Consequently, the individual projects contributed to broader portfolio-specific objectives around: implementation of a state-wide workforce model for genomics; establishment of a genomics clinical advisory service and a molecular tumour board for cancer; proposed data integration solution for the integrated electronic Medical Record (ieMR); and providing evidence for genomics as a routine tool in infectious disease outbreaks.In the Strategy and Legacy Rounds, 11 Clinical projects were funded. These projects were organised into three clinical portfolio areas; Cancer, Whole-of-Life, and Infectious Diseases (Table These portfolio-specific objectives came from commonalities identified in Clinical projects’ plans, or needs identified by capabilities projects, health services or the Queensland Genomics business team. These objectives fostered collaboration by building and funding co-dependencies between projects and by association with their clinical services. Thereby, lifting capability, creating value for each partner, and creating inter-departmental connections. This approach did require additional time and effort to develop collaborations and, in some instances, was met with reluctance from project members. Implementation of this approach required additional resourcing at the program’s administrative level, plus support and funding for the project partners.35. In addition, seven projects designed by the CAG were funded in the Strategy and Legacy Rounds (Table 36.Sixteen Capability projects were commissioned within the Strategy and Legacy Rounds Table . Projectds Table  that focThe aim, and ultimate achievement, of Queensland Genomics was the development of capacity, infrastructure and policies to enable safe, appropriate and consistent application of genomics in Queensland’s public health system. The choice to design the Queensland Genomics program using an adaptive management approach had a number of benefits particularly to the program’s flexibility and adaptability. However, this direction was not without its challenges. The lesson, and actions suggested, based on Queensland Genomics’ experiences are summarised in Table The Discovery Round was designed as a time to capture lessons from program activities, and utilise this information to assess and make changes to the program direction. Critically, the governance review found that the program was unlikely to affect sustained uptake of genomics into Queensland’s health system once the program ended if it continued along a research orientated path. This led to a program-wide restructuring whereby the program envisioned a future where genomics was business-as-usual in Queensland’s health system. This led to a program of work in the Strategy and Legacy Rounds that was customised to the health system and its specific health implementation needs, with ongoing smaller adjustments as new opportunities were identified. The recognition that the program’s initial direction would not facilitate health system implementation and the capacity to pivot the program in response to the recommendation was only possible due to the original program design. The design provided time and assessment mechanisms to determine the most appropriate direction for the program.Health system ownership of program activities was imperative to ensure post-program uptake of genomics. For successful implementation within a health system upskilling and supporting providers needed to occur across multiple interrelated services. For example, Clinical projects were run by medical specialities who at the completion of the project may be responsible for test ordering and management of patients, but sample processing and sequencing are the responsibility of pathology services, and clinical data management is the responsibility of information systems. The adaptive approach allowed for health services’ needs in the uptake of genomics to be identified whilst projects were running, and project plans to be revised accordingly through engagement with those future owners.2. It is an inherent issue in the adoption of new technologies in complex healthcare systems that stakeholders without pre-existing knowledge or involvement are difficult to effectively engage. The Queensland Genomics program was initiated with stakeholder consultation with predominantly researchers, and clinical specialists, services and administrators that already had involvement with clinical genetics or genomics. However, it has been shown that high-level stakeholders, such as healthcare executives and regulators, need to be engaged for long-term adoption of genomics in clinical care37. The use of an adaptive model allowed the program to be established with the support and input of engaged stakeholders, whilst having the ability to engage and upskill new stakeholders during the program.Genomics is a discipline that requires a level of genetic literacy before people can contribute to discussions38. The adaptive strategy enabled Queensland Genomics to accommodate these changes into the program and post-program planning by providing investment, data management processes and workforce planning around genomic sequencing into Queensland Health’s statewide public pathology service.The adaptive design allowed the Capability and Clinical arms to reflect and respond to the changing landscape of clinical genomics in Australia. During the program’s lifecycle the Australian clinical genome sequencing market fundamentally changed. This demonstrated a national vulnerability in accredited sequencing capacity and affected Queensland Health’s plans for clinical genomics. There was also the deployment of a federal program to standardise aspects of clinical genomics across states in response to the National Health Genomics Policy FrameworkPivoting the program to reflect the governance review findings required substantial change management for both funded stakeholders and program administrators. The program was established with a granting and research style funding and reporting structure that was familiar to the Discovery Round stakeholders that had pre-existing experience with these processes. When the governance review identified limitations in the program’s ability to enact health system implementation, the program structure was changed from a granting process to a commissioning process with focus on Queensland Health as the beneficiary. For existing external stakeholders the change in funding structure and program expectations was difficult to manage. There was also a significant change in responsibilities and work load for the program administration, as they shifted from a largely oversight and administration role to being active in project deployment, system capability development, and coordinating a genomics ecosystem for Queensland.39. Much of this was avoided by implementing the program outside the health system bureaucracy to enable the development of an agile program business structure. However, the program struggled to maintain this agile business structure whilst working within standard government processes of the health system. This influenced project approvals, contracting, intellectual property agreements, ethical and governance approvals, and ultimately, the timeline for implementation of the program. This was exacerbated by the post-governance review changes.A challenge with adaptive management is that it can impose unfamiliar demands on the managing organisation40. Yet, as genomics isn’t mainstream it was difficult to convince executive leaders, with health system-wide responsibilities, that genomics is worthy of their time and attention. The Queensland Genomics program made numerous attempts throughout the program to engage executive leaders in program activities and post-program planning for clinical genomics, with varying degrees of success. In the case of post-program planning the impending program end date created an urgency that motivated action within Queensland Health, in the form of an Executive-level working group to develop a State policy position for genomics in Queensland as a precursor for the development of funding frameworks.Engaging organisation leadership is critical for the successful uptake of genomics in clinical settings8. By structuring Queensland Genomics using an adaptive management philosophy, the program had the opportunity to collect and apply lessons from earlier activities and feed this information into an engagement strategy that could affect real change to the health system from within the program. This ultimately enabled the program to evolve over time and create a program that was relevant to Queensland Health’s system needs. Overall, this approach:Prompted broad health system awareness, knowledge and readiness for clinical genomics - not only at clinical level, but also at an operational and executive levelCoordinated involvement from health information management, infrastructure planning, information technology, pathology services, patient services, multiple medical disciplines, and health policyFocused on developing a program that can deliver post-program sustainability within existing Queensland Health governance and funding frameworksPlanned for post-program continuation of genomics within clinical services and health systems.It is recognised that genomics as a clinical test requires system-wide change to be effectively implemented into a health systemThrough adapting funding structure and program development to enable a customised approach to genomics implementation needs, Queensland Genomics has prompted administration of funding and resources that reflected health system stakeholder needs and system readiness.


In healthy adolescents, delayed pubarche is generally a benign condition that is caused by a physiological discrepancy between gonadarche and adrenarche. In presence of other clinical signs and symptoms, delayed pubarche can be caused by single or multiple hormones deficiency  and/or genetic conditions . Exposition to endocrine disruptors has also been described as a possible cause of delay of pubic hair development. Basic blood tests, karyotype and first level imaging studies are helpful in the differential diagnosis. Pubarche is a term that indicates the first appearance of pubic hair, which normally develops between 8 and 14 years of age in females, and between 9 and 15 years of age in males. The combination of pubarche and axillarche, which is the first appearance of axillary hair, along with the development of sebaceous and apocrine glands, is called adrenarche  Although pubarche is strictly connected with puberty and its appearance is considered as one of the typical manifestations of pubertal progression, this event is independent from the gonadic activity of the hypothalamic-pituitary axis. Since pubarche and puberty follows different pathways, they can manifest at different times and, if one of them is pathological, the other can still function normally. Therefore, it is important to not confuse the onset of puberty with the appearance of pubarche, especially in boys .Pubarche is defined premature if it commences before 8 years of age in girls and 9 years of age in boys (which is − 2 standard deviations [SD]). On the other hand, pubarche is considered delayed if it happens after 14 years of age in girls and after 15 years of age in boys (which is + 2 SD) [Premature pubarche is a common condition and it is one of most frequent requests for an endocrinologist consultation . Its diaA discrepancy in the time onset of pubarche and gonadarche, both in males and females, is not infrequent and it is considered as a variation of the normal growth sequence. In males, pubarche usually appears between state G3 and G4 of the genital development. However, the timing of pubic hair appearance is not strict and, basically, a given stage of pubic hair development can be achieved in presence of any stage of genital development. Marshall and Tanner reported that, in a cohort of 228 healthy boys, around 16% of those who already reached stage G4 had not displayed their first pubic hair yet . Since sIn females, a discrepancy between pubarche and breast development is possible as well. Marshall and Tanner reported that it is not rare to identify girls who reach stage B4 (breast stage 4) before pubic hair appears . SimilarTherefore, before considering a delayed pubarche as pathological, a close attention must be given to all the pubertal changes taking place in a child and to their specific timing. As a matter of fact, most of the cases of delayed pubarche in a healthy adolescent with no history and/or signs of a concurrent disease are expected to fall under the physiological range.Occasionally, in conjunction with other symptoms or medical signs, a delayed pubarche can be expression of an underlying disease Table .Table 1Primary adrenal insufficiency is a life-threatening condition caused by impaired secretion of adrenal glucocorticoid and mineralocorticoid hormones . ImpaireAlbeit still being unclear, the contribution of ACTH in the regulation of adrenarche has been proved multiple times, so that an absent adrenarche can be indicative of an ACTH deficiency , 19. SecPIT1/POU1F1 deficiency, a complex pathological entity that typically causes growth hormone, prolactin and thyrotropin deficiency. Taha et al. described 8 patients affected by this condition, all with spontaneous onset and progression of puberty, presenting very low levels of DHEAS and absent or delayed pubic hair appearance [(IGSF1) deficiency, is characterized by central hypothyroidism, delayed surge in testosterone during puberty, macroorchidism and, occasionally, also by hypoprolactinaemia and transient growth hormone deficiency. Van Hulle et al. reported the case of a 19-year-old patient with IGSF1 that presented delayed adrenarche and pubarche [Two rare variants of anterior panhypopituitarism have been associated with delayed pubarche, even with normal circulating levels of ACTH and cortisol. The first of them is pearance . The secpubarche .In both these two genetic conditions, the authors suspect that a reduction of prolactin secretion could be the cause of pubic hair’s delayed appearance, since prolactin receptors are expressed in the adrenal cortex and act synergistically with ACTH in the surge of adrenal androgens . This stTherefore, in presence of a delayed pubarche, ACTH should always be tested with a baseline blood sample, along with the remaining pituitary hormones. In presence of panhypopituitarism, genetic consultation and testing should be requested to identify the mutation responsible for the condition. As reported above, in few patients affected by genetic panhypopituitarism, normal ACTH values are not sufficient to guarantee the normal appearance of pubarche, due to the complex interactions between the various pituitary hormones.(AR) gene in the region q11–12 on the X chromosome that leads to an androgen resistance in androgen-dependent tissues. Therefore, subjects with AIS, while having a 46,XY karyotype, typically present feminization of the external genitalia at birth, abnormal sexual development and infertility [AR gene.Androgen insensitivity syndrome (AIS) is a rare condition, with a prevalence that varies from 1/20,000 to 1/100,000 people. It is characterized by a mutation of the androgen receptor ertility . The cliertility . AIS subertility . DHEAS iertility  Primary ertility . Apart fTurner syndrome (TS) is a common chromosomal disorder  caused by a total or partial loss of the second X chromosome  , 31. TS The relationship between thyroid dysregulation and pubertal development is complex and the studies that explored this matter showed controversial results, especially regarding precocious and delayed puberty . On the Wilken et al. reported that, in a cohort of 323 of young females, higher prepubertal concentrations of free thyroxine (FT4) were associated with an earlier onset of pubarche, without the appearance of thelarche. This result was independent from common confusing factors, especially obesity . As for Phthalates diesters are chemicals with anti-androgenic properties that are used as softeners in plastic production and can be found in various products, such as food packaging, medical devices and building materials. Some studies have showed a relationship between phthalate exposure and changes in girls pubertal timing, but this finding has not been consistently confirmed in the medical literature and the topic remains controversial –44. In aIn healthy adolescents, delayed pubarche is generally a benign condition that is caused by a physiological discrepancy between gonadarche and adrenarche. In rare occasions, a late appearance of pubic hair can be caused by pathological conditions, most of which are characterized by single or multiple hormones deficiency  and/or genetic alterations . In these cases, delayed pubarche is always associated with other clinical signs and symptoms. Basic blood tests are helpful in the differential diagnosis, as well as imaging studies and karyotype. Physicians need to be aware of these conditions to avoid late diagnosis that may worsen patients’ quality of life or even put their life at risk.


Safe spaces are increasingly utilized to reduce alcohol-related harm, violence, crime and improve public safety in nightlife settings. This study aimed to determine the impact and return on investment of the Take Kare Safe Space (TKSS) program—a harm reduction program implemented to address alcohol-related violence and disorder in three locations in Sydney’s night-time economy between 2014 and 2019. TKSS ambassadors provided support at static safe spaces and patrolled designated nightlife precincts to provide practical assistance to vulnerable and intoxicated people. Ambassadors recorded information relating to these interactions including ‘client’ age, gender, perceived level of intoxication, time and length of engagement with the program. Costs of program implementation and benefits of major incidents averted were obtained to allow calculation of return on investment. From December 2014 to April 2019, 66,455 people were supported by TKSS ambassadors. Most users were male (62%) and aged 18–25 years (66%). Of 3633 interventions by ambassadors, serious risk of harm was averted in 735 cases (20%). The program’s return on investment is estimated at 2.67, suggesting that a $1 investment results in $2.67 in benefits. Safe Spaces are extensively utilized, particularly by young males with high levels of intoxication, and represent a positive return on investment. Despite the growth of such services, there remains a notable absence of rigorous, independent evaluation regarding the outcomes and/or social benefit of safe space programs. From a policy perspective, there is a need for more high-quality economic evaluations to better inform decisions about competing uses of limited resources. Alcohol use has a significant negative social and public health impact . In AustIn Australia, night-time alcohol-related violence and disorder have been a significant community concern over the last decade, with numerous high-profile deaths linked with heavy alcohol and other drug use . In respFor the purposes of this paper, ‘safe space’ refers to a harm reduction service often run in partnership with health, community, emergency or welfare services to increase public safety and amenity in town centers or other public spaces . Night-tDespite the increasing use of safe spaces as part of local government and community responses to addressing alcohol-related violence, there remains a notable absence of rigorous, independent evaluation of the effectiveness of programs in achieving benefits . To dateAccordingly, this study aimed to determine the impact and return of investment of the TKSS program implemented in Sydney, Australia, as part of a city-wide plan to address alcohol-related violence and disorder.Design. The methodology underpinning the evaluation was guided by the NSW Government’s Program Evaluation Guidelines to examine process, outcome and economic indicators . The evaTKSS program. The TKSS program commenced operations in Town Hall in the central business district of Sydney, Australia, in December 2014. A second site in Kings Cross commenced operations in July 2015 and a third site at Darling Harbor commenced operation in February 2017. All three sites are in or near the central business district entertainment precinct of Sydney and are renowned for their nightlife. TKSS operated year-round from 10 pm to 4 am on Friday and Saturday nights. Each site of the TKSS program is staffed by groups of 3–4 team members, called ambassadors, including a paid team leader and volunteers. Volunteers are drawn from the public and student recruitment pathways from established relationships between the program and local universities. Volunteer training is provided by the program, and includes instruction on basic first aid, the provision of care for, and communication with, intoxicated patrons, de-escalation and other program safety protocols. Alongside public safety interventions in situations of “risk” (including high-level intoxication and conflict), the TKSS program provides first aid support, escort to accommodation, assistance with accessing transport, a phone charging station, help connecting with friends or family, and general assistance with directions and local information. When the support required exceeds the program’s scope and capacity, ambassadors refer incidents to appropriate emergency or other social services. Integral to the TKSS program is its connection to, and interaction with, other agencies and nightlife service providers to ensure the most appropriate care is provided for those in need of assistance. This includes collaboration with City Rangers, licensed premises, venue security, police, closed circuit-television operators, and transport staff.As part of program delivery, ambassadors record a range of information relating to client interactions with TKSS, either on a paper-based form or via a mobile phone application. Data recorded included age and gender of the service user, the time support was provided, the length of time each user was in contact with the service, and the perceived level of intoxication of the person/people receiving support (based on the NSW Responsible Service of Alcohol intoxication guidelines ). The tyReturn on investment. Costs were determined by combining information on TKSS operating costs with the value of volunteer time implementing the program. The value of one hour of volunteer time was calculated using Australian Bureau of Statistics data on average weekly earnings and adjusting for a 35 h working week . BenefitCosts of major incidents averted were derived using estimates of incidents averted together with a cost estimate for each incident. The cost of assault and sexual assault (reported crime) was valued using estimates derived by Byrne et al. (2012) . The cosThe outcome of the return on investment is reported as a ratio (benefits/costs). Two additional sensitivity analyses were conducted to explore the robustness of results to changes in assumptions. Sensitivity analysis 1 examines changes in the attribution of serious risk of injury diverted from 100% (baseline) to 75%. Sensitivity analysis 2 examines the benefit cost ratio for the year 2016–2017 when the TKSS program was fully operational in three sites—Town Hall, Kings Cross and Darling Harbour.n = 41,395). Sixty-two percent  of users were males and 38%  were female. The largest proportion of service users were aged 18–25 years, accounting for 66%  of the sample, followed by those aged 26–39 (n = 7808 or 19%). Of the 8872 assessments completed to determine level of intoxication, 46% (n = 4061) were perceived to have had a high level of intoxication, 36% (n = 3190) a mild level of intoxication, 11% (n = 955) were perceived as sober and 8% (n = 666) were perceived to be under the influence of drugs.User profile. Cumulatively, from December 2014 to April 2019 (inclusive), 66,455 people were supported by the TKSS program. n = 5356) occurred between the hours of 11 pm and 2 am. Seventy percent (n = 5392) of users were in contact with the service for between 1 and 20 min, 17% (n = 1360) for between 21 and 60 min and 13% (n = 981) for greater than one hour. Most users  spent time at the Safe Space, 19%  were supported in other ways (defined as incidents) and 18.7%  requested directions, primarily related to transport.Service Use. Sixty-nine percent of all contacts . Return on investment. The purpose of the current study was to determine the impact and return on investment of the TKSS, a harm reduction program embedded within the community to address alcohol-related violence and disorder in the night-time economy. The evaluation relied on an existing framework endorsed by the NSW Government to examine process, outcome and economic indicators . Using aThe results indicate that the Sydney Safe Spaces were well represented by young men with high levels of intoxication. This is also consistent with global estimates of alcohol use that show higher rates of alcohol use among young men compared to women . Young mThe Take Kare Safe Space aimed to improve public safety and reduce alcohol-related harm and violence by providing a place where vulnerable people could access support in the weekend nighttime economy. One of the objectives of the evaluation was to examine the return on investment. Edmunds et al. (2018) conducted a systematic review of economic evaluations of interventions for high risk young people . AlthougFurther, there is a lack of evidence pertaining to community preferences for such interventions. Attaching a value to community benefit is seen as a valuable input in economic evaluations that adopt a social perspective . A recenn = 735) of all interventions made by ambassadors that averted serious risk of harm were included in the formal return on investment analysis out of 3633 actual interventions. Other likely benefits of the program that are difficult to quantify such as improvements in public safety and amenity, more efficient resource allocation for service providers, improved partnership, communication and resourcing, and flow-on effects for tourism and investment, suggest the true return on investment is likely to be much higher. Indeed, the three-month pilot evaluation of the TKSS program estimated a positive return of $9 for every $1 invested, largely due to savings attributed to the cost of police, ambulance and additional medical services [The current research estimated a return on investment at 2.67, suggesting that every $1 invested in TKSS resulted in $2.67 in benefits. This return on investment is conservative, as the analysis did not quantify the full spectrum of potential benefits associated with the TKSS program or the value of community benefit. For example, only 20%  since 2009. Although the trends pre- and post-implementation of the LLA and the TKSS program were not significant, the downward trend suggests that Safe Spaces may have had a dampening effect on emergency department presentations and ambulance dispatches  but not the occurrence of assaults. This would be consistent with previous research on community-based responses to alcohol harms, which showed that alcohol harms can be less in communities with community action responses (compared to communities who do not have those responses), but sustaining reductions in alcohol harms over time is likely to need more targeted and restrictive legislation to control the availability of alcohol, especially at high-risk times . Strengths and limitations. This evaluation should be interpreted with reference to the following strengths and limitations. A limitation of the evaluation is the use of internal data collected by Safe Space ambassadors to estimate impacts on alcohol harms. These data were self-reported and collected using an unvalidated tool that may have been prone to recall or reporting bias. Second, during the period of this study (December 2014–April 2019), funding to operate TKSS was sporadic. This sporadic funding impacted on operational capacity to the extent that the Kings Cross Safe Space was closed on Friday nights, operating only on Saturday night. Third, the responsibilities of operating the Safe Spaces transitioned from an external organisation, St John Ambulance, to another organisation—Stay Kind. This transition resulted in additional financial burdens associated with operating the TKSS program. Previously Stay Kind operated from offices provided pro bono with one paid staff member. Bringing the program inhouse required establishing a range of additional systems and processes for running the program including directly employing the team leaders, inhouse management of staff, training and induction of volunteers as well as leasing an office and acquiring vehicles within the constraints of reduced and sporadic funding. Fourth, for the duration of this study, the City of Sydney Central Business District underwent significant capital works that impacted on the mobility of ambassadors and capacity to operate efficiently. This may have also impacted on the number of users to the service. Despite these challenges, the TKSS program has met it objective to provide a harm reduction service where vulnerable young people can access support and a safe place or a safe passage home. A real strength of the program is the ambassadors collection of real-time data. The impacts on alcohol harms are usually not accurately recorded, or captured at all, because they do not come to the attention of relevant authorities , or are Safe Spaces are extensively utilized, particularly by young males with high levels of intoxication, and represent a positive return on investment. Harm reduction programs such as TKSS play a key role in de-escalating conflict and averting the risk of serious harm. Safe Spaces offer a positive return on investment and should play a key role in any city-wide management plan to address alcohol-related violence and disorder in the night-time economy. However, despite the growth of such services, there remains a notable absence of rigorous, independent evaluation regarding the outcomes and/or social benefit of safe space programs. From a policy perspective, there is a need for more high-quality economic evaluations to better inform decisions about competing uses of limited resources.


Plasmodium falciparum contains an N-terminal region, a conserved Region I (RI), a junctional region, 25–42 copies of major (NPNA) and minor repeats followed by a C-terminal domain. The recently approved malaria vaccine, RTS,S/AS01 contains NPNAx19 and the C-terminal region of CSP. The efficacy of RTS,S against natural infection is low and short-lived, and mapping epitopes of inhibitory monoclonal antibodies may allow for rational improvement of CSP vaccines. Tobacco Mosaic Virus (TMV) was used here to display the junctional epitope (mAb CIS43), Region I (mAb 5D5), NPNAx5, and NPNAx20 epitope of CSP (mAbs 317 and 580). Protection studies in mice revealed that Region I did not elicit protective antibodies, and polyclonal antibodies against the junctional epitope showed equivalent protection to NPNAx5. Combining the junctional and NPNAx5 epitopes reduced immunogenicity and efficacy, and increasing the repeat valency to NPNAx20 did not improve upon NPNAx5. TMV was confirmed as a versatile vaccine platform for displaying small epitopes defined by neutralizing mAbs. We show that polyclonal antibodies against engineered VLPs can recapitulate the binding specificity of the mAbs and immune-focusing by reducing the structural complexity of an epitope may be superior to immune-broadening as a vaccine design approach. Most importantly the junctional and restricted valency NPNA epitopes can be the basis for developing highly effective second-generation malaria vaccine candidates.The Circumsporozoite Protein (CSP) of  Of the five species of Plasmodium that can infect humans, P. falciparum (Pf) is the primary cause of malaria-associated mortality, with most deaths occurring in children under 5 years of age. The development of an effective vaccine would positively augment the fulfillment of 2040–2050 malaria eradication goals2.Malaria is a disease of significant public health importance and according to the WHO 2020 World Malaria Report, 409,000 deaths globally were attributed to malaria in 2019. Although there has been a decline in the number of cases and deaths since 2010, emerging drug and insecticide resistance threaten these successes. The spread of COVID-19 pandemic across Africa could further reverse decades of progress on malaria control as was noted during the 2014 Ebola virus outbreak in West AfricaAnopheles mosquito takes a blood meal Plasmodium sporozoites are injected in the skin and travel to the liver to establish infection. Sporozoites are covered with the Circumsporozoite protein (CSP) which has a N-terminal ‘domain’ encompassing Region I (RI), that includes an inter-species conserved 5 amino-acid motif (KLKQP), followed by a junctional sequence NP-DPNA-NPNV-DPNA, 25-42 copies of the major NPNA repeats, interspersed minor NPNV-DPNA repeat5, and a C-terminal region consisting of an alpha thrombospondin type-I repeat domain . The RTS,S/AS01 vaccine contains ~19 copies of the major NPNA repeats and a majority of the C-terminal region of CSP, linked to a Hepatitis B S antigen particle and is formulated in the adjuvant AS017. RTS,S/AS01 received a positive scientific opinion by the European Medicines Agency8 and has recently been recommended by the WHO for use in children living in areas of moderate to high transmission of P. falciparum malaria. RTS,S efficacy in the field, measured as the incidence of first or only episode of clinical malaria, is estimated to be ~30%9. An improved form of RTS,S/AS01, which consists of R21 and Matrix-M adjuvant10, has also shown promise with >70% efficacy in an area of seasonal malaria transmission11. Additional efforts to improve the efficacy of CSP-based vaccines are currently underway12.When an infected n repeats, although flexible, frequently form short structured motifs: type-I beta and pseudo 310 turns13. Stabilized by homotypic interactions several repeat-specific mAbs can bind to a single CSP molecule15. Pf sporozoite invasion inhibition (in vitro), as well as protection against transgenic parasites (in vivo), are primarily associated with mAbs binding to the central NPNA repeat region of CSP23. Similar to the emerging evidence from respiratory syncytial virus vaccine development, efforts to improve CSP vaccines involve the use of well defined protective epitopes identified using inhibitory mAbs27.Structural studies of RTS,S-elicited monoclonal antibodies (mAbs 311 and 317) have shown that the major (NPNA)28. The RI and a putative proteolytic cleavage site within the N-terminal region play an important role in the process of hepatocyte invasion31. Mechanistically an N-terminal cleavage event is purported to convert CSP into an adhesive or invasive conformation34. Vaccine-elicited antibodies against cryptic N-terminal epitopes may block invasion35 and the presence of naturally-elicited N-terminal antibodies may be associated with protection36. Despite evidence of function, the N-terminal region was not included in the RTS,S vaccine construct and nearly full-length versions of CSP vaccines are now entering human clinical trials with the assumption that immune-broadening the response to the N-terminal region of CSP would improve vaccine efficacy39.The structure-function relationship of CSP domains are not completely understood. The C-terminal region is anchored to the sporozoite via a GPI anchor and the rest of the molecule appears to have a flexible rod-like structure10 turn conformations22. These junctional mAbs cross-react to varying degrees with the NPNA major repeats and initial indications show that vaccines that include the junctional region may be more effective than major repeat-based vaccines23. Multiple groups have tried to use this information in vaccine design: Tan et al. chemically conjugated the epitope for the junctional sequence cross-reactive mAb MGG4 n repeats has been hypothesized as means to improve CSP vaccines17.Whole sporozoite vaccines PfSPZ- or PfSPZ-CVac-elicited mAbs , along with some RTS,S-elicited mAbs  can bind to the junctional DPNA and NPNV motifs that adopt type I and pseudo 344. The tobacco mosaic virus (TMV) is being tested as an epitope display platform for developing vaccines against papillomavirus, influenza, tularemia, and plague49. TMV-based vaccines can be optimized using E. coli expression system and equivalent virions could also be produced in planta, to reduce manufacturing costs. Structurally, a TMV VLP is composed of 17 coat protein monomers that form a 20 nm disk. The disks can stack into rods up to ~300 nm in length allowing up ~2100 copies of an epitope to be displayed per TMV rod51.A problem with using short discrete epitopes as vaccines is that they are inherently weak immunogens. Virus-like particles  are ideally sized for uptake by antigen-presenting cells and repetitive display of an epitope on VLPs further improves immunogenicityn repeat epitope27. Only five copies of NPNA (TMV-NPNAx5) induced higher antibody titers, avidity, and in vivo protection compared to a nearly full-length soluble CSP that contained 19 copies of NPNA per molecule. The observed superiority of TMV-NPNAx5 over full-length CSP was reproducible in the non-human primates where the TMV-NPNAx5 adjuvanted with ALFQ (the Army Liposome Formulation containing the QS-21) showed 3-fold improvement of repeat titers over the existing RTS,S/AS01 benchmark in humans. Here we used the TMV platform to test if vaccines based on the junctional sequence and the RI can compare to the TMV-NPNAx5 vaccine effectiveness. A TMV construct containing 20 copies of the NPNA repeat (TMV-NPNAx20) was also compared to TMV-NPNAx5. Our data support the continued development of CSP vaccines based on TMV VLPs and we show that epitope focusing by restricting (NPNA)n valency or by displaying the junctional epitope can elicit equally potent protective antibodies.Previously we used TMV to optimize the valency and flexibility of the (NPNA)27. The circular permutant sequence allows the N- and C-termini of the TMV coat protein to be engineered into the pore of an assembled disk, such that a peptide epitope can be displayed on the exposed loop52  at week 0, 3, and 6  and 23-fold higher than FL-CSP (p ≤ 0.001). The titer against the junctional peptide was highest for the TMV-Junc group, which was 2.3-fold higher than TMV-NPNAx5 (non-significant), and 90-fold higher than for the FL-CSP group (p ≤ 0.01)  on individual mouse sera at 2WP3 showed similar geometric mean titers in the three vaccine groups Fig. . The NAN01) Fig. . The rat01) Fig. , while ip < 0.05). Repeat-avidity of the TMV-NPNAx5 group was higher than the other two groups , and TMV-Junc repeat-avidity was higher than FL-CSP group (p ≤ 0.01). Strikingly, junctional peptide avidity was low for the TMV-Junc and TMV-NPNAx5 induced antibodies (<1.5 M) and it was not detectable for the FL-CSP induced antibodies, suggesting that in general antibodies bound weakly to the junctional peptide in an ELISA, as compared to binding to the major repeat peptide.An avidity ELISA conducted using a chaotropic agent sodium thiocyanate (NaSCN), was used to show the relative strength of antibody binding Fig. . FL-CSP p = 0.0055) and TMV-Junc  elicited similarly high levels of sterile protection. In vitro liver stage sporozoite development assay (ILSDA) conducted using serum pools at 2WP3 for this group of challenged mice showed that TMV-Junc and TMV-NPNAx5 pools inhibited >90%, while FL-CSP pool consistently inhibited <70% at 1:30, 1:100 and 1:300 dilutions  were challenged at 2WP3 with 3000 trangenic parasites  and 40% protection in the TMV-Junc and FL-CSP groups (n.s.)  Fig. . ILSDA us.) Fig. .Fig. 4StStudy 1 demonstrated that the CSP junctional epitope (mAb CIS43 target) was weakly immunogenic on FL-CSP, and its display on TMV-Junc induced polyclonal antibodies that recapitulated the cross-reactive binding properties of mAb CIS43. TMV-NPNAx5 induced higher repeat-specific antibody titer than TMV-Junc and the TMV-NPNAx5 antigen induced similar levels of junctional epitope binding antibodies as TMV-Junc. Avidity data showed that the major NPNA repeat was the primary target of high-affinity binding of antibodies elicited by TMV-NPNAx5 and also TMV-Junc. The TMV-NPNAx5 and TMV-Junc antibodies had comparable in vitro and in vivo efficacy and were both superior to FL-CSP. The longevity of antibody responses did not differ between these vaccines.Protective responses induced by the junctional and the NPNAx5 epitopes prompted the design of a single TMV antigen displaying both epitopes. The sequence NP-DPNA-NPNV-DPNA-NPNV-DPNA-NPNAx5 was inserted at the exposed loop of TMV and the antigen was termed TMV-Junc-NPNAx5 Fig. . TMV-Jun27, was also included in this study , reported previously by Langowski et al.udy Fig. . TMV-NPNudy Fig. . DLS meal23 Fig. . MAb 317ely Fig. . In contn = 10) received three 1 µg doses of TMV-Junc-NPNAx5, TMV-RI, TMV-NPNAx5, or TMV-NPNAx20 vaccines formulated in 50 µL adjuvant ALFQ at 0–4 and 7-wk time-points  to 2.6-fold against NPNAx6 peptide (non-significant). Therefore restricted repeat valency in the vaccine preferentially improved titers to structurally constrained conformations of NPNA.Four groups of mice  and TMV-NPNAx20 (p < 0.01) were both higher than TMV-Junc-NPNAx5  and was 5-fold higher than TMV-Junc-NPNAx5 (not significant) , the anti-FL titer of the TMV-NPNAx5 group was 2.3-fold higher than TMV-NPNAx20 (not significant) and 3.6-fold higher than TMV-Junc-NPNAx5 (p < 0.05). TMV-RI elicited no detectable FL-CSP specific responses (not plotted) Fig. . FL avidAx5 Fig. . The repnt) Fig. . TMV-RI nt) Fig. .Fig. 7Stp = 0.0007, relative to controls) and TMV-NPNAx20  groups. Consistent with the lower titers, the protection achieved with TMV-Junc-NPNAx5 was lower  and TMV-RI elicited no protection. ILSDA using serum pools at the 5WP3 showed that inhibition for TMV-NPNAx5 and TMV-NPNAx20 remained >90%; TMV-Junc-NPNAx5 inhibition dropped to 78% at the lowest serum dilution and TMV-RI induced no ILSDA activity  mice in Study 2 were challenged and all controls were infected by day 4 Fig. , but higity Fig. .Fig. 8StStudy-2 confirmed that epitopes for mAbs CIS43, 317, 580, and 5D5 epitopes could be recapitulated by TMV display and all except the mAb 5D5 epitope were displayed in immunologically favorable conformations. Restricting the repeat valency to NPNAx5 reduced the binding of a low-affinity mAb to this antigen and the resulting polyclonal antibodies to NPNAx5 reacted preferentially to shorter repeat peptides as compared to NPNAx20 antibodies. The TMV-NPNAx5 vaccine performance was as good if not better than TMV-NPNAx20. Most remarkably, combining the junctional epitope with NPNAx5 in one construct negatively impacted vaccine immunogenicity and efficacy. The RI region targeted by mAb 5D5 epitope was non-immunogenic even when displayed on the TMV particle and it induced no functional antibodies.54. RTS,S/AS01 contains the NPNA repeats and the C-terminal region of CSP55. While the NPNA repeats are highly conserved, the C-terminal region of CSP is polymorphic56 and it induces strain-specific antibodies that may not be particularly inhibitory58 and lead to parasite escape61. A way to improve CSP vaccines is to eliminate non-neutralizing epitopes from the vaccine. We have shown that immune-focusing the antibodies to only the NPNAx5 epitope, elicited better protection than a nearly full-length recombinant CSP in mice and monkeys27. Here, epitopes of monoclonal antibodies that bind outside the (NPNA)n repeats were tested for inhibitory effects and to determine if immune-broadening can improve upon TMV-NPNAx5 vaccine efficacy.The use of ‘complete’ proteins or organisms as vaccines can potentially lead to co-induction of non-neutralizing antibodies that in extreme cases can exacerbate disease62. A recent study using recombinant mAb 5D5 has also confirmed that this mAb may lack parasiticidal activity63.A region I construct (TMV-RI) did not induce CSP-reactive antibodies and our data did not support the inclusion of RI in a future vaccine. The high polarity and predicted flexibility of the RI sequence likely renders it non-immunogenic, even when stabilized as a loop on TMV. Despite being shown to be the target of an inhibitory mAb 5D5, RI-containing vaccines have not been highly protective without the repeat epitope22, as evidenced by sterilizing protection observed in humans following mAb CIS43 passive transfer64. However, RTS,S  antibodies also bind to the junctional region65, and it has been suggested that junctional binding was a result of affinity maturation of (NPNA)n-specific antibodies66. To our knowledge, there has been no direct comparison of vaccines displaying the junctional and the major repeat region of CSP. Our TMV-Junc vaccine performed, as well as the TMV-NPNAx5 vaccine in a mouse model . Therefore, immune-broadening using multiple short inhibitory epitopes on a single vaccine construct was complex and, in this instance, was found detrimental to vaccine efficacy.Having established the protective efficacy of the junctional and (NPNA)n conformation, which has been shown as being potentially less protective as compared to high-affinity antibodies like mAb 317, which recognizes a “curved” conformation72. Thus TMV-NPNAx5 immunogen may represent a significant reduction in potentially sub-optimal conformations of (NPNA)n while still retaining the high-affinity conformations. We speculate that long repeat lengths present on parasite CSP function as a smoke-screen to allow the induction of non-effective low-affinity antibodies that may compete with high-affinity functional antibodies. Seder and Cockburn, have suggested that long CSP repeats could diminish the quality of CSP-specific B cells by allowing the formation of multivalent complexes with low-affinity B cells which then prematurely exit from germinal centers73. Our data shows that immune-focusing by restricting repeat valency may be a way to improve upon the current long-repeat containing CSP vaccine candidates.While no evidence favoring epitope broadening with TMV-Junc-NPNAx5 was found, immune focusing by restricting repeat valency to NPNAx5 showed >80% sterile protection in two challenge experiments with 3 × 1 μg antigen dose and increasing the repeat valency to NPNAx20 did not further improve efficacy 78. The predicted molecular model figures were generated using PyMOL software .The TMV-NPNAx5, TMV-Junc, and TMV-Junc-NPNAx5 structural models were submitted to the Rosetta Protein Structure Prediction Server  cells. Cells were grown in shake flasks, induced with 0.1 mM IPTG at 37 °C for 2 h, and harvested by centrifugation. The insoluble cell pellet obtained after microfluidization and centrifugation was dissolved in 7 M urea containing buffer and purified using Qiagen Ni-NTA Sepharose (Gaithesburg MD) for the removal of non-specific proteins followed by Q-sepharose column for the removal of endotoxin (GE Lifesciences) under denatured conditions. The Q-sepharose flow-through was dialyzed overnight against a 20 mM Tris, 20 mM sodium chloride buffer (pH 9.0) containing 0.05% β-mercaptoethanol. The dialysis buffer was exchanged with fresh 20 mM sodium phosphate dibasic, 20 mM sodium chloride buffer (pH 7.4). After dialysis, the proteins were filtered using a 0.22 µm filter and concentrated on an Amicon ultra centrifugation filter . Purity was of the final products was >95% as confirmed by analyzing the coomassie blue-stained SDS-PAGE gels  for 5 min and excess liquid was blotted away with bibulous paper. Grids were then stained with 3 µL of 1–2% uranyl acetate for 10 s before wicking of excess liquid with bibulous paper. The grids were imaged between 30,000X and 40,000X.27. Serial dilutions of mAbs were tested and the concentration that resulted in OD415nm = 1 was determined using the Full-length CSP, TMV-Junc, TMV-NPNAx5, and TMV-Junc-NPNAx5 coated ELISA plates (100 ng protein-coated/well). ELISA plates were washed three times with 0.05% PBS-Tween solution, blocked for 1.5 h with blocking buffer (0.5% casein + 1% Tween) and mAbs were incubated for 1 h, starting at 1000-100 ng/mL concentration and serially diluted three-fold. A 1:4000 dilution of Goat anti-human HRP conjugated antibody  was applied to the wells and the plate was developed using ABTS Peroxidase Substrate (KPL Seracare Milford MA)53.Human monoclonal antibodies 317, 580, and CIS43 were recombinantly produced79.Mouse mAb 5D5 was kindly provided by Leidos Health . A western blot was conducted using 1 μg protein separated by reducing PAGE and transferred to PVDF membrane. Immune-blots were stained with 1 μg/mL mouse mAb 5D5, followed by 1:5000 anti-mouse HRP conjugated antibody (Southern Biotech) and the blot was developed for 2 min using the Blue POD Substrate (Roche)17 were determined following overnight coating at 100 ng/well; the NANPx6 peptide was coated at 20 ng/well, and the FL-CSP was coated at 100 ng/well. The remaining ELISA protocol and wash steps were similar to the antigenicity ELISA and secondary antibody was 1:4000 Goat anti-mouse H + L HRP . Titer was defined as the serum dilution that resulted in OD415 = 153.Mouse serum antibody titers against the junctional peptide (KQPADGNPDPNANPNVDPN)ELISA plates were coated with Avidin (100 ng/well) followed by 100 ng/well N-terminally biotin tagged peptides NPNAx1 to NPNAx6 . The remaining mouse serum ELISA was as described above.79. ELISA plates coated with FL-CSP protein, NANPx6 or the junctional peptide were blocked with 200 µL of 0.5% Casein-PBS for 2 h at room temperature. Following a PBS-Tween wash, 100 µL of 1:1000 diluted serum were plated for 1 h at room temperature. Unbound antibodies were washed with 100 µL of aqueous NaSCN solution  for 15 min at room temperature. After washing, the mouse serum ELISA was developed as above. To determine avidity index, the PBS washed well OD415 was used as the reference maximal, and linear regression was used to determine the concentration of NaSCN that reduced the OD415nm to half maximal OD415.A sodium thiocyanate ELISA was used to determine serum avidityPf parasites incubated with pooled sera from naïve or immunized mice at 1:30, 1:100, and 1:300 dilution or a positive control anti-CSP monoclonal NFS180. This mixture was then added to wells containing human hepatocytes and after 3 days of incubation, the levels of Pf 18S rRNA levels were determined by quantitative real-time PCR. Percent inhibition was calculated against the naïve serum negative control and the assay was repeated more than once. All assay results were replicated in at least two independent experiments.ILSDA was performed using NF54 All the TMV vaccine candidates tested were solubilized in 20 mM sodium phosphate dibasic, 20 mM sodium chloride (pH 7.4). Antigens were produced and vaccinations were conducted at different times. The antigens were also co-analyzed using SDS-PAGE, Electron Microscopy, and particle Malvern Zetasizer Nano S particle sizer . Animals were housed in a AAALACi accredited facility in compliance with the Animal Welfare Act and other federal statutes and regulations relating to animals and experiments involving animals and adheres to principles stated in the Guide for the Care and Use of Laboratory Animals, NRC Publication, 2011 edition.TM and QS21)82. For Study 1, female C57BL/6 mice were vaccinated with 1 µg of either TMV-NPNAx5, TMV-Junc, or FL-CSP in ALFQ or sterile 1×PBS (control) at 0, 3, and 6 weeks. For Study 2, mice were vaccinated with 1 µg of either TMV-NPNAx5, TMV-NPNAx20, TMV-Junc-NPNAx5, or TMV-RI at 0, 3, and 7 weeks. All vaccines were 50 μl formulations administered intramuscularly (IM) in alternating thighs.All antigens were formulated in Army Liposome Formulation (ALFQ) that contains immune-modulators  containing 5% fresh C57BL/6 mouse serum. Mice were challenged with 3,000 sporozoites per mouse in 100 µL volume injected intravenously (IV) into one of the lateral caudal veins of the tail with a 27G needle insulin syringe57. Mice were evaluated for parasitemia beginning at day 3 post-challenge through day 14 via Giemsa-stained thin blood smears. Thin blood smear slides were prepared by snipping the distal end of the tail and spotting a drop of blood onto a microscope slide. Smears were fixed with 100% methanol and stained with a 1:15 Giemsa in staining buffer solution for 30 min at room temperature. The slides were then rinsed with water and submerged in staining buffer for 1 min. After air drying, the slides were read using 100× oil immersion microcopy. Mice were euthanized with carbon dioxide, followed by cervical dislocation after showing positive blood smears for two consecutive days. Mice that remained non-parasitemic for 14 days post challenge were considered as protected and any mouse that was slide positive on two consecutive days was sacrificed and considered non-protected.Mice were challenged following the third dose with p-values were corrected for multiple comparisons using the Tukey’s method. Statistically significant differences in group means was indicated in figures as: **** (p < 0.0001), *** (p < 0.001), ** (p < 0.01), or * (p < 0.05). Protection outcomes were compared using Fisher’s Exact Test between groups. All statistical analyses and graphing were performed using GraphPad Prism Software .Statistical significance for ELISA data was calculated by ANOVA and Further information on research design is available in the Supplementary MaterialREPORTING SUMMARY