Companion animals are valued members of many families, providing companionship, affection, and someone to nurture and love. Some situations such as being suddenly hospitalized for mental or physical health or becoming homeless make it difficult for people to keep their animal companions, especially for those who have a limited support network. The RSPCA NSW Emergency Boarding and Homelessness program supports people experiencing a crisis to access boarding and veterinary treatment for their animals, helping to keep the family together in the long term. This study aimed to understand the social value created by the program and to express this value in monetary terms. We interviewed 13 stakeholders including program clients, RSPCA Inspectors, and human service providers to understand what changed for them because of the program and how they valued that change. The main changes were experienced by program clients and their animals. The most valuable changes were being able to keep their companion animal and improved mental health and wellbeing. We estimate that this program results in social value worth AUD 8.21 for each AUD 1 invested into running the program. This study shows the importance of considering companion animals as part of the family unit and supporting people experiencing a crisis to keep their companion animal.Companion animals play a central role in many families and are especially valued by those who are socially isolated. Crisis situations such as acute hospitalizations, homelessness, and natural disasters can make it difficult to preserve the human–animal bond and can result in animals being surrendered or euthanized. Social support programs like the RSPCA NSW Emergency Boarding and Homelessness program support people experiencing crisis situations with emergency pet boarding, access to veterinary treatment and individualized case management. This study aimed to estimate the social return on investment (SROI) for this program using the standard SROI methodology. In-depth interviews were conducted with 13 program stakeholders and questionnaire responses were received from 29 program clients. Outcomes were quantified for four stakeholder groups: program clients, client’s animals, RSPCA Inspectors, animal pounds, and shelters. Clients and their animals experienced the bulk of the benefit from the program, estimated to have a combined value of over AUD 5 million for the 2020–2021 financial year. The estimated social return on investment was AUD 8.21 for each AUD 1 invested. The study demonstrates that keeping people together with their companion animals or ensuring they are reunited as soon as possible can reduce stressors and improve outcomes for people and animals. Companion animals play a central role in many families, providing their humans with companionship, affection, and someone to nurture and love. In Australia, more than two in three households have companion animals, mostly dogs and cats . HoweverThere are more than 116,000 people experiencing homelessness on any given night in Australia, a rate that is increasing . HomelesThose experiencing a crisis such as homelessness or mental illness are more likely to be socially isolated and lack a support network . As suchThe RSPCA NSW Emergency Boarding and Homelessness program supports people experiencing homelessness and various other crises including acute hospitalizations for mental or physical health conditions or natural disasters with their companion animals. The animals of those experiencing crises can be left in dangerous situations or without care while their owners access refuge or treatment. While separation from their companion animals might be unavoidable for a period, being able to reunite the family unit when possible is invaluable . CompaniSocial Return on Investment (SROI) is an evaluation approach based on methodologies used in economics, accounting, and social research . It asseEvaluating social support programs is crucial to ensure their effectiveness and impact. It allows policymakers and program administrators to assess whether the intended outcomes are being achieved and whether resources are being used efficiently. Evaluation helps identify strengths, weaknesses, and areas for improvement, leading to evidence-based decision-making and program refinement. As such, this study aimed to determine the social return on investment of the RSPCA NSW Emergency Boarding and Homelessness program, using the SROI methodology established by Social Value International .The RSPCA NSW Homelessness & Emergency Boarding program supports pet owners experiencing a range of difficulties, enabling them to keep their animal companions while they focus on their own safety, recovery, or treatment. The program helps to keep animals together with their family in the long term by providing case management, pet boarding, assistance with transportation, assistance to access veterinary treatment, and other support as needed.Individuals are eligible for the program if they are in hospital or temporary accommodation that does not allow pets and they have no support network (friends or family) who can help. Clients either self-refer or are referred by the RSPCA Inspectorate, police, hospitals, housing, or other social support services.RSPCA NSW program case workers collaborate with other housing and social support services to ensure clients can access supports they need to find suitable accommodation so they can be reunited with their animals as soon as possible.Ethical approval for this study was granted by the University of Sydney Human Research Ethics Committee (protocol number: 2020/856).The stakeholders for the RSPCA NSW Emergency Boarding and Homelessness program were identified through interviews with RSPCA NSW Community Programs staff who are involved in service design, planning, and delivery. Only stakeholder groups deemed material were included in the SROI calculation. The materiality of stakeholders was determined through discussion with RSPCA NSW Community Programs staff and validated during stakeholder interviews.A theory of change describes how certain resources (inputs) are used to deliver activities (outputs) that result in outcomes and is central to SROI analysis. A workshop was held with key staff from RSPCA Programs to review program objectives, identify and value inputs, define the services offered (activities), identify stakeholders to engage, and draft a theory of change for the RSPCA NSW Emergency Boarding and Homelessness program.A series of in-depth stakeholder interviews were conducted with current and previous clients of the program, RSPCA Inspectors, and representatives from partner organisations, e.g., NSW Police, domestic violence services, and local council staff. Interviews were conducted to understand what changes because of the three RSPCA NSW Community Programs , the nature of the change, and who experience the change. These interviews, along with the theory of change workshop were also used to establish the materiality of various stakeholder groups and explore ideas of attribution (how much of change is due to other influences), deadweight (what would have happened anyway), benefit period (how long does the change last), and drop-off . Interviews also invited participants to reflect on other stakeholders that might have experienced change because of the program. Client interviews were used as proxy to identify and reflect on changes experienced by their animals.Current and previous RSPCA NSW Community Programs clients were invited to participate in interviews by their RSPCA case workers. Participants were chosen based on their diversity of experiences. Interviews were semi-structured and followed a planned interview guide for clients, RSPCA NSW staff, and external stakeholders, respectively . They weInterviews were audio-recorded and transcribed using the transcription tool in Microsoft Word. The interview audio was reviewed manually by GM to correct errors in the transcription. Thematic analysis of interview transcripts was conducted using an inductive approach. Transcripts were coded in NVivo independently by GM and JR. Emergent themes were discussed jointly by the research team to ensure the validity and reliability of the findings. Transcripts were re-reviewed frequently during the analysis to ensure the interpretation of themes remained true to the interview transcripts.A questionnaire was used to determine outcome incidence and quantify the amount of each outcome experienced by clients of RSPCA NSW Emergency Boarding and Homelessness program . The queClients were asked to choose the extent to which they agreed with a set of statements about the nature of the change they experienced because of their participation in the program on a 5-point Likert scale from A lot WORSE to A lot BETTER. Clients were also asked open-ended questions about what they considered to be the most important change they experienced, and what would have been different if they were not able to access the RSPCA NSW program.Clients that responded a little BETTER or a lot BETTER on the corresponding indicator question were considered to have experienced an outcome. Where there was more than one indicator question, a client was considered to have experienced that outcome if they responded a little BETTER or a lot BETTER to any of the indicator questions. Client responses to the open-ended questions were coded and responses that showed the client had experienced an outcome were also included.The value of outcomes was explored through the in-depth stakeholder interviews using a relative value and stated preference approach. Desktop research was undertaken to identify suitable financial proxies for each outcome experienced by each stakeholder and to determine appropriate discount factors .The value of program inputs for the 2020–2021 financial year was determined by examining program financial records. Interviews with program staff identified additional donated and in-kind inputs.The social value of the RSPCA NSW Emergency Boarding program was calculated by assigning financial proxies to represent the social value created by each outcome as experienced by that stakeholder group. Financial proxies were selected based on client interviews and desktop research. The social value associated with each outcome was calculated by multiplying the outcome incidence by the value of the financial proxy. The relative value of outcomes was determined through client interviews and questionnaire responses . Clients were asked during interviews to state the value of changes experienced. Clients were also asked in the questionnaire “What are the most valuable changes that have happened for you as a result of your experience with RSPCA?”.Discount factors for each outcome including deadweight (the amount of the outcome that would have happened anyway), attribution (how much of the outcome was caused by other organisations or people), displacement (how much the outcome displaced other outcomes), and drop-off  were estimated based on client interviews and questionnaire responses to the questions “have you received support from other agencies/people?”, “how much of this difference in your life is due to RSPCA? Can you estimate a %?”, and “what do you think would be different for you now if you had not accessed assistance for your animal/s through RSPCA?”.The results of the SROI analysis were presented to RSPCA Programs staff, interview participants, and current clients for validation and feedback before finalizing the model. A conservative approach was adopted for decisions on data and assumptions used in the SROI calculation. Values in this report might understate the actual value created.Note: the names of all research participants and their animals have been changed to protect their identities.The RSPCA NSW Emergency Boarding and Homeless program assisted 627 animals belonging to 259 clients in the 2020–2021 financial year. Of these, 446 animals accessed emergency boarding or foster care, spending a total of 12,206 days in care. Veterinary treatment was also facilitated for 296 of these animals, worth AUD 71,291 in total.In depth interviews were conducted with 3 program clients, 8 RSPCA inspectors, and 2 external stakeholders, one from NSW Police and one from a local council. Questionnaire responses were received from 29 program clients. Interviews and questionnaire responses were used to refine the RSPCA NSW Emergency Boarding and Homelessness program theory of change .The SROI investigated outcomes for four stakeholder groups who were determined to be material: (1) program clients; (2) animals of program clients; (3) RSPCA Inspectors; (4) animal pounds and shelters .The importance of the human–animal bond to clients of this program cannot be overstated; the support, affection, and encouragement provided by clients’ animal companions helped them to feel safe, motivated, needed and in many cases, gave them a reason to live .Clients who were assisted by the RSPCA NSW Emergency Boarding and Homeless program were able to extend or enhance their bond with their companion animal and because of the support received clients experienced improved mental health and wellbeing. These were the outcomes most valued by clients and the ones experienced most frequently. The language used by clients both in interviews and in open-ended questionnaire responses indicated that in some circumstances their animals were their reason to live or the thing preventing them from dying by suicide.“That I got to be able to still have my cat because there are times in my life where my cat has saved me [from suicide].”Stacy—program client.“[My cats] are the reason I feel carpet under my feet in the morning. Over the years my life got to the point where I’ve lost so much, and not just material stuff and money, but all the other shit that goes with it, and they were my anchor. They made me come home every night. Be there to feed them by 6 o’clock every. Single. Day. I’m up every single morning to put food in their bowls. Their bowls are washed out religiously. These are the things I’ve got to do every day. That’s what the cats mean to me. They basically took the place of an antidepressant.When you’re in the situation I was in—a crisis situation—you’re not connected to anything. You’re quite alone and even though there were services around me the cats kept me focused on what I needed to do. Do you know what I mean? It wasn’t all lost. The girls were coming with me and that made it sort of better. [When I knew the cats were safe] it was just like “I’m good now. I can do anything.” It was just really, really important that I had them there and I knew that they were coming back. Yep 100%, that was just everything.”Amber—program client.“He means everything to me. He is my life. He is the only one that I have in my life. He keeps me sane. I don’t know what I would do or where I would be without him. He is very special to me. He’s not just a pet, he is my companion and my support. He is my everything. I cannot see myself living without him. I am one of the lucky ones to have a dog.”Kimberly—program client.“[Without RSPCA] it would be a lot different. My life would be a little bit empty without them because I’ve spent the last 10 years looking after them. My life would feel empty without them. Like if you have animals and cats in your life and then all of a sudden you don’t have them, because you had an accident and went to hospital, and the cats get re-homed, you know that would be a devastating thing to go through. I would miss them terribly. And even small jobs like cleaning the kitty litter and stuff like that. There’s nothing that can replace the happiness having a cat can bring you. I’d hate it and the animals wouldn’t like it either.”Scott—program client.Clients also experienced reduced financial stress and social isolation, improved safety, improved physical health, and were better able to access care for themselves .“[The most valuable thing was] knowing there is help out there, as I am a single mum and do not have a job, so I could not afford the vet treatment. I would have been forced to surrender her, but because of RSPCA’s help we could stay together as a family and I am so grateful to be able to pay this off, it means everything to me. RSPCA will never know how much I appreciate what they have done for us.”Diana—program client.All except one of 29 clients who completed the questionnaire (97%) reported experiencing at least two positive outcomes. The median number of positive outcomes experienced was five (average 4.8). The outcomes experienced most frequently by clients were an extended or enhanced human–animal bond , improved mental health and wellbeing , decreased financial stress , and increased social inclusion and decreased social isolation  improved wellbeing as a result of preserving or improving the human–animal bond ; (2) access to safe accommodation ; (3) improved physical health , the animals would be seized and taken into RSPCA custody, separated from their families, and housed in an animal shelter. Without community assistance programs like the RSPCA NSW Emergency Boarding and Homelessness program, taking animals into custody can be the only option available to Inspectors to ensure the safety of the animals involved. Hence, the RSPCA Inspectorate was determined to be an important stakeholder.Animals that are assisted by the RSPCA NSW Emergency Boarding and Homelessness program might otherwise be referred to the RSPCA Inspectorate for several reasons. Animals might be abandoned at a property without care when a client is taken to hospital or becomes homeless, animals might also be referred for investigation by the Inspectorate for possible neglect or failure to provide veterinary care. Through the Inspectorate, these cases are investigated and potentially prosecuted under the Two outcomes experienced by RSPCA Inspectors were considered material and included in the model . While b“It’s a huge relief to have programs available to assist. If I had more cases that programs couldn’t assist with, I’d feel stressed, and it would be quite difficult. I can go home and sleep at night and not worry that I left the dog in an environment that wasn’t ideal and there’s no monitoring. Or I’m going to have to take someone through a court system for something that I know ahead of time, it’s going to go under the Mental Health Act. You have to question why you’re taking that route in the first place. But if there are significant animal welfare issues and you can’t leave the animal, you’ve got no other choice. I mean, I would hate to be plagued with that on my mind because my mental health will start to be affected and I’d probably end up needing assistance myself.”RSPCA Inspector.The RSPCA NSW Emergency Boarding and Homelessness program prevented animals arriving at animal shelters or pounds by reducing the number of animals abandoned without care or surrendered by their owners. Abandoned and surrendered animals often experience considerable distress when separated from their families and placed in a shelter environment. They are not always medically or behaviourally suitable to be rehomed, in which case they might be euthanized. Those who are suitable to be rehomed can require substantial investments of time and resources to ensure they are behaviourally and medically ready to be adopted.Animal pounds and shelters were determined to experience two material outcomes that have been included in the model based on interviews with RSPCA NSW Community Programs staff, RSPCA NSW Inspectors, and program clients. The outcome incidence was determined based on client responses to the question ‘What would have happened to your animal/s if RSPCA could not assist you?’ The two outcomes included were (1) fewer animals abandoned without care; (2) fewer animals surrendered for rehoming .“They would’ve taken my dog away from me if I couldn’t find help.”Nathan—program client.“I was very stressed at the time and financially strained. [Without RSPCA] I would have had to surrender my animals.”Angela—program client.No clients reported negative outcomes as a result of the RSPCA NSW Emergency Boarding program through in-depth interviews. In the client questionnaire, six of 29 clients (21%) selected ‘a little worse’ or ‘a lot worse’ for one or more of the Likert-scale outcome questions . In addi“I was not happy at all with the services from RSPCA… they didn’t ask my permission before treating my dog.”Veronica—program client .Negative outcomes expressed through the Likert-scale questions were included in the SROI model.Financial proxies selected for each outcome and the rationale for their selection are detailed for each stakeholder group in tables in When asked “what are the most valuable changes that have happened for you as a result of your experience with RSPCA?” the outcomes most often mentioned by clients were (1) improved mental health and wellbeing ; (2) extended or enhanced human–animal bond or improved animal health or wellbeing ; (3) decreased financial stress ; (4) increased social inclusion/decreased isolation .Deadweight is a measure of the amount of outcome that would have happened even if the activity had not taken place . ClientsAlmost all clients reported that they would have experienced some negative outcome if they had not been able to access the program  based on their response to the open-ended question ‘What do you think would be different for you now if you had not accessed assistance for your animal/s through RSPCA?’ in the client questionnaire. Two clients did not answer this question.The most frequently reported negative outcomes were: (1) losing their animal/s  ; (2) increased stress or worse wellbeing ; (3) worse animal health or wellbeing ; (4) worse financial stress .A total of 2 of 29 clients (7%) mentioned other options that might have provided some of the positive outcomes experienced as a result of their participation in the RSPCA program. The two alternatives suggested were seeking emergency boarding for their animal with friends or boarding their animal through a private boarding facility:“I would have found a friend to look after him.”Jason—program client.“I would have been forced to find the money for private boarding services.”Angela—program client.Hence, a deadweight of at least 7% has been applied for all outcomes. We consider this conservative given the demand for the program, the experiences described by RSPCA Inspectors, and the lack of alternatives available as described by clients and external stakeholders in interviews. A deadweight of up to 25% has been applied for some outcomes, e.g., ‘Improved wellbeing as a result of preserving the human–animal bond’ and ‘Fewer animal surrendered by their owners’ acknowledging that these outcomes might have occurred for some clients without receiving support with their companion animals .Attribution is an assessment of how much of the outcome was caused by other organisations or people . For outWe consider this discount to be conservative. Several clients mentioned not receiving assistance from any other agencies. For example, when asked about the support received from other agencies or organisations through their experience two interview clients responded:“No. No, I don’t know anybody.”Robert—program client.“I got online so you know like 12 months through the whole disaster, the drug raids and bloody jail and everything like that. And the cheapest thing I could find was over in [a distant suburb] and it was like $450 a week… for 5 cats per week was around 700 bucks.”Amber—program client.Displacement is an assessment of how much the outcome has displaced other outcomes, for example, if the activities of the program prevent people experiencing the same changes somewhere else . For mosOne outcome where some displacement has been included is ‘Extended or enhanced human–animal bond’. Supporting clients to keep their animal/s displaces potential adopters of these animals from experiencing this outcome if the animal were rehomed or surrendered to a pound or shelter. Likewise, animals rehomed might ultimately have similarly beneficial relationships with a new family, notwithstanding the stress and trauma this rehoming process would cause to the animal in the short to medium term. Hence, a displacement of 25% has been included for these outcomes .A benefit period is the amount of time an outcome is applied in the SROI model, while drop-off reflects the deterioration of an outcome over time . The benOther outcomes continue for the remainder of the life of the companion animal, for example, ‘extended or enhanced human–animal bond’ as experienced by clients and ‘improved wellbeing as a result of preserving or improving the human–animal bond’ as experienced by clients’ animals. Other benefits might endure for much longer, for example, the benefits to mental health and wellbeing of supporting a person in crisis; however, a benefit period of one year has been applied so as not to overclaim.Likewise, the drop-off value varied between outcomes depending on the nature of the outcome. For some outcomes, there was no drop-off over time, for example, the value of preserving the human–animal bond as experienced by the animal. Rather than decreasing in value over time, this relationship strengthens and becomes more important to the animal, while the animal also becomes less able to find a different home with age. A 5% drop-off was applied for ‘extended or enhanced human animal bond’ as experienced by both clients and their children, however this is likely to be conservative . In manyNB to be conservative when claiming, the net present value of outcomes with a benefit period of nine years has only included the value of these outcomes for the first three years.Inputs for the RSPCA NSW Emergency Boarding and Homelessness program, which supported 259 people with 627 animals in the 2020–2021 financial year, were determined through interviews with RSPCA NSW Community Programs staff and examination of program financial records. Values are based on actual costs for the 2020–2021 financial year. Where products and services have been donated or discounted the full market value has been included in the analysis. The total inputs were valued at AUD 642,489 including people costs, veterinary treatment, animal boarding, volunteer time, and other non-people and overhead costs.The benefits experienced by clients and their animals accounted for 95% of the total value created for all stakeholders, which was assessed to be worth over AUD 5,000,000 .The greatest share of benefits was created for program clients, valued at AUD 2,887,569 for the 2020–2021 financial year or an average social value generated per client of AUD 11,149. Program clients’ animals were the stakeholder group experiencing the next largest share of social benefit, with a total of AUD 2,151,633 for all animals, or AUD 432 per animal assisted. Overall, the RSPCA NSW Emergency Boarding and Homelessness program was estimated to have generated AUD 8.21 in social value for each AUD 1 invested into running the program .This SROI demonstrates the RSPCA NSW Emergency Boarding and Homelessness program provides a considerable social return on investment, with most of the benefit shared between clients and their companion animals. The bulk of the benefit can be attributed to preserving the human–animal bond and to improvements to the mental health and wellbeing of both human and animal family members. This study demonstrates the value of using SROI methodology to provide a comprehensive and quantifiable analysis of both financial and non-financial outcomes for a social program, enabling a more holistic understanding of its impacts.The RSPCA NSW Homelessness and Emergency Boarding program supports people experiencing the greatest challenges of their lives, a time when they have their greatest need for social support. This study, consistent with previous studies, demonstrates that companion animals can be critical for crisis recovery, providing a sense of safety, family, purpose, routine, and a source of unconditional, judgement free love, affection, and understanding ,5. CompaFor those experiencing homelessness, physical or mental health crises, or any other crisis, the support and encouragement they receive from their animal companions can be literally lifesaving. Clients of the RSPCA NSW Homelessness and Emergency Boarding program often experience almost complete social isolation, with their companion animals providing their only source of social support . Social Many people assisted by the RSPCA NSW Homelessness and Emergency Boarding program are referred to the program by the RSPCA NSW Inspectorate, often after receiving a complaint of animal cruelty or neglect from a concerned neighbour or member of the public. These complaints to the Inspectorate can be the first indication a socially isolated person with an animal is not coping. Opportunities for enforcement agencies, especially police, to act as referral pathways for social support have been noted in the literature ,33, and Caring for companion animals can be an important barrier to seeking help, especially for those with a limited support network. Accessing health care, especially in-patient treatments, and accessing temporary accommodation can be difficult or impossible for those without someone to care for their animals while they are away. The possibility of having to relinquish their animals, or worrying about their care while staying with neighbours or family can be enough of a deterrent to cause people to delay or avoid seeking the care they need. This has been noted as important amongst older people, those with disabilities, those experiencing domestic and family violence, and those experiencing homelessness ,9,29,36.There is currently a major shortage of crisis accommodation options for animals and demand for these services far outstrips their capacity. This is compounded by a general shortage of animal-friendly housing including private rentals, temporary accommodation, and crisis refuges ,38,39. FWhile the study provides valuable insights into the social value created by the RSPCA NSW Emergency Boarding and Homelessness program, it also has some limitations. The relatively small sample size might not have captured the full range of experiences of clients and their perspectives related to the program. In addition, there is inherent subjectivity in assigning monetary value to social outcomes. In this study, clients were not asked how RSPCA NSW could have provided a better service—this is a potential limitation. Future qualitative research could explore how various program activities contribute to outcomes experienced by stakeholders and how these can be optimized for clients with different needs.Programs that support those experiencing homelessness, physical, or mental health crises or any other crisis situation with their companion animals are a vital component of aiding recovery. When their animals are safe and cared for, people can focus on their own recovery and safety. For those who have been through challenging times with their animals, having survived together creates an even stronger bond between humans and animals. Hence, being able to live and recover together is essential. Keeping people together with their companion animals or ensuring they are reunited as soon as possible can reduce stressors and improve outcomes for people and animals.There is a need to expand wholistic animal-specific social support programs like the RSPCA NSW Emergency Boarding and Homelessness program. There is also a need for greater integration of services that consider animals as part of a person’s family unit and part of their social support system. A companion animal can be a person’s greatest strength and should be considered as such by enforcement agencies, medical providers, mental health services, and social workers.


Algorithms are becoming prevalent but are often opaque and need external validation to assess whether or not they meet their purported objectives. The purpose of this study is to validate, using the limited information available, the algorithm used by the National Resident Matching Program (NRMP) whose intention is to match applicants to medical residencies based on applicants’ prioritized preferences. The methodology involved first using randomized computer-generated data to overcome the inaccessible proprietary data on applicant and program rankings. Simulations using these data were run through the compiled algorithm’s procedures to obtain match outcomes. The study’s findings are that the current algorithm’s matches are related to program input but not to applicant input, the applicant’s prioritized ranking of programs. A modified algorithm with student input as the primary factor is then developed and run using the same data, resulting in match outcomes that are related to both applicant and program inputs, improving equity. With the rise and spread of computing power, algorithms have become prevalent and process large volumes of transactions in diverse functions and organizations. The algorithms intend to institute standard operating procedures, reduce transaction costs, and provide benefits for the purveyors, adopters, and users of these algorithms.In healthcare delivery, algorithm protocols compared to physician decision-making may play a beneficial role by alleviating racial biases by treating everyone equally through standardized methods and have been found to reduce the incidence of amputations in favor of revascularizations for black patients .Over time, however, concerns have been raised about unconscious algorithm bias that may perpetuate and exacerbate social inequities. Algorithms are designed often to serve mainly the needs of the privileged, and their performance or procedures cannot be adequately scrutinized ,3. AlgorA widely used algorithm was found to be biased against Black patients because it used health costs as a proxy for health needs, and Black patients spend less money than other patients for the same level of need . SimilarAlgorithmic bias is widespread across a variety of contexts, and will increase with the growing prevalence of algorithm use for analysis of big data, creating an increasing need for their validation There have been calls for algorithms to be validated with respect to whether or not they meet an organizations clinical needs , whetherAlgorithms, however, are often not validated by practitioners. “How to ensure that the algorithm is fair, how to make sure the algorithm is really interpretable and explainable—that’s still quite far off” . FairnesFour possible areas to address bias in practice are: 1. At the problem formulation stage, 2. At the data level, 3. Through modification of the algorithm, and 4. By post-processing the output . StakehoThis article examines the algorithm used by the NRMP that matches more than 60,000 medical graduates to more than 30,000 residency positions in over 4000 hospital programs annually. The matching algorithm that is currently used was last revised more than a generation ago ,24 and tEach medical graduate submits to the NRMP her/his ranking of programs to which they have applied. Similarly, each program submits to the NRMP its ranking of applicants. The NRMP’s algorithm has been called applicant-proposed because it first tries to place an applicant into her/his most preferred program, subject to the program’s number of residency positions available. Initial matches are tentative, and a student may be dematched in a subsequent iteration if that program ranks another student higher. The matching algorithm runs until all applicants and programs are considered. The NRMP intends to use an algorithm that prioritizes the preferences of applicants, placing them in the most preferred program possible on their rank ordered list .The main objective of the matching algorithm is to fill residency positions by matching applicants to programs based on applicants’ prioritized preferences. Students have long raised concerns, however, that the current algorithm favors programs over students ,27. StudThere is a research gap in accessing and validating algorithms in health care. Machine learning for health compares unfavorably to other areas regarding reproducibility metrics such as dataset and code accessibility . The inpThis study of the medical residency matching algorithm is novel in that it shows how the widespread problems of inaccessibility to input data and limited information about the algorithm’s procedures can be overcome. It introduces a statistical method for measuring the relative benefits that various stakeholders derive from the algorithm. Finally, it shows how an algorithm can be modified to move towards being more beneficial to a disadvantaged stakeholder.The NRMP provides limited information about the annual Match. It provides aggregate statistics on the annual residency match and a video description of how the algorithm seeks to match five students and three programs ,33. It dUsing computer-generated randomized data helps to cover a broad range of variable values. Student rank order lists ranged between 0 and 20 programs, ranked in random order. Program rank order lists were generated with lengths between 60 and 90 students, ranked in random order. The program lists are generated to be consistent with the student lists, i.e., only those students that ranked by a particular program are in turn ranked by the program. We ran 2600 simulations of the algorithm using the randomized student and program rankings data to determine match outcomes. All simulations were done in Python version 3.9.12. The researchers ran the current NRMP algorithm using the software code available in Henry Wilde’s matching library  to obtaiNext, the study performed multiple logistic regression of the binary dependent variable, whether or not a student matched to a program, against the focal independent student and program ranking variables in order to examine their influence. If the algorithm is fair, both the student and program rankings are expected to be significantly related to the probability of a match. The logistic regression included additional independent control variables–length of a student’s rank ordered list, length of the program’s rank ordered list, number of positions available in a program, and interaction variables. Related to a bivariate analysis, a logistic regression analysis with multiple linear and non-linear independent variables provides a more stringent test for the existence or absence of a relationship between the data inputs and dependent variable.After finding that the match outcomes were related to program rankings, but not student rankings, the researchers modified the current algorithm. In the current algorithm, students’ rankings are considered first but matches are tentative. Program rankings are then used to determine if a match is permanent and to break ties among students. Thus, program rankings are the primary factor in the matching decision and break ties among students. The algorithm was modified so that student rankings are the primary factor in attempting a match, while the program rankings are the secondary factor and used to break ties among students. The revised algorithm was developed by the researchers in Python. Logistic regression analyses were conducted using R version 4.2.2.Using the same data used in the current algorithm, 2600 simulations were also run with the modified algorithm to determine match outcomes. Running the same data across the two algorithms, controls for the data, and focuses the analysis on the different procedures across the two algorithms. The match outcomes were again logistically regressed against the student and program rankings, as well as the control independent variables. The coefficient estimates for the program-centric and student-centric algorithm models were examined to assess their relation to the match outcomes. We examined the coefficients and the odds-ratios for the program ranking and student ranking variables, comparing them within an algorithm model and across the two algorithm models.The third row shows that in the modified algorithm model the program’s ranking of students (-0.135) remains significant at the 0.01 level, but now the student rankings coefficient (-4.991) become significant at the 0.01 level, suggesting that a lower (better) program ranking and student ranking are associated with a higher probability of a match. Thus, modifying the roles of student and program rankings in the algorithm has reduced the current algorithm’s bias and has enabled both programs and students to have a significant influence on the matching outcome.In the second row, the current algorithm’s model shows that a lower (better) program’s ranking (0.64) significantly (at the 0.01 level) improves the odds of a match. A change in student’s ranking of the program (1.01) has no effect on the odds of matching, however. Thus, the main purpose of the matching algorithm to take into account both the preferences of students and programs is not being achieved.In the third row, the modified algorithm’s model shows that both the program’s ranking of students (0.87) and student’s ranking of programs (0.01) odds ratios are below 1 and significant at the 0.01 level. Thus, the bias in the current model has been reduced in the modified model, where both students and program rankings influence the odds of a match.The table results also show that program’s number of positions increases the odds of a match in the current model (2.10) and in the modified model (1.20). The odds of a match can be increased by students applying to programs offering more positions, and by programs offering more positions. The rest of the variables have odds ratios close to 1, suggesting little relation to the probability of matching.Finally, row 4 in Medical residency positions are competitive and challenging. Research is gradually providing insights on how to improve the recruitment, management, and well-being of residents and their patients. Candidates have been found to favor an 8-9am start time, four-hour duration, utilizing one-on-one interviews, lasting 15–30 minutes each . Though This study in the medical residency match context showed how two challenges of bias verification and algorithm modification to reduce the bias may be overcome despite little information available. The study first used randomized computer-generated data for the inputs–student and program rankings–that were consistent with their publicly disclosed aggregate mean and range statistics. It then compiled the algorithm’s steps using publicly reported descriptions of the algorithm. It ran thousands of simulations of the current algorithm using the computer-generated data to obtain match outcomes. These match outcomes were then logistically regressed against the data inputs of two stakeholders–graduate medical applicants and programs–to examine their explanatory power. The use of multiple logistical regression including linear and non-linear control variables provides a more stringent for the existence or absence of a relationship with the dependent variable than a bivariate analysis.The study found that the current algorithm, which has been in use for over a generation, is related to programs’ input but not applicants’ inputs. The students’ rankings of positions was found unrelated to match outcomes, contrary to the intended purpose of current algorithm that seeks to prioritize the preferences of applicants, and place them in the most preferred program possible on their rank ordered list. The study provides additional evidence from the medical residency context, which has used the same algorithm for over a generation, that the existence of biased algorithms may be widespread.After having observed bias in the current algorithm, we modified the algorithm’s order of procedures to make student rankings have a primary influence. When inputing the same data, the modified algorithm resulted in significant influences by both stakeholders–applicants and programs–suggesting that the matching algorithm can be improved and result in a positive sum, rather than a zero-sum, win-lose situation among stakeholders. The overall match performance, in terms of the percentage of applicants matched to programs, was not significantly different between the current and modified algorithms.The findings of the study helped develop a framework shown in The initial phase shows what an algorithm considers first and displays the face of the algorithm. This initial face may be deceiving because the outcomes are mostly determined by procedures in the middle and final phases. The current matching algorithm is described as student-proposed, but matches so derived are tentative, and students may be dematched from a program by later procedures. In contrast, the modified algorithm incorporates from the start the joint ratings of both students and programs.The middle phase of the algorithm shows the primary screen used for processing the majority of the cases. The current algorithm’ main screen is the program’s ranking of students, while the modified algorithm employs the student’s ranking of the program.The final phase shows how the last residency spot in a program is decided when a tie among students exists. In both algorithms, the ties are broken by the program’s ranking of students.For the algorithm to be more inclusive, participation of multiple stakeholders is desirable in a phase and across phases. The current algorithm has only students participating in the initial phase, and only programs are active in the middle and final phases. In the modified algorithm, both students and program rankings are taken into account in the initial phase, students are active in the middle phase, and programs are active in the final phase, suggesting joint and interweaving participation over time.Lastly, the transparency of the algorithm is influenced by its number of steps. Complex algorithms go through more steps and reduce public understanding. In the current algorithm, initial matches proposed by a student are tentative and can be reversed by later procedures, increasing the number of steps it goes through for each applicant. In contrast, the modified algorithm considers jointly all students and programs, and arrives at final matches with fewer steps in a simpler, more transparent way.A validation examination of the existing residency matching algorithm suggests that it does not achieve what it purports to do–prioritize the preferences of applicants, and place them in the most preferred program possible on their rank ordered list. Its initial tentative matches are proposed by student applicants, but student ranking are not related to the final match outcomes. The match outcomes are related to input program rankings. The current algorithm goes through many iterative steps as initial matches may be dematched reducing its transparency. The modified algorithm, in which student preferences are primary, results in matches that are related to the inputs of both students and programs, arrives at final matches with fewer iterative steps, and is more transparent and inclusive.The literature has noted that along with the growth and prevalence of algorithms, concerns have arisen about whether they also create, perpetuate, and exacerbate social inequities among affected groups. Biases may arise in an algorithm’s data inputs, processes, or application. Some stakeholders that are affected by the algorithm often have limited access to the data which may be proprietary and confidential, as well as have limited access to the code and little understanding of how an algorithm’s procedures work. Consequently, stakeholders face formidable challenges in validating the algorithm, identifying, and proving that a bias exists, and then modifying the algorithm in a way that reduces the bias.This study’s findings provide evidence that bias can exist even in a longstanding and widely used algorithm, that an algorithm may not deliver on its intended purpose, and that it can favor one stakeholder and disadvantage another. While data inaccessibility and opaque algorithm procedures pose difficulties, this study showed novel ways of overcoming them through randomizing data inputs and using the limited descriptive information on an algorithm’s procedures to replicate the algorithm and conduct an external verification. The study also showed that by modifying the order of its procedures, an algorithm can be improved to satisfy more stakeholders.Algorithms are difficult to access, understand, monitor, and change. Therefore, policymakers, such as government regulators or industry professional associations, should take an active role and require multiple stakeholders be engaged early in the design of the algorithm, External independent evaluators, with technical expertise, must be given access to de-identified data inputs and to the algorithms procedures, and must periodically monitor that the algorithm delivers on its intended purpose and that its benefits are equitably distributed across stakeholders. When benefits are not equitably distributed, the algorithm’s procedures should be modified to improve the distribution of benefits across stakeholders.Additional research is needed on how to design algorithms from the outset that meet multiple stakeholders’ interests. How to fine-tune an algorithm’s outputs to optimize or equalize inclusion is another area needing research. This study examined one algorithm at a time. Research is needed on how to combine inputs of multiple stakeholders and on the combination of the outputs of multiple algorithms. This study did not examine how to implement and integrate a modified algorithm into an organization. Future research is needed on how to implement in organizations algorithm changes that improve inclusion of multiple stakeholders.


Although governments have launched energy performance certification (EPC) programs to increase residential energy efficiency, residents’ perception and acceptance of these programs have been little studied. This study contextualizes residents’ intentions to participate in EPC programs in terms of their attributions to and attitudes toward climate change to find mediating factors and effects to help trigger collective action to reduce residential sector energy demand.This study employed a partial least squares structural equation modeling approach to empirically analyze a survey conducted on 400 residents of Edmonton, Canada, who participated in the Canadian energy efficiency rating and labeling program, EnerGuide.Using EnerGuide, a Canadian energy efficiency rating and labeling program, as an example, this study establishes a framework to explain that local residents’ internal and external attributions to climate change elicit positive attitudes (need to take action), increasing their recognition of energy efficiency program benefits, which further promotes their EnerGuide program acceptance and participation intentions. This study also reveals the mediating effects between variables. Residents’ attitudes toward climate change mediate the relationship between internal/external attributions and EnerGuide program acceptance, and they indirectly impact residents’ program acceptance and participation intentions, with this effect moderated by energy efficiency program benefits. Residents’ program acceptance also mediates the relationship between climate change attitudes/energy efficiency program benefits and the intention outcome.The study provides an example of the use of climate change discourse to motivate residents’ energy efficiency program participation. Despite overwhelming scientific evidence and a comprehensive set of policy instruments, concrete actions on climate change remain vague, and cooperation across the various sectors is lacking. Admittedly, climate change has become a collective action problem, and research is needed to understand the public’s perceptions, attitudes, benefits, acceptance, and intentions to act on climate-related issues and solutions . This stAs one of the most effective measures, building energy performance certification (EPC) aims to provide policymakers and relevant stakeholders with “how-to” guidance on the essential elements of implementing a building EPC program . BuildinArguably, EPC programs not only can be powerful instruments for conserving energy and reducing greenhouse gas (GHG) emissions in the residential sector but also give policymakers access to better building stock data and enable them to effectively monitor the impact of policies in the process of implementation. Despite the promising role of EPCs in the decarbonization of the building sector, low public participation handicaps the large-scale implementation and execution of EPCs . The wayA growing body of research on pro-environmental intentions and behaviors makes extensive use of the theory of planned behavior (TPB)  and valuAttribution theory (AT) concerns a cognitive process by which people analyze their own or others’ behaviors and infer the reasons for these behaviors . In receAn attribution may be caused by factors beyond the individual’s control , or an attribution to the individual’s own behavior  may be made . InternaAttitude is a psychological structure that refers to an individual’s positive or negative evaluation of a particular object, such as a person, an idea, an emotion, or an event . AttitudIn addition to the general perception of climate change, the benefits of environmentally friendly or energy efficiency programs have a positive role in motivating people’s acceptance and participation, which has been confirmed in research on nuclear power , home enBased on the literature review above, this study infers that internal and external attributions of climate change, residents’ attitudes toward climate change, and recognition of the benefits of energy efficiency have a positive impact on improving residents’ acceptance of the EnerGuide program and their intentions to participate in this program. The conceptual framework to be tested is shown in The EnerGuide program is the main energy rating and labeling system used in Canada to certify the energy efficiency of major consumer products  . EnerGuiAdditionally, cities offer a $400 discount on fees to help residents pay for assessments, which typically cost between $400 and $800. Since 2016, the EnerGuide rating system has changed. The new EnerGuide tab shows the gigajoules  that a house uses each year. The closer the annual GJ measurement is to zero, the more efficient the house is. Compared with other EPC programs such as ENERGY STAR in the U.S. and the EPBD in Europe, the EnerGuide is less investigated. This study on EnerGuide not only enriches the literature on residential energy efficiency but also provides a good example for developing countries such as China and India to learn from.The data used in this study were obtained from the Edmonton government’s open data website . EdmontoThe City of Edmonton -–2018 Climate Change and Energy Perceptions survey covers a series of questions . PLS-SEM is a component-based estimation method  that is CFA was used to assess the reliability and validity of the measurement model. As shown in This study also assessed the discriminant validity of all constructs using the Fornell–Larcker criterion . The ForBefore testing the structural model, to examine common method bias in the empirical results, we adopted the studies of R2 of residents’ attitudes toward climate change, residents’ recognition of the benefits of energy efficiency programs, residents’ acceptance of the EnerGuide program, and residents’ intentions to participate in the EnerGuide program were 0.734, 0.553, 0.480, and 0.499, respectively. Therefore, the model explained 73.4% of residents’ attitudes toward climate change, 55.3% of their recognition of the benefits of energy efficiency programs, 48.0% of their acceptance of the EnerGuide program, and 49.9% of their intentions to participate in the EnerGuide program. In addition, a measure of approximate fit of the study model, the standardized root mean square residual (SRMR), was 0.052, which was lower than the recommended threshold of 0.080, and the normed fit index (NFI) was 0.879, which was higher than the recommended threshold of 0.800, indicating that the model fit was good  and external attributions  had a positive impact on their attitudes toward climate change. At the same time, residents’ climate change attitudes had a significant positive impact on their recognition of the benefits of energy efficiency programs  and acceptance of the EnerGuide program . In addition, residents’ recognition of the benefits of energy efficiency programs had a significant positive impact on their acceptance of the EnerGuide program . Residents’ recognition of the benefits of energy efficiency programs  and their acceptance of the EnerGuide program  had a significant positive impact on their intentions to participate in the EnerGuide program.2f values , 0.15 (medium effect) and 0.35 (large effect). As shown in 2f values of the important paths in this study were between 0.037 and 1.237. Therefore, the results of the study showed that the exogenous variables of the structural model had good explanatory power with respect to the endogenous variables.In addition, to evaluate whether omitted constructs had a substantive effect on endogenous variables, this study calculated the effect size 2 values . Accordiβ = 0.213, p < 0.001) and external attributions  indirectly influenced their acceptance of the EnerGuide program through the mediating effect of residents’ attitudes toward climate change; this mediating effect was statistically significant at the 5% level, with t values of 5.149 and 2.817, respectively. The 95% bias-corrected bootstrap confidence intervals (CIs) of the indirect effects  did not include 0, indicating a mediating effect  indirectly influenced their acceptance of the EnerGuide program through the mediating effect of their recognition of the benefits of energy efficiency program; this indirect effect was statistically significant at the 5% level, with a t-value of 7.653. The 95% bias-corrected bootstrap CI of the indirect effect  did not include 0, indicating a mediating effect. Meanwhile, residents’ attitudes toward climate change  indirectly influenced their intentions to participate in the EnerGuide program through the mediating effect of their recognition of the benefits of energy efficiency programs; this indirect effect was statistically significant at the 5% level, with a t value of 3.455. The 95% bias-corrected bootstrap CI of the indirect effect  did not include 0, indicating a mediating effect. In addition, when testing the mediating effect of residents’ acceptance of the EnerGuide program, this study showed that residents’ attitudes toward climate change  indirectly influenced their intentions to participate in the EnerGuide program through the mediating effect of their acceptance of the EnerGuide program; this indirect effect was statistically significant at the 5% level, with a t value of 4.732. The 95% bias-corrected bootstrap CI of the indirect effect  did not include 0, indicating a mediating effect. Meanwhile, residents’ recognition of the benefits of energy efficiency programs  indirectly influenced their intention to participate in the EnerGuide program through the mediating effect of their acceptance of the EnerGuide program; this indirect effect was statistically significant at the 5% level, with a t value of 7.130. The 95% bias-corrected bootstrap CI of the indirect effect  did not include 0, indicating a mediating effect. Therefore, we conclude that there is a mediating effect of residents’ attitudes toward climate change, residents’ recognition of the benefits of energy efficiency programs, and residents’ acceptance of the EnerGuide program , internal attributions (β = 0.678) have a stronger direct influence on attitudes toward climate change. This finding also confirms the perspective of The main factors influencing residents’ intentions to participate in the EnerGuide program have several important theoretical implications. First, the study results indicate that both internal and external attributions of climate change have a significant positive impact on attitudes toward climate change. Compared to external attributions (β = 0.314), climate change attitudes have a greater impact on people’s recognition of the benefits of energy efficiency programs (β = 0.744). According to our literature review, no studies have investigated this particular scenario.Second, the relationship between residents’ attitudes toward climate change and the benefits of energy efficiency programs is positive and significant. Similarly, residents’ attitudes toward climate change also have a positive impact on acceptance of the EnerGuide program. This result means that the level of pro-environmental attitude not only enhances residents’ perception of the benefits of energy efficiency programs but also increases people’s acceptance of the EnerGuide program . The resThird, the benefits of energy efficiency programs have a significant positive impact on residents’ acceptance of the EnerGuide program – a concept that has not been studied before. This observation suggests that the higher residents perceive the benefits of energy efficiency programs, the more likely they are to accept the EnerGuide program.β = 0.181), residents’ acceptance of the EnerGuide program (β = 0.573) has a greater influence on their intentions to participate. This study demonstrates that residents’ acceptance of the EnerGuide program greatly affects their intentions to participate. Specifically, the higher the acceptance of the EnerGuide program, the more likely residents are to participate in it.Fourth, the benefits of energy efficiency programs and acceptance of the EnerGuide program have a significantly positive impact on residents’ intentions to participate in the EnerGuide program. Similar findings have been noted in previous studies . CompareThe conceptual model of climate change discourse established in this study may be an ideal theoretical model for predicting individuals’ intentions to participate in the EnerGuide program and can provide an important theoretical and practical basis for the government to effectively mitigate climate change and improve residential energy efficiency.This study highlights that climate and energy policies should be formulated in a two-pronged approach to effectively tackle today’s crises. In this respect, the study of EPCs in the residential sector has important implications.First, residents’ attitudes toward climate change have an indirect mediating effect on the relationship between the internal/external attribution of climate change and residents’ acceptance of the EPC program. This observation suggests that energy policy can effectively motivate residents’ participation in the EPC program within the discourse of climate change. In other words, the government’s efforts to promote educational programs on climate change and energy shortages can enhance residents’ attitudes toward climate change, thereby increasing their acceptance of the EPC program .Second, the benefits of energy efficiency programs have an indirect mediating effect on the relationship between residents’ attitudes toward climate change and acceptance of the EPC program. This result shows that residents’ awareness of the benefits of energy efficiency plays a vital role in linking climate change action and participation in specific energy efficiency programs. The government can use both traditional offline methods, such as public lectures and exhibition boards, and various new media, focusing on a combination of online and offline promotional activities to make residents fully aware of the positive effects of EPC programs on mitigating climate change and reducing residential energy consumption. Notably, the government should highlight in publicity the real benefits brought to residents by energy efficiency programs .Although this study provides important theoretical and practical implications by using climate change discourse as the basic theoretical framework, it focuses on intentions rather than outcomes or behaviors. Future research can consider studying actual outcomes, such as the number of households that joined the energy certification program. Meanwhile, the intention–behavior gap has recently been widely discussed in behavioral research in support environments. The proposed model can be used in practice, e.g., in interventions and policy formulation, to bridge this intention–behavior gap. Further work might also consider the role of long-term stable thinking and feeling patterns  in attitudes and intentions. Furthermore, since sociodemographic factors have a broad scope in explaining residents’ environmental protection intentions, future researchers can include more sociodemographic factors such as gender, age, income, educational level, ethnicity and religion.This paper developed a theoretical model based on climate change discourse to promote Edmonton residents’ intentions to participate in a local EPC program. The use of individuals’ internal and external attributions to climate change, attitudes toward climate change, and recognition of the benefits of energy efficiency programs can clearly explain residents’ acceptance of the EnerGuide program and their intentions to participate in this program. This study also identified the mediating effects among these variables. First, residents’ attitudes toward climate change have an indirect mediating effect on the relationship between internal/external attributions and acceptance of the EPC program. Second, residents’ attitudes toward climate change have indirect effects on their acceptance of the program and intentions to participate in the program, and this effect is mediated by the benefits of energy efficiency programs. Third, residents’ acceptance of the program has an indirect mediating effect on the relationship between climate change attitudes and intention outcomes. Similarly, residents’ acceptance of the program has an indirect mediating effect on the relationship between the benefits of energy efficiency programs and intention outcomes.Based on the results of the empirical analysis, this study recommends that the government carry out educational programs on climate change and energy shortages, enrich publicity strategies, and increase subsidy amounts as long-term intervention strategies to promote Edmonton residents’ intentions to participate in the energy efficiency certification program. The findings of this paper regarding climate policy and energy policy complement each other to help government agencies better understand the public’s attitudes toward climate change and perceptions of EPC programs to formulate comprehensive and effective energy and climate change management strategies to reduce residential GHG emissions. This study also provides a clear pathway out of the woods for collective action on climate change in the residential energy sector and makes it easier to build public support for policy action. More significantly, the established climate change discourse framework can be adapted and used in other contexts to understand and encourage local residents’ participation in a range of government energy efficiency ratings and labeling programs.The original contributions presented in the study are included in the article/XC: methodology, investigation, data analysis, and writing draft. ZG: supervision and editing draft. HZ: financial support and editing draft. All authors contributed to the article and approved the submitted version.The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.