Tax Exempt Organizations: IRS Increasingly Uses Data in Examination Selection, but Could Further Improve Selection Processes

Exempt organizations often provide charitable services, or in some instances, membership benefits in furtherance of an exempt purpose. They generally do not pay federal income tax. IRS examines exempt organization returns (Form 990 and others) to address noncompliance, which may promote confidence in the tax exempt sector. In 2016, IRS started using three analytical models using Form 990 data to identify potential noncompliance and select returns for examination.

GAO was asked to review IRS's use of Form 990 data. This report assesses (1) IRS's use of data to select returns for examination and, (2) the process IRS has established for selecting returns. GAO analyzed (1) examination data from fiscal years 2016 through 2019 including results from the largest Form 990 model, and (2) model documentation for a generalizable sample. GAO interviewed IRS officials and assessed IRS policies and procedures using relevant standards for internal control.

The Internal Revenue Service (IRS) used data to select almost 70 percent of its examinations of Form 990 returns in fiscal year 2019. Almost half of these examinations were selected using models that score returns for potential noncompliance (see figure).

Of the returns examined that were selected using the model, 87 percent resulted in a change to the return, indicating that IRS identified noncompliance. GAO found that the model did not improve change rates compared to prior selection methods and a higher model score is not associated with a higher change rate.

IRS has not fully implemented or documented internal controls in its established processes for analyzing data for examination selection. For example:

IRS has not defined measurable objectives for using data to select returns for examination . Without measurable objectives, IRS cannot assess how well it is doing or fully implement other internal controls.

IRS's models have deficiencies affecting the validity and reliability of return scoring and selection . IRS has incomplete definitions and procedures and did not always follow its definitions when assigning point values for identifying potential noncompliance for examination. As a result, return scoring by the models is not always consistent.

IRS did not consistently document the processing and use of data in decision-making on examination selection . Without such documentation, IRS cannot support its use of data in examination selection in all cases.

IRS does not regularly evaluate examination selection. Examination data were inconsistent across years and IRS only tracks one prior year of data. IRS also did not save data on all returns that the models scored. Without data and regular evaluations, IRS cannot assure that its models are selecting returns as intended and that deficiencies are identified and corrected.

GAO makes 13 recommendations, including that IRS establish objectives, revise model documentation, fully document processing and using data in decisions, and regularly evaluate examination selection. IRS agreed with all recommendations except one related to evaluating examination selection methods using consistent historical data over time. GAO continues to believe that this recommendation is valid as discussed in the report.

To be exempt from federal income tax, an entity must be organized for a  purpose specified in the Internal Revenue Code—such as providing  charity, enhancing social welfare, or furthering the interests of the  organization’s membership—and must operate in accordance with that  purpose. Internal Revenue Service (IRS) oversight can help ensure that  exempt organizations abide by the purposes that justify their tax  exemption. This oversight can also help safeguard the public’s confidence  in the integrity of the charitable sector.

The Tax Exempt and Government Entities (TE/GE) division within IRS  oversees exempt organizations by conducting examinations and other  activities to ensure compliance with the Internal Revenue Code.  Examinations are reviews of the books and records of exempt  organizations to determine whether they operated in accordance with  their exempt purposes, and paid taxes they owed. If TE/GE finds  noncompliance, it may impose excise taxes for certain violations, or—in  appropriate circumstances—it may revoke an organization’s tax-exempt  status. Limited resources have prompted TE/GE to try refining its  examination selection methods to focus examinations on the  organizations with the highest potential for noncompliance.

In a 2015 review, we found deficiencies in TE/GE’s methods for selecting  tax-exempt organizations for examination. However, we noted that in  fiscal year 2012, TE/GE had started analyzing more data reported on the  Form 990, Return of Organization Exempt from Income Tax, to identify  potential noncompliance and to select returns for examination. We did not  make recommendations on the use of data for examination selection.

You asked that we review TE/GE’s use of Form 990 data for exempt  organization compliance efforts. This report assesses (1) the use of data  to select tax exempt organization returns for examination; and (2) the  process TE/GE has established to select returns for examination.

To assess TE/GE’s use of data to select exempt organization returns for  examination, we analyzed TE/GE data for examinations closed from fiscal  years 2016 to 2019. The analyses included information on examination  closures and changes made to returns because of examinations. Based  on our testing of the data and review of documentation and interviews, we  determined that the examination data were reliable for the purposes of  assessing TE/GE’s selection processes. We reviewed the examination  selection process and outcomes from TE/GE’s Form 990 examination  selection model.

To assess the process that TE/GE established to select returns for  examination, we analyzed TE/GE documentation relative to five key  internal controls steps and four other controls that we selected from the  Standards for Internal Control in the Federal Government (Green Book or  GB). Internal control comprises the plans, methods, policies, and  procedures used to fulfill an agency’s objectives. We reviewed the  Internal Revenue Manual (IRM), work plans, desk guides, procedures,  examination selection processes, model documentation and other  documents. In addition, we analyzed a generalizable stratified random  sample of data queries from three examination selection models to verify  whether TE/GE tests and follows its approval and documentation  procedures for new data queries. We interviewed officials in TE/GE’s  Compliance Planning and Classification (CP&C) office and IRS’s  Research, Applied Analytics and Statistics (RAAS) division. For details on  our scope and methodology, see appendix I.

We conducted this performance audit from November 2018 to June 2020  in accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.

Filing Requirements for Exempt Organizations

IRS Form 990-series return or notice must be filed by most organizations  exempt from income tax under Internal Revenue Code section 501(a),  and certain political organizations and nonexempt charitable trusts.  TE/GE uses Form 990 reporting for promoting compliance and enforcing  federal tax law for tax-exempt organizations (see appendix II for a copy of  the Form 990 and a list of its schedules). Form 990 asks for information  about an organization such as: employees, governance, and compensation;  revenue and expenses; assets and liabilities; employment tax compliance; and  specific organizational issues, such as lobbying by charities and  private foundations.

TE/GE redesigned the Form 990 for the first time in nearly 30 years for  tax year 2008, and has made subsequent changes to the form (see  appendix III for a summary of the changes). For tax year 2017, which is  the most recent year of completed filing data, organizations filed 319,183  Form 990s. Beyond the basic Form 990, other versions include:

Form 990–EZ, Short Form Return of Organization Exempt from  Income Tax. This form reduces the filing burden on small tax-exempt  organizations. Organizations with less than $200,000 in gross receipts  and less than $500,000 in total assets may use it. For tax year 2017,  232,764 Form 990-EZ’s were filed.

Form 990–N, Electronic Notice (e-Postcard) for Tax-Exempt  Organizations Not Required to File Forms 990 or 990–EZ. Most small  organizations whose annual gross receipts are normally $50,000 or  less may file Form 990-N. For tax year 2017, 652,280 Form 990-N’s  were filed.

Form 990–PF, Return of Private Foundation. In addition to private  foundations, nonexempt charitable trusts treated as private  foundations are required to file Form 990-PF. For tax year 2017,  113,658 Form 990-PF’s were filed.

Certain larger organizations are required to electronically file their  returns. The Taxpayer First Act of 2019 requires all organizations to  electronically file Form 990’s for tax years beginning after July 1, 2021.  TE/GE can assess financial penalties for failing to file a required Form  990. As an employer, or if an exempt organization generates unrelated  business income, additional tax reporting requirements may apply, such  as for employment tax or unrelated business income.

Several TE/GE Entities Are Involved in Examination Selection Decisions

A 2017 TE/GE reorganization created CP&C to provide a centralized  approach to compliance planning, examination selection and assignment  and planning and monitoring activities. CP&C has three groups as  follows:  1.  Issue Identification and Special Review identifies and develops  issues for examinations or compliance activities and certain criteria for  examination selection.  2.  Classification and Case Assignment uses IRS staff known as  “classifiers” to review returns for examination under different  examination sources (see appendix IV). Classification is the process  of determining whether a return should be selected for compliance  activities, what issues should be the primary focus of the compliance  activity, and the type of compliance activity that should be conducted.  3.  Planning and Monitoring develops an annual work plan and  monitors performance. The work plan details the number of  examination starts, closures and other measures. It develops  classification requests to ensure that enough returns are available to  meet work plan goals.

TE/GE’s Compliance Governance Board (Governance Board) oversees  TE/GE’s compliance program, including CP&C operations such as  approving priority issue areas—known as compliance strategies. The  Governance Board also reviews program goals, considers metrics and  reporting, and reviews the performance of compliance strategies. The  Governance Board has five TE/GE executives plus counsel who are  voting members as well as three non-voting members.

The Exempt Organizations examinations group is responsible for  compliance activities. Examinations have various outcomes for an  organization. The most severe outcome is revocation of tax exempt  status.  Taxes—such as employment or excise—may be assessed as a  result of an examination. In fiscal year 2019, approximately $131 million  in taxes were assessed.

TE/GE conducts compliance contacts—non-examination correspondence  such as compliance checks and soft letters—that are used to handle  some compliance issues. For example, compliance checks determine  whether specific reporting or filing requirements have been met. A “soft  letter” notifies an organization of changes in tax-exempt law or potential  compliance issues. A response to these letters is not required. TE/GE  also reviews tax-exempt hospitals for compliance with certain community  benefit requirements. In fiscal year 2019, TE/GE closed 1,470  compliance checks, sent 3,955 soft letters, and closed 750 hospital  reviews. Compliance checks and hospital reviews can result in an  examination while responses to soft letters may result in a compliance  check.

Exempt Organizations Identified for Examination Originate from Many Sources

TE/GE identifies exempt organization returns for examination from many  sources and categorizes examinations into three groups, known as  portfolios: (1) Data Driven Approaches, (2) Referrals and Other  Casework, and (3) Compliance Strategies. All three rely on data, to some  extent, to make decisions on selecting returns for examination.

This portfolio uses analytical models and queries based on quantitative  criteria to identify potential examinations. TE/GE has three separate  models that review exempt organization data from Forms 990, 990-EZ,  and 990-PF for compliance. The models “score” returns for examination  based on potential noncompliance. The Form 990, 990-EZ and 990-PF  models have 354 unique queries. For purposes of this report, a query  reviews databases to identify responses on returns that may indicate  noncompliance because they do not meet certain criteria or expected  values, such as exceeding a dollar threshold. Exempt Organizations  Examination staff developed many of the queries, based on information  collected on the Form 990 after it was redesigned for tax year 2008,  according to TE/GE officials. As queries were developed, staff tested and  used them to identify certain potentially noncompliant populations and to  identify returns that were flagged by multiple queries.

Starting in fiscal year 2016, TE/GE began using queries in models. The  models use a scoring system that applies weights, or points, to each  query result to generate a score—which for the Form 990 model has  ranged from zero to more than 50—for a return. The models also screen  out returns that are approaching a statute of limitations date, if the  organization is not active, or has a current or recent examination history.  Since November 2017, staff have been able to submit potential  compliance issues for consideration through an online submission portal  for Governance Board approval. CP&C has the option of considering  whether these ideas result in model changes, according to IRS officials.

Twice a year, each model is run using the latest data, and generates a  Model Score Sheet (MSS). The MSS is a ranked list of returns that score  above a minimum threshold. A classifier uses the ranking to identify  returns for potential examination. Although the models screen for  examination status and statute of limitations, a TE/GE official said the  classifier also checks whether the statute of limitations date is near and  whether the organization recently had undergone an examination or  compliance check, as well as whether the return was identified under  another selection method. This official explained that a classifier checks  the criteria because conditions may have changed since the model’s last  run. The classifier selects returns to fulfill a stocking plan, which identifies  the number and type of returns to be examined to meet work plan  requirements. See figure 1.

Aside from the three models, TE/GE also uses other methods and data to  identify and develop compliance work. The Data Driven Approaches  portfolio includes approaches that TE/GE developed in partnership with  IRS’s RAAS division. The partnership began in 2016 and continues today,  according to IRS officials. The portfolio also includes some of the queries  that TE/GE ran prior to fiscal year 2016 for examination selection. Some  of these examinations remained open as of fiscal year 2019.

Although not all of the returns selected for examination in this portfolio  rely on data for examination selection, we describe them all below.

Referrals. Referrals are complaints about exempt organization  noncompliance made by third parties, including the public and other  IRS offices or divisions.

Post Determination Compliance. Sampling and queries are used to  identify organizations that file Form 1023-EZ.

Claims. Claims are requests for tax refunds, adjustments of tax paid,  or credits not previously reported or allowed.

Form 990 Queries (pre-model): These queries were run prior to  fiscal year 2016. Some of these examinations remained open as of  fiscal year 2019.

Training. TE/GE uses these examinations, selected based on various  methods, to teach examiners.

Other Projects. TE/GE initiated these examinations under older  compliance projects, using a variety of selection methods.

The Compliance Strategies portfolio consists of compliance issues that  originated from a Compliance Issue Submission Portal for TE/GE staff.  The strategies are approved by the Governance Board, which results in  adding the compliance strategy to the work plan. In fiscal year 2019,  TE/GE closed examinations under three compliance strategies, including  private foundation loans, and for-profit entities that converted to 501(c)(3)  organizations. Returns are selected using sampling or other uses of data.

Table 1 shows examinations closed for the three portfolios.

Once an examination is underway, an examiner may expand it to include  an organization’s returns for other tax years or other types of returns such  as employment tax returns. IRS refers to these additional examinations  as “pick-ups,” each of which is counted as a separate examination.  Examiners must obtain manager approval to expand an examination.

Examiners are required to check that an organization filed all returns that  are required. If the examiner finds that a return was not filed—such as an  employment tax return—and is unable to secure the return, he or she  may prepare a “dummy” return called a substitute for return (SFR). The  organization’s activities, records, and documents may then be examined.

In 2017, TE/GE hired a contractor to assess aspects of the exempt  organization process for examination selection, with a focus on the Form  990 model. In January 2018, the contractor released a report on the  development and operation of the models. The contractor released a  second report in July 2018 on the Form 990 model performance. The  contractor found the model was not always identifying the “next best  case” as TE/GE intended because scores did not consistently predict  certain measures of noncompliance.

Across both reports, the contractor made 17 recommendations, which we  discuss later in this report (see appendix V). As of March 2020, TE/GE  implemented one recommendation on model update submissions and  part of another on hiring assessments. In September 2019, TE/GE  initiated another study with the same contractor—with a planned release  of the report in September 2020—on developing alternatives to the Form  990 model.

Over Half of Exempt Organizations Selected for Examinations Are Identified Using Data, with No Assurance That the Models Produce Better Outcomes Reliance on Data for Examination Selection Has Increased in Recent Years

Since the Form 990 model was first run for fiscal year 2016, the  percentage of examinations closed that were identified by using data,  such as through models or queries, has increased each year, as shown in  figure 2. Almost half of these examinations are from the models.

This increased reliance on using data in selecting returns for examination  offers potential efficiencies. For example, a potential efficiency from using  data to find possible noncompliance could mean fewer steps for staff who  classify returns. Ultimately, this could allow TE/GE to shift staff from  classifying returns to doing compliance activities such as examinations to  confirm any actual noncompliance.

Another potential efficiency would be selecting more examinations that  find changes to the return. To measure the outcomes of examinations,  TE/GE computes a “change rate,” or the percentage of closed  examinations with a change to the return. In general, a higher change  rate indicates that more examinations found noncompliance.

Examinations selected using data have a slightly better change rate than  other selection sources (84 percent versus 82 percent) for closures in  fiscal years 2016 through 2019.

The Form 990 Model’s Contribution to Improving Change Rates Is Not Clear

Similar to all examinations that used data, the change rate for  examinations selected using data through the Form 990 model (87  percent) was higher than the change rate for other selection sources (82  percent) in fiscal years 2016 through 2019. However, we found evidence  that the changes identified in examinations did not clearly result from  using the Form 990 model’s scoring system. Specifically:

The model has not improved change rates compared to pre-model  Form 990 queries.

A higher model score is not associated with a higher change rate.

Most examination changes credited to the model come from pick-up  returns and SFR’s that examiners identify rather than from primary  returns identified by the model score.

Form 990 Model Scoring Has Not Resulted in Higher Change Rates, Compared with Pre- Model Queries

The scoring generated by the Form 990 model has not improved change  rates compared with the Form 990 queries that TE/GE used prior to the  model. The change rates for both the Form 990 model and the pre-model  queries, for fiscal years 2016 through 2019, was 87 percent.

Similarly, for the last 2 fiscal years, the change rate for all Form 990  models was roughly equivalent to the change rate for other selection  sources of exempt organization examinations. As shown in table 2, the  models had a slightly higher change rate in fiscal year 2018, and a slightly  lower change rate in 2019, compared to the other sources.

Higher Model Score Is Not Associated with Higher Change Rate from Examination

Form 990 model scores for returns do not consistently predict  examination change rates based on our analysis of examination closures  since the model’s first run in 2016 through fiscal year 2019; the scores  better predicted the rate at which returns were selected for examination.  See figure 3.

The figure shows little relationship between model scores and change  rates; change rates remained relatively flat as model scores increase.  While change rates were slightly higher for the less than 1 percent of  returns scoring 45 or above relative to lower-scoring returns, TE/GE only  examined 65 returns during fiscal years 2016 through 2019 that scored  this high. The overall correlation between model scores and change rates  is -.02.

A TE/GE official said that it is not difficult to find a small issue on a return,  which allows for a change regardless of score. To attempt to measure the  severity of an examination change, TE/GE developed a weighted disposal  score (WDS). However, TE/GE does not have documented criteria or  justifications for how the weights were developed. A TE/GE official  acknowledged that TE/GE has not used WDS because of questions  about how consistently the weights have been developed. If WDS was to  be used as a measure, TE/GE would need to ensure the adequacy of the  support for the related weights and scores.

According to TE/GE’s fiscal year 2020 Program Letter, the model relies  on quantitative criteria, “which allows TE/GE to allocate resources that  focus on issues that have the greatest impact.” To the extent that a higher  model score does not predict a higher change rate, the model is not  selecting returns with the greatest impact. Further, taxes assessed per  return also indicate that examinations are not having the greatest impact.  For fiscal years 2016 through 2019, the examinations credited to the  model averaged $2,460 in proposed tax assessments per return,  compared with an average of $19,042 for the rest of the exempt  organization examinations.

TE/GE acknowledged that its scoring methods are limited because it does  not utilize modern data practices. It contracted for a study, to be  completed in September 2020, of alternative model architectures and  scoring methods that incorporate best practices for using criteria and  options for scoring returns.

Most of the Changes Credited to the Form 990 Model Are Driven by Examinations of Returns Not Identified by the Model Score

As shown in table 3, the Form 990 model scoring did not account for most  closed examinations and examination changes credited to the model  during fiscal years 2016 through 2019. Rather, examinations of “pick-up”  returns and substitutes for returns (SFRs) accounted for most closed  examinations and produced a higher change rate than examinations of  primary returns scored by the model. Examiners find these other returns  during examinations of returns identified by the model.

The higher change rates for pick-up and SFR returns compared to the  primary returns identified by the model support TE/GE’s policy to examine  all pick-up returns and SFRs that meet examination criteria. However, this  raises questions about how well the model identifies noncompliant  returns. Given the lower change rate for the returns the model scored, the  queries for noncompliance on the Form 990 may not be effective. While  the model includes queries on noncompliance related to “pick-up” issues  such as unfiled employment tax returns, the necessary data were not  available to allow us to analyze how often these queries identified the  primary return for potential noncompliance. As discussed later, an  analysis of queries could provide insight into the validity of the model.

TE/GE Has Not Fully Implemented and Documented Internal Controls for Assessing and Using Data for Examination Selection

Internal control should be an integral part of an agency’s operational  processes and structure to help managers achieve their objectives on an  ongoing basis. When evaluating implementation, management  determines if the control exists and is operational. A deficiency in  implementation exists when no such control is present or is not  implemented correctly, preventing objectives from being met.

Documentation is required to show the effective design, implementation,  and operation of an internal control system. The level and nature of  documentation can vary based on the size of the agency and the  complexity of its processes. Management exercises judgment in  determining the extent of documentation that is needed.

TE/GE has not fully implemented or documented internal controls for  analyzing data for examination selection, meaning it cannot be assured  that its selection decisions will produce the desired outcomes. The  internal controls range from two controls that TE/GE adequately  documented and implemented to seven others where TE/GE did not. The  seven include five controls presented as sequential steps in using data for  making selection decisions as well as two controls addressing timely  documentation of Internal Revenue Manual (IRM) sections and risk  management.

TE/GE Has Implemented Two Controls for Building a Positive System in Selecting Returns for Examination

The first internal control TE/GE implemented involved assessing staff  competence. To ensure competence in using data to make decisions,  TE/GE officials contracted with data specialists for modeling expertise to  incorporate statistical and machine learning into examination selection.  Bringing in this modeling expertise was an important step because  exempt organization examinations staff, rather than statisticians or data  analysts, initially developed the examination selection models, according  to TE/GE officials. TE/GE also provided documents on training and basic  duties for staff when analyzing data.

What are Internal Controls and Why Do  They Matter?  One way federal agencies can improve  accountability in achieving their missions is to  implement an effective internal control  system. Effective internal control comprises  the plans, methods, policies, and procedures  used to fulfill objectives on an ongoing basis.  It serves as the first line of defense in  safeguarding assets and increases the  likelihood that an agency will achieve its  objectives while adapting to changing  environments, demands, risks, and priorities.  Effective internal control provides reasonable,  not absolute, assurance that an organization  will meet its objectives.

The second internal control involved communicating inside and outside of  TE/GE. Internally, TE/GE staff could provide feedback through an online  compliance issue submission portal in fiscal year 2018. Submissions may  become compliance strategies or model queries. As for external  communication, TE/GE collaborated on data-related issues in an IRS- wide group and with statistical specialists in the RAAS division. For  example, RAAS identified potential data sources for compliance issues  and drew samples for certain compliance strategies to test rates of  noncompliance. In addition, to show how it communicates essential  information with staff and outside parties, TE/GE provided examples on  disseminating guidance and examination accomplishments, including  examination starts and closures.

TE/GE Did Not Fully Implement and Document Controls over Processes and Data Used to Select Returns for Examination

TE/GE did not fully implement and document internal control over the  processes and data used to select returns for examination. These  processes cover five key steps for using data to decide which returns to  select for examination (see figure 4).

Effective internal controls would enable TE/GE to show how feedback  and lessons learned in Step 5 can help it better determine how to create  and use quality information (Step 3) and what decisions to make (Step 4)  when pursuing the established objectives (Step 1). However, TE/GE has  not defined measurable objectives or undertaken regular evaluations to  assess progress toward objectives. Although TE/GE was able to describe  its approach for accessing relevant and reliable data, processing those  data into quality information and using the data to make decisions, it was  not able to fully document how its control processes worked, as  discussed below.

TE/GE Has Not Defined Measurable Objectives for Selecting Returns for Examination

Since its 2017 reorganization, TE/GE has not established measurable  objectives to select exempt organization returns for examination (see  figure 5).

Specifically, TE/GE has not produced formal objectives that are aligned  with its mission and the IRS strategic plan, are expressed in quantitative  terms, and are related to examination selection and program outcomes.  TE/GE documents, including Program Letters and Business Performance  Reviews, refer to outcomes that could constitute objectives—such as  improving the models and advancing data analytics to drive decisions  about identifying and addressing existing and emerging areas of  noncompliance—but they do not identify them as such.

TE/GE officials acknowledged the need to establish measurable  objectives. They said their efforts are evolving and they need to improve  analytical abilities to help assess the capacity for meeting objectives. For  example, one official said they are working to establish objectives at the  onset of a compliance strategy. Without measurable and defined  objectives, TE/GE cannot effectively analyze how well it selects returns  for examination and lacks a clear vision of what it is trying to achieve. A  lack of measurable objectives also hinders implementing other internal  controls, such as evaluating performance or assessing risk, as discussed  later.

TE/GE Could Not Demonstrate that It Has Controls in Place to Catch Certain Form 990 Errors but Electronic Filing Will Likely Increase Data Reliability

The IRM has procedures for processing Form 990 data, which include  controls over acceptance and transmission of the data (see figure 6).

TE/GE provided data that showed error rates for electronically filed  returns filed in 2019 were between 1 and 4 percent. However, taxpayer or  transcription error rates for paper returns filed in 2019 were between 19  and 32 percent of filed returns, depending on the version of the Form 990.  TE/GE was not able to show that it regularly reviews and remediates such  errors to ensure the reliability of Forms 990 data. However, under the  Taxpayer First Act of 2019, electronic filing of all Forms 990 will be  required for tax years starting July 2, 2021. This change should remediate  the known errors from paper-filed returns and increase data reliability.

Processing Queries for the Model Did Not Always Produce Quality Information

We found several issues with TE/GE’s processing of queries in the Form  990, 990-EZ and 990-PF models that affect the validity or reliability of the  scores that the models generate to rank returns for examination selection  (see figure 7).

As a result, TE/GE cannot ensure that the model scores properly rank the  returns for examination selection. Specifically, TE/GE does not  consistently assign point values for the queries used to generate the  model scores and inform selection decisions. We also found errors in  TE/GE’s documentation of the queries, which lead to redundant queries,  and inflated model scores. Finally, TE/GE has no control procedures to  ensure consistent testing of proposed queries.

Inconsistent Point Values for Queries Raise Concerns about Model  Scores  We estimate that for 24 percent of queries (83 queries) from the models,  TE/GE staff did not assign point values for queries consistent with its  definitions for the four categories (see table 4). Not implementing the  defined point values puts the model scores at risk of inconsistent scoring  and examination selection.

We found three types of queries involved with the inconsistent  assignment of point values.  1.  Miscategorized queries were not assigned to the category that  matches TE/GE’s definition. These occurred because TE/GE has not  documented specific rules for query categorization. As a result, we  found an estimated 7.4 percent of queries (26 queries) where TE/GE  staff overrode the category definitions when assigning points without  documenting the reasons. Absent the reasons, TE/GE cannot  ensure consistent treatment of similar queries. In our sample, these  override decisions included assigning:

Three queries to the Speculative category, which is worth five  points, when the definitions supported the Automatic category,  which is worth 10 points. TE/GE officials said they did this to offset  potentially confusing language in the return lines or instructions.

One query to the Automatic category rather than the Speculative  category supported by the definitions. TE/GE officials said they  used the higher point value category to increase the chance of  selection so that certain Form 990-PF attachments, which the  queries do not cover, would be more likely to be considered for  examination.  2.  Queries could fit into more than one category based on TE/GE’s  definitions. We estimate that 16 percent (55) of the queries could fit in  more than one category. Of these, 18 in our sample could have  been placed in the Missing Schedule/Form category. In addition, we  found one query in our sample that TE/GE labeled as having a  duplicate but one query was assigned to the Automatic category worth  10 points and the other was assigned to the Inconsistencies category  worth one point. TE/GE officials acknowledged that some queries  could fit in more than one category. When we asked why certain  queries for missing schedules and forms were not categorized as  such, these officials described a hierarchy of missing forms based on  being subject to penalties and interest, such as employment tax  returns, and their associated categories. They did not document or  consistently implement this hierarchy as queries identifying the same  missing form sometimes were in different categories.  3.  Sliding scale queries whose point values differ from those stated in  TE/GE’s model documentation. We found nine queries with sliding  scale point values that involved Form 1099 information returns. The  sliding scales reduce point values based on the severity of the  compliance issue, such as reducing the query point values if the  organization filed a low number of information returns. TE/GE did not  provide documentation about the rationale and associated definitions  for these queries. Without documentation on the different treatment of  these queries, TE/GE is not transparent about the rationale for  assigning points through a sliding scale to support its model scoring.

TE/GE officials said they have not updated definitions and criteria for  using the categories and sliding scales because of a decision to keep the  model operating as is and to update documentation as time permits. After  our preliminary analyses, TE/GE provided updated definitions for the four  categories, and descriptions of the sliding scales that were used for  queries. However, these definitions and descriptions do not include any  decision rules or criteria that document how to apply them. Further, the  sliding scale descriptions do not offer definitions for words like “low,”  when referring to the volume of information returns filed.

Definitions that are incomplete and not always followed when assigning  point values raise concerns about consistency and transparency in  scoring returns for examination selection. TE/GE’s assignments affect  scores and whether a return is placed on the MSS for examination  consideration. Inconsistent or invalid assignment of point values may  distort the potential for examination. For example, of the nine  miscategorized queries we analyzed in our sample, we determined that if  their categorizations were corrected, hits on three of the queries would  make a return eligible for the MSS and hits on two others may make a  return eligible, depending on the other queries the return hit. Changes to  two queries would have made returns no longer eligible for the MSS.

Query Documentation Has Errors That Forestall Valid Analysis of  Queries  We estimate that about 27 percent (96 queries) of the queries in the  models had errors in the documented descriptions. Query descriptions  detail the logic and data used from specific forms and line numbers that  the queries scan. The errors we found include:  references to older versions of the forms as well as omissions of form  lines used in the query; and query descriptions that did not match programming code.

To address these differences, TE/GE proposed corrections to the query  descriptions. A TE/GE official said re-visiting the query documentation is  part of the contractor’s 2020 study and that TE/GE does not have a  timeline for correcting the documentation.

In addition to errors, the descriptions also use inconsistent language,  which prevents easy identification of queries by issue. For example, to  identify all queries related to excess benefit transactions, one must  manually search different fields for terms such as “excess benefit,”  “excessive benefit,” and “EBT” (excess benefit transaction).  Furthermore, TE/GE’s database fields only capture one issue per query.  Since many queries involve multiple issues, these fields cannot be used  to fully inventory the queries.

These errors and inconsistencies in the query descriptions occurred  because TE/GE has no procedures for regular reviews of queries as  forms or laws change. TE/GE Compliance Governance Board  (Governance Board) members review query descriptions prior to  implementation but do not review details of the queries in the context of  the entire model. Further, TE/GE procedures only require review of  programming code before queries are sent to the Governance Board.  Review of the code once it is integrated into the model program is  optional, according to TE/GE procedures.

The errors and inconsistent descriptions prevent TE/GE from having a  comprehensive and accurate inventory of queries within and across  models. Without regular reviews, TE/GE cannot be assured that its  programming code is correct and that any analyses of the performance of  queries or the models as a whole are valid. When we asked about the  lack of regular reviews of queries, TE/GE officials said they plan to  implement reviews but did not provide us with a plan or timeframes for  doing so.

Another effect of not having a comprehensive and accurate inventory is  that TE/GE cannot analyze query performance and identify queries that  look for the same compliance issue to prevent redundancies and to  ensure valid and consistent scoring. As a result, we found queries that  address the same or similar issues with the same criteria, inflating scores  for returns and making selection for examination more likely. Our analysis  of the July 2019 Form 990 model run showed 90 pairs of queries,  involving 78 unique queries that hit together at least 90 percent of the  time. By having two queries that rely on the same criteria, returns  accumulate extra points for the same behavior. For example, all 910  returns that hit an employment tax query also hit a query that shares  some of the same criteria and thresholds. As a result, these returns  accumulated 10 points rather than five points, making them eligible for the  MSS. Aside from our sample, we found queries seeking certain  organizations with political campaign activities and political expenditures  that would total 15 points in the Form 990 model. Queries identifying  these same activities and expenditures would total 30 points in the Form  990-EZ model.

TEGE’s contractor recommended in 2018 that TE/GE eliminate  “redundant” queries, which is similar to our finding. TE/GE officials said  they do not believe the redundant queries are duplicates and they are  awaiting the results of the contractor’s study in 2020 before making  changes. Until TE/GE resolves the extent to which it has redundant  queries, it cannot do a valid analysis of whether its queries identify the  most noncompliant returns.

TE/GE Lacks Procedures and Criteria for Testing Proposed Model  Queries  TE/GE has no procedures requiring the testing of proposed model  queries. Even so, based on our sample of the new queries in the fiscal  year 2018 Form 990 model, TE/GE would be able to provide evidence of  tests for an estimated 94 percent of all new queries. However, TE/GE  also does not have procedures for how to conduct testing or what data to  use. The testing that has been performed consisted of running the query  on certain tax years of returns to count the number of returns flagged,  according to a TE/GE official. TE/GE does not run the queries on data  from closed examinations to see whether the queries would identify  known compliance issues that justify an examination. Interactions with  existing queries are not tested. When considering new queries,  Governance Board members see the number of returns flagged by each  query during testing, but have no criteria to determine whether a query  flags an appropriate number of returns. A TE/GE official said TE/GE does  not believe it needs to document procedures for testing.

In the absence of procedures and standards, TE/GE cannot ensure that  testing of new queries is done consistently with appropriate data sources  and research standards. By only testing the number of returns that a  query flags, TE/GE cannot validate that proposed queries can effectively  identify the noncompliance that would be worth examining. Using tested,  validated and documented data is a critical step in ensuring that research  is proper, reliable, and accomplished in accordance with expectations,  according to the IRM. Without testing queries on reliable data, and  making adjustments based on criteria, TE/GE risks implementing queries  that do not produce reasonable numbers of hits that are worth pursing  through examinations.

TE/GE Did Not Fully Document Its Processes for Transforming Data into Quality Information for Other Examination Sources

For examination sources that used data other than the models, we found  that TE/GE did not always document its processing of data into quality  information. We identified common “start-to-finish” segments to this  processing of data, including:  submitting a proposal and supporting data to find noncompliance;  reviewing the potential data sources and queries or thresholds to be  used as examination selection criteria; and  recommending the proposed effort for approval through the  appropriate executives.

On one hand, TE/GE provided documentation of the required approvals  for these segments in processing data for five compliance strategies.  These strategies included examining loans by private foundations and  collecting information on organizations that exceed investment income  limitations.

On the other hand, TE/GE did not provide similar start-to-finish  documentation on processing quality information from other examination  sources that use data outside of the models; examples include research  projects under the Data Driven Approaches portfolio and projects that use  queries under the Referrals and Other Casework portfolio (see table 1).  Over several discussions, TE/GE did not explain why it did not fully  document such projects. By not fully documenting how it processes data  into quality information, and by not linking such processes to measurable  objectives, TE/GE cannot ensure that it is analyzing quality information in  selecting examinations.

TE/GE Did Not Consistently Document Use of Quality Information to Make Decisions

For the compliance strategies, TE/GE showed evidence of using the  quality information to decide which returns to select for examination, such  as for Governance Board decisions. However, TE/GE did not provide  documentation on how it made selection decisions using data for other  projects that use queries (see figure 8).

In addition, TE/GE did not use quality information to decide how  frequently to run the model. TE/GE decided to run the Form 990 models  twice per year without analyzing the effects. Moreover, we found that the  time between runs is inconsistent. Since the Form 990 model’s first run,  the time between runs ranged from 84 days to 251 days. Since returns  are ranked on the MSS, eliminations result in the classification staff  selecting lower scoring returns. The average score for examined returns  was 27.1 for the list that was used for 84 days compared to 23.2 for the  list used for 251 days.

To the extent that TE/GE ensures that its model scores are as reliable  and valid as possible, analyzing data could help TE/GE identify the  frequency of model runs that maximizes the use of model scores to guide  decisions on examination selection. For example, analyzing Form 990  filing patterns could help identify the optimal timing of model runs,  allowing for adequate time remaining under the statute of limitations.

Lack of Regular Evaluations and Inconsistent Data Prevent TE/GE from Fully Evaluating Its Selection Methods

TE/GE does not regularly evaluate its models and other selection  processes that use data. In particular, model scores for all returns are not  retained or are inconsistent from year to year which limits the ability to  conduct evaluations. Furthermore, TE/GE does not evaluate reasons why  some selected returns are not examined, which could help improve  selection methods (figure 9).

TE/GE Has No System for Regularly Evaluating Examination  Selection Decisions  TE/GE has not regularly evaluated its examination selection decisions  that rely on data to improve its selection methods. While TE/GE  commissioned the contractor evaluations of its Form 990 model, it has no  documented process for continued evaluations of the model or any  evaluations of other sources, such as research projects, that rely on data  to select returns for examination. For its compliance strategies, not  enough examinations have closed under the strategies to warrant  evaluations yet, according to a TE/GE official.

Data limitations have challenged evaluation efforts, according to a TE/GE  official. To address this, TE/GE started capturing more detailed data on  examination outcomes; however no evaluations of outcomes have  resulted. The officials noted that they have been spending more time  reporting and monitoring compared to analyzing and evaluating, which  they said needs to occur more often. Without evaluation, TE/GE cannot  ensure that its use of data to select returns is working as intended.

In addition to not evaluating selection decisions and their outcomes,  TE/GE has also not addressed the Form 990 model deficiencies the  contractor previously identified. In its 2018 reports, the contractor made  17 recommendations (see appendix V for the status of each  recommendation). A TE/GE official said it had not acted on many  recommendations because all examination selection strategies are being  evaluated with the transition to the Compliance Planning and  Classification (CP&C) office. TE/GE initiated another study in 2019 with  the same contractor to address its 2018 recommendations among other  tasks. As of March 2020, TE/GE implemented one recommendation and  part of another, deferred action on nine recommendations until after the  contractor finishes the new study, deferred action on three due to other  reasons, and did not clearly provide a status for two. In addition, TE/GE  will likely not implement the other recommendation.

According to contract documentation, the study will explore architectures  and alternative designs to the model, propose up to three compliance  actions other than examinations, and recommend measures to monitor  the actions’ effectiveness. TE/GE expects a final report by September  2020. To the extent that TE/GE has not implemented the contractor’s  recommendations from 2018, the related deficiencies identified in the  Form 990 model will have persisted for more than 2 years by the time the  contractor issues its 2020 report. Unless TE/GE documents its  consideration and action of the recommendations, the value of the  contractor’s work is diminished and possible improvements may be  overlooked.

TE/GE Has Not Retained Complete Data to Allow for Full Evaluation  of Its Models  Until recently, TE/GE did not retain model scores for each return and  query performance data that would be useful for evaluation. The January  2018 contractor report recommended that TE/GE save model data. For its  July 2018 report, the contractor had to recreate historical scoring data for  its evaluation. TE/GE officials said they increased storage space and  saved the fiscal year 2018 data. When we asked for these data, TE/GE  officials said that each time they run a model, they overwrite the old data.  The officials said they did not have space on their server to save all of the  data. Instead, TE/GE had been saving the MSS’s for each run. However,  the MSSs have only limited value for evaluating the model and queries.  Specifically, the MSS for each model run contains score information for  only about 20,000 returns (out of about 300,000 scored) that have a  certain minimum score and hit queries in certain categories. Further, the  MSS does not contain data on model queries that are flagged.

In September 2019, TE/GE officials said the Research, Applied Analytics  and Statistics division provided temporary server storage space to save  model data through September 2020 while the contractor assesses  TE/GE’s models. Starting with the July 2019 model run, TE/GE is saving  score and query performance data for all filed returns. In January 2020,  TE/GE officials told us they developed a way to save data on query hits  for all returns run through the model. However, TE/GE has not provided  documentation to show exactly what data will be saved over the long term  for all filed returns run through the model. Without complete historical  data on model scores and query hits, TE/GE cannot assess the full  performance of its models. Such data would facilitate an analysis of the  queries, and whether they identified returns with changes or related pick- up returns.

Historical Data on Examination Outcomes Lack Consistency, Which  Complicates Evaluation  TE/GE does not analyze consistent multi-year data on examination  outcomes, which would facilitate evaluation of its use of data in selecting  returns for examination. TE/GE officials said they use historical data— such as change rates—to determine the success of an examination  source. TE/GE provided historical data on examination starts, closures,  and pick-up returns covering 2 years but did not provide data beyond that  and change rates were not always included. Further, TE/GE has used  different methods to organize and report examination outcomes over the  years. These differences in reporting outcomes affect TE/GE’s data in  the following ways:

Starting with fiscal year 2018, data on exempt organizations  examinations include federal, state and local employment tax  examinations. Prior to 2018, TE/GE reported these employment tax  examination data separately.

After TE/GEs reorganization in 2017, it grouped examinations into  portfolios and changed the portfolio definitions during 2018.

As of March 2020, TE/GE has not produced a consistent method of  summarizing of historical data. TE/GE officials acknowledged data  limitations, and said they are working to implement recommendations  from a 2019 study to improve capturing examination data. TE/GE officials  said the staff member analyzing data has been doing so for many years,  allowing them to reconcile the data. However, this poses a risk that other  IRS or oversight entities cannot reconcile the data. According to internal  control standards, agencies should establish effective methods for  retaining organizational knowledge and mitigate the risk of having that  knowledge limited to a few personnel, as well as contingency plans to  respond to sudden personnel changes.

TE/GE’s inconsistent data limit its ability to conduct evaluations. These  inconsistent data also prevent TE/GE from establishing baselines or  targets for examination outcomes such as change rates to help measure  the success of its selection methods.

TE/GE Did Not Evaluate Reasons for Not Examining Some Returns Selected for Examination

In fiscal year 2019, TE/GE did not examine about 20 percent of the  exempt organization returns that had been selected for examination.  Although this rate of non-examined returns has improved in recent years,  TE/GE has not analyzed data to explore why the rate has improved and  how to reduce it further.

Our analysis showed that almost 30 percent of these returns were not  examined because they were too close to the statute of limitations date.  TE/GE officials did not have a reason why the returns were sent to the  field for examination if the statute date was so close. TE/GE officials said  they do not regularly analyze reasons for non-examined returns. They  said they have analyzed only the number of non-examined returns by  manager and area. In addition, TE/GE officials said they implemented  new guidance in fiscal year 2019 for staff who make decisions to not  examine returns, which is intended to improve the information they have  on these decisions. As of fiscal year 2019, TE/GE began tracking certain  non-examined returns by project code but has not committed to analyzing  the data.

Non-examined returns are not an efficient use of resources, as the time  spent reviewing and rejecting these returns—even if minimal—reduces  the time staff have for conducting examinations. Routinely analyzing  reasons for non-examined returns, as well as related data, could help  TE/GE identify actions to reduce the number of returns that are sent to  the field but are then declined for examination by a manager or examiner.

Updating Examination Selection Procedures and Identifying Risks Could Help TE/GE Use Data in Decision Making

TE/GE Has Not Annually Reviewed and Updated Procedures in Certain Internal Revenue Manual Sections or Issued Related Interim Guidance on Examination Selection

TE/GE did not annually update procedures on examination selection and  databases in certain IRM sections since the May 2017 reorganization.  The Internal Revenue Manual (IRM) states that procedures in IRM  sections must be annually reviewed and updated as needed. TE/GE  released updated IRM sections for two of the three groups in CP&C. It  released a section on the Issue Identification and Research in September  2018, and one on the Classification and Case Assignment procedures in  September 2019. However, these sections do not cover the steps the  model classifier takes when reviewing returns from the MSS. As of  December 2019, no IRM section has been released on the Planning and  Monitoring group. As such, TE/GE staff does not always have official  information on roles and responsibilities for new entities and processes  created since May 2017.

For certain updated or new IRM sections, TE/GE did not release interim  guidance while those sections awaited approval. IRS requires issuance of  interim guidance to address deviations from the IRM, even if temporary.  Instead of developing interim guidance, TE/GE officials stated that, in the  wake of the reorganization, they decided to use desk guides, such as for  the IRM section on classification and case assignment processes.  However, TE/GE did not update its desk guides on processes until more  than 2 years after the reorganization. Furthermore, the desk guides do  not cover the specific duties of the model classifier, or the steps for  classification of returns identified for compliance strategies.

IRM guidance states that management must develop and maintain  documentation on data systems; collection and analysis; and  responsibilities for data collection, input and analysis. Timely  documentation of new procedures and responsibilities improves the  accuracy and reliability of IRM content. According to the IRM, when the  IRM and related guidance are not current, TE/GE increases the risk that  staff follow incorrect procedures, use guidance that is not transparent to  the public, administer tax laws inconsistently, and misinform taxpayers.

TE/GE Has Not Identified Risks from Using Data for Examination Selection

Good federal government practice requires risk management, without  which, TE/GE could undercut its use of data to enhance decisions on  examination selection. Although the use of data in examination selection  has the potential to improve efficiencies in classifying and examining  returns to identify noncompliance, any new endeavor carries risks.

TE/GE did not identify any TE/GE-specific risks that could undercut its  success in using data to select exempt organization returns for  examination. As of December 2019, the TE/GE risk register identified 12  risks, ranging from aging technology and infrastructure to employee  engagement and morale. One risk— data access and analytics—involved  using data in general decision making at the IRS level rather than TE/GE  decisions about examination selection or its related models. TE/GE  officials said they are analyzing and responding to this risk under the IRS- wide risk management process.

TE/GE did not document why it did not identify any TE/GE-specific risks  in using data for examination selection. Our report discusses a number of  deficiencies that could be potential risks to TE/GE using data in selecting  returns for examination. For example, TE/GE lacks program objectives  that would be necessary to identify and assess risks. We also found  weaknesses in how TE/GE processes and analyzes data to inform  examination selection and how it evaluates selection decisions. Further,  the IRM states that TE/GE’s Compliance Governance Board (Governance  Board) should consider risks in its decisions and we saw that risks were  considered in documents proposing examination selection criteria to the  Governance Board. We did not find evidence that TE/GE’s risk  management process recorded these risks for analysis and any response  if needed.

After we shared our concerns about the lack of identified risks, TE/GE  officials noted that TE/GE participates in mitigation steps as identified by  the IRS Risk Office. TE/GE officials also mentioned CP&C representation  in an IRS pilot program designed to explore ways to better select  employment tax cases. While such actions could be a component of a  risk management strategy, it is incomplete and it is unclear how this  initiative would help TE/GE identify, analyze, and mitigate risk.

Not identifying and managing risks identified in this report leaves TE/GE  open to errors and examination selection decisions that are potentially not  transparent or not fair. As such, without objectives and a consistent and  documented process for identifying and managing risks, TE/GE cannot  effectively address risks that may hamper its efforts to use data to  enhance its compliance work.

Increasingly constrained resources underscore the importance of  TE/GE’s efforts to efficiently identify and examine exempt organization  returns that have the highest noncompliance potential. TE/GE has  developed ways to use data to aid in examination selection. However,  opportunities exist to strengthen internal controls to help ensure that data  used are reliable, decision rules are clear and documented, and  objectives are identified and being achieved.

TE/GE should take several steps to improve the reliability and validity of  the models. These steps include improving documentation of decision  rules and criteria for scoring; regularly reviewing model documentation  and programing; testing new queries and their interaction with existing  queries; retaining model and query data; and periodically evaluating the  performance of selection methods.

In absence of regular evaluation of its examination selection decisions,  TE/GE misses opportunities for improving its selection processes.  Deficiencies that TE/GE’s contractor already identified provide an opening  for improving its models. Without consistent historical data, TE/GE will be  limited in assessing progress and making improvements. A review of the  reasons why certain returns selected for examination are not examined is  an example of an evaluation that could help inform process  improvements.

Ensuring that all procedures are current and accurate would reduce the  potential for employees following incorrect procedures and administering  tax laws inconsistently. TE/GE’s lack of identified risks from using data in  examination selection precludes TE/GE from analyzing and responding to  those risks. By taking actions to further strengthen these internal controls,  TE/GE could enhance its efforts to identify and examine the most  noncompliant exempt organizations and enhance IRS’s oversight of tax  exempt organizations and help maintain the integrity of the charitable  sector and the larger exempt community.

We are making the following 13 recommendations to IRS:  The Commissioner of Internal Revenue should document measurable  objectives for using data in selecting exempt organization returns for  examination. (Recommendation 1)

The Commissioner of Internal Revenue should document and  consistently use clear criteria and decision rules on assigning point values  to queries, using categories and sliding scales. (Recommendation 2)

The Commissioner of Internal Revenue should require a regular review of  query descriptions and programming to ensure their accuracy and  minimize queries that flag the same or similar compliance issue.  (Recommendation 3)

The Commissioner of Internal Revenue should develop procedures and  criteria to test new queries prior to implementation in the models.  (Recommendation 4)

The Commissioner of Internal Revenue should more fully document how  TE/GE processes data and uses data to make examination selection  decisions for sources outside of the model such as research projects and  other projects that use queries. (Recommendation 5)

The Commissioner of Internal Revenue should conduct an analysis to  identify the optimal interval between model runs. (Recommendation 6)

The Commissioner of Internal Revenue should establish a process for  regularly evaluating selection decisions and related outcomes for the  models and other processes that use data to select returns for  examinations. (Recommendation 7)

The Commissioner of Internal Revenue should document consideration or  action on recommendations from its 2018 and 2020 contractor  assessments. (Recommendation 8)

The Commissioner of Internal Revenue should document how score and  query data for all returns in the models will continue to be saved over the  long term. (Recommendation 9)

The Commissioner of Internal Revenue should ensure that historical data  on examination outcomes are consistently defined and used when doing  analysis of examination outcomes. (Recommendation 10)

The Commissioner of Internal Revenue should routinely analyze the  reasons for not examining selected returns and identify any necessary  actions to address the reasons. (Recommendation 11)

The Commissioner of Internal Revenue should annually review and  update procedures as needed in relevant IRM sections on examination  selection and issue interim guidance until the affected IRM sections are  updated. (Recommendation 12)

The Commissioner of Internal Revenue should document why TE/GE has  not identified any risks in its risk register for using data to select exempt  organization returns for examination. If risks are subsequently identified,  TE/GE should document how it plans to analyze and address them.  (Recommendation 13)

We provided a draft of this report to IRS for review and comment. IRS  provided written comments, which are reproduced in appendix VI and  summarized below. Of our 13 recommendations, IRS agreed with 12 and  disagreed with one. IRS also provided technical comments, which we  incorporated as appropriate.

IRS disagreed with our recommendation on ensuring that historical data  on examination outcomes are consistently defined (Recommendation 10),  pointing out that its raw data are consistently defined in its information  systems.  Our concern, however, is with how the outcome data are  reported and analyzed, which inhibits understanding of outcome trends  over time.  In response to IRS comments, we added language to the final  recommendation to more clearly focus on the consistency of the outcome  data used and analyzed over the years.

In addition, although IRS agreed with our recommendation to more fully  document how TE/GE processes and uses data to make examination  selection decisions outside of the model (Recommendation 5), IRS said  that it would provide documentation on a project (other than compliance  strategies) that is approved by the Governance Board. While we look  forward to such documentation, we are primarily interested in IRS  documenting a system for how it processes and uses data to select  returns for examinations for projects outside of the model, regardless of  Governance Board approval.  As discussed in the report, IRS has such a  system for projects in its compliance strategies portfolio, which could  provide a framework to follow.

Similarly, IRS agreed to analyze return due dates of the filing populations  commonly associated with the examinations (Recommendation 6).  We  will be interested to see how that analysis helps IRS to determine the  optimal interval between model runs, which is the focus on our  recommendation.

We are sending copies to the appropriate congressional committees, the  Secretary of the Treasury, the Commissioner of Internal Revenue, and  other interested parties. In addition, this report is available at no charge  on the GAO website at http://www.gao.gov.

If you or your staff have any questions about this report, please contact  me at (202) 512-9110 or mctiguej@gao.gov. Contact points for our  Offices of Congressional Relations and Public Affairs may be found on  the last page of this report. GAO staff who made key contributions to this  report are listed in appendix VII.

Appendix I: Objectives, Scope, and Methodology

This report assesses (1) the use of data to select tax-exempt organization  returns for examination; and (2) the process the Tax Exempt and  Government Entities (TE/GE) division has established to select returns for  examination.

To assess the use of data to select tax-exempt organization returns for  examination, we reviewed data from the Internal Revenue Service’s (IRS)  Returns Inventory and Classification System (RICS) for fiscal years 2016  to 2019. Table 5 defines the variables and measures we analyzed.

We analyzed aggregated data at the project code level, and we grouped  project codes by examination source (for example, examinations from  referrals occurred under several project codes). Based on our testing of  the data and review of documentation and interviews, we determined that  the data were reliable for purposes of assessing TE/GE’s selection  processes.

We analyzed outcomes from the Form 990, Return of Organization  Exempt from Income Tax model. We used RICS data and Model Score  Sheets (MSS), for examinations closed from October 1, 2015 through  September 30, 2019. Each model run generates an MSS, which is a  ranked list of Form 990s that hit certain types of queries and have a  minimum score. We matched Form 990 scores from the MSS with  selection information and examination outcomes in RICS for  examinations closed under all project codes, though the data presented in  objective one is specific to examinations started under the Form 990  project code. We used source codes—which indicate whether an  examination was a pick-up, substitute for return or primary return—to  analyze what types of examinations produced the highest change rates  under the Form 990 model project code. To inform this work, we reviewed  recent TE/GE contractor assessments of exempt organization  examination selection and the Form 990 model.

To assess the process that TE/GE has established to select returns for  examination, we reviewed internal controls steps in Standards for Internal  Control in the Federal Government (Green Book). Given TE/GE’s  emphasis on using data in examination selection, we identified five  internal control steps related to analyzing data to select returns for  examination to address our objectives. We selected four other internal  controls because they constitute practices common to all five steps in the  selection process. These are presented in table 6.

Define objectives in measurable terms so performance in achieving objectives can be  assessed. (Green Book (GB) 6.04)  Obtain relevant data from reliable internal and external sources in a timely manner based  on identified information requirements. (GB 13.04)  Process the obtained data into quality information that supports the internal control system  (i.e., using data in decision making); use quality information to achieve the entity’s  objectives; and document policies on the responsibilities for data collection, input, and  analysis. (GB 13.05, 13.01, and 12.02)   Use the quality information to make informed decisions in achieving key objectives. (GB  13.05)  Evaluate performance (outcomes) for key objectives and take actions to remediate  deficiencies. (GB 13.05, 16.03, and 17.06)

Develops, maintains, and updates in a timely fashion documentation on the  responsibilities for data collection, input and analysis for using data in decision making.  (GB 12.02 and 12.05 and IRM)  Defines risk tolerances in specific and measurable terms, considers internal and external  factors to identify risks, analyzes risks to estimate significance, and designs specific  actions for response. (GB 6.09, 7.04, 7.05, and 7.09)  Ensures that personnel possess competence to meet responsibilities as well as  understand the importance of effective data analysis in decision making. (GB 4.04)  Communicates necessary information to enable personnel to perform key roles for  analyzing data in decision making and with external parties. (GB 14.03 and 15.20)

To identify criteria specific to IRS, we reviewed the Internal Revenue  Manual (IRM), which provides standards and guidance similar to the  criteria we identified. We shared the Green Book and IRM criteria with  TE/GE, as well as our expectations of the documentation that would show  adherence to these criteria.

Our assessment focused on examination sources developed after the  2017 reorganization and sources that rely on data for selection (such as  models and projects that use queries). Examination sources that did not  rely on data, such as claims, were not assessed. We reviewed the  referrals classification process to consider how data might be used to  enhance it. We analyzed TE/GE documents such as Program Letters,  Business Performance Reviews, desk guides, memorandums, work  plans, performance data, contractor reports and training documents. In  addition, we assessed documents—such as meeting minutes and  research results—showing the development and approval of data queries  and projects used in examination selection. We reviewed the MSSs for  the Form 990 model, and procedures for the Form 990 model, the Form  990-EZ, Short Form Return of Organization Exempt from Income Tax,  and Form 990-PF, Return of Private Foundation, models.

We selected a generalizable stratified random sample of 114 of the 354  unique queries in the three models (see table 7).

Because we followed a probability procedure based on random  selections, our sample is only one of a large number of samples that we  might have drawn. Since each sample could have provided different  estimates, we express our confidence in the precision of our particular  sample’s results as a 95 percent confidence interval (e.g., the margin of  error is +/- 10 percentage points). This is the interval that would contain  the actual population value for 95 percent of the samples we could have  drawn. Our sample is designed to control the margin of error of attribute  estimates within the overall scope query sample as well as the combined  Form 990 query sample (a combination of strata 2 and 4 plus certainty  selections). The sample was designed as follows.

There is one certainty stratum for Form 990-PF queries where we  selected a 100 percent sample (i.e. a census), and this stratum does  not have a margin of error. We selected these queries with certainty  because of the smaller population size in this stratum.

For remaining strata, we selected the necessary sample size to  achieve an overall 95 percent confidence interval for attribute  (percentage) estimates with a margin of error of about +/-10  percentage points under proportionate allocation. In addition, the  sample size was increased in strata 2 and 4 (combining Form 990  model queries) to achieve the necessary sample size for a 95 percent  confidence interval with a margin of error of about +/-10 percentage  points within this group.

For the sampled queries, we compared their category and descriptions as  provided in the model documentation, with TE/GE’s definitions of the  categories to assess whether the query was categorized appropriately.  We also compared the query descriptions with the forms to assess  whether the referenced lines were relevant to the query. Additionally, we  reviewed the model programming code to check for errors and  consistency with the query descriptions. For query categorizations that  did not match TE/GE’s definitions or queries that appeared to have errors  in the descriptions or programming, we asked TE/GE to review and  explain its decisions. To identify potentially redundant queries, we  analyzed output from the July 2019 Form 990 model run, the only one  available at the time of our analysis.

Within our sample, we reviewed 36 of the 104 newly added queries in  the fiscal year 2018 model. Specifically, we reviewed approval  documentation and meeting minutes to test whether two levels of  management and the Compliance Governance Board approved new  queries, consistent with TE/GE procedures. We also reviewed evidence  that TE/GE tested each query prior to its approval for inclusion in the  models.

We held two telephone focus groups with the nine classifiers who review  exempt organizations referrals. We asked questions about the data and  resources they use to classify referrals, how they convey their results,  and how they are provided feedback. We interviewed officials from the  Compliance Planning and Classification office and IRS’s Research,  Applied Analytics and Statistics division who worked on several  compliance research initiatives. We met regularly with TE/GE to share  ongoing assessments.

We conducted this performance audit from November 2018 to June 2020  in accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.

Appendix II: Form 990, Return of Organization Exempt from Income Tax

The figure below shows the text of the 2019 version of Form 990, Return  of Organization Exempt from Income Tax. A list of schedules for the Form  990 is provided in table 8 following the form.

The remaining pages of the Form 990 are available at IRS’s website,  accessed March 23, 2020: https://www.irs.gov/pub/irs-pdf/f990.pdf.

Appendix III: Changes to Exempt Organization Form 990, Tax Years 2009- 2019

Most exempt organizations are required to file an annual form to report  their activities, structure, revenue and expenses, and other items. The  organization’s classification under the Internal Revenue Code and its  gross receipts and total assets, determines which form must be filed.  Most organizations file one of the following:

Form 990, Return of Organization Exempt from Income Tax;

Form 990-EZ, Short Form, Return of Organization Exempt from

Form 990-PF, Return of Private Foundation or Section 4947(a)(1)

Form 990 Has Undergone Changes Since 2008 Redesign

The Internal Revenue Service (IRS) last redesigned the Form 990 series  for tax year 2008. The redesign added 14 schedules to the existing two,  and reflected changes in the tax-exempt sector and tax law. Some  changes from the redesign were phased in and implemented for tax year  2008 and 2009 filings. We summarized changes as found in the “What’s  New” section of the form instructions for each of the three Form 990 types  and for each year. We grouped the changes into two categories as  defined by:

New or revised question(s): The addition of new lines, check boxes,  narratives or schedules. This includes changes to accommodate new  laws or reporting requirements, such as new reporting thresholds or  standards.

Instructions and format: New descriptions or details in the  instructions, such as specifying examples or how to provide certain  information to IRS. This also includes changes that affect order of  lines or schedules, but not the content.

For the Form 990 changes since the redesign, IRS made 56 changes to  the form or its instructions for tax years 2009 through 2019 (see table 9  below).

These changes include three to the 2018 form implementing new excise  taxes on net investment income of certain colleges and universities and  on certain tax-exempt organization executive compensation. Aside from  new electronic filing requirements for tax years beginning July 2, 2019,  the 2019 form did not have any changes. In addition to the 56 changes,  IRS made 95 clarifications to existing lines or instructions, or revisions to  definitions from tax years 2009 through 2018. These clarifications provide  more specific definitions or other details.

Further, several of the schedules had additions. For example, the Patient  Protection and Affordable Care Act led to additional reporting on  Schedule H, Hospitals, to fulfill requirements that hospitals report on each  of their facilities and conduct a Community Health Needs Assessment  every 3 years.

Most of the Form 990-EZ’s 27 changes occurred in tax years 2009  through 2012, of which 12 were for 2011 and several of them focused on  compensation reporting. IRS also made 27 clarifications for 2009-2013.  Public Law 115-97 did not affect Form 990-EZ.  There were no changes  to the 2019 form. See table 10.

For the Form 990-PF, IRS made the fewest changes compared to Forms  990 and 990-EZ, with only 11 changes and four clarifications for tax years  2009 through 2019. The Form 990-PF had three changes prompted in  2018 by Public Law 115-97. Electronic filing requirements apply to Form  990-PF for tax years starting July 2, 2019, but there were no other  changes for the 2019 form. See table 11.

Appendix IV: Exempt Organizations Examination Selection Process

Appendix IV describes the general examination selection process for  exempt organization returns, and specific classification steps that apply to  certain returns.

General Selection Process for Exempt Organizations Examinations

The annual work plan is the foundation for identifying and assigning  returns for examination. The Compliance Planning and Classification  (CP&C) office follows various steps to identify returns to fulfill the work  plan, which end in the assignment of returns for potential examinations to  field work groups. The intended process is in figure 11 and discussed  below.

Annual work plan. CP&C’s Planning and Monitoring group develops  the annual work plan. The work plan provides estimates of  examination starts and closures. It also has estimates for the number  of hours to be spent per return examination and the number of days to  complete an examination. Planning and Monitoring develops  estimates at the project code level, which corresponds to a specific  examination source or project such as the Form 990 model. The Tax  Exempt and Government Entities’ (TE/GE) Compliance Governance  Board approves the work plan. TE/GE provides a summary of the  work plan in its annual Program Letter.

Stocking report. The Planning and Monitoring group uses the work  plan to issue “stocking” reports to guide classifiers on types of returns  to identify for potential examination. Planning and Monitoring  considers available examiners, and progress in meeting work plan  numbers. The report lists the number of returns needed by grade,  project code, and classification source.

Classification. Classifiers review stocking plans to identify returns for  potential examination. Classifiers are to eliminate returns for  consideration if the (1) return is approaching its statute of limitation  date, (2) organization has been examined in the last 3 years, or (3)  organization is under a compliance check.

Establishing the return and initial case building. If classifiers  identify examination potential, they establish returns in the Audit  Information Management System and Reporting Compliance Case  Management System (RCCMS). The returns are sent for initial case  building—developing paperwork to initiate the examination— according to a TE/GE official.

Virtual shelf. Established returns and the initial case material are  sent to the virtual shelf, which is an electronic inventory of returns that  may be assigned for examination. Certain referrals, claims,  compliance strategies, and other returns are prioritized, according to a  TE/GE official. Returns remain on the shelf until assigned for  examination or otherwise closed due to statute of limitations,  according to a TE/GE official.

Examination assignment. Functional Assignment Coordinators pull  returns from the virtual shelf to fulfill field group work requests.  Returns on the virtual shelf that matched a work order undergo  additional case building before delivery to field examination groups.

Monitoring. Planning and Monitoring staff regularly review reports  that compare work plan goals with current work, and run algorithms to  forecast upcoming work. These reviews are intended to ensure that  sufficient work is available for assignment, excess work is not created,  and returns approaching statute of limitations are identified. The  monitoring informs new stocking reports.

Specific Classification Steps for Models and Certain Other Examination Sources

Classification steps vary depending on how a return was identified for  potential examination. For returns identified with queries or models,  classifiers check a limited set of criteria once a return is identified. For  returns identified through other sources, such as referrals, the classifier  also reviews facts and circumstances about potential noncompliance in  returns. We focus here on examination sources that rely on data—such  as models or queries—and referrals. Referrals are complaints of exempt  organization noncompliance made by third parties, including the public  and other parts of the Internal Revenue Service. We describe referrals  classification because it is one of the top sources of exempt organizations  examinations.

The models are run to identify returns with potential noncompliance and  lists them on a Model Score Sheet (MSS). The MSS is a ranked list of  returns by scores from the model. According to a TE/GE official, the  classifier: works down the list, starting with the highest scores, to fill stocking  checks whether the return was also identified for a compliance  strategy; and. eliminates returns based on the statute of limitations and recent  examination activity.

For some projects in the Compliance Strategies portfolio, a query is run or  returns are sampled to identify a population meeting indicators of  potential noncompliance. Then, the classifier uses the stocking report to  select returns with certain geographic or case grade criteria and  eliminates returns based on statute of limitations, recent examination  status, and resolving non-filing issues, according to a TE/GE official.

TE/GE classifiers do a triage to review and eliminate referrals that are not  relevant to tax administration or do not have substantiated information.  The triage classifier sorts referrals and reviews the following: organization status (for example, already revoked or terminated); examination history of the organization; and evidence of substantial inurement or private benefit, non-exempt  activities, or material employment tax or unrelated business income  that would result in a significant tax assessment.

Referrals that pass triage are either sent to classification or, if they deal  with political issues, are sent to a committee of three TE/GE managers,  who vote on a selection decision. For all referrals, the classifier  researches the referral. Research sources include websites, external  databases, and IRS taxpayer account databases. The classifier may look  at the organization’s website, information about officers, or prior  examination history.

Referrals with examination potential are either assigned immediately or  placed on the virtual shelf. Referrals that must be immediately assigned  include those with strong indicators of fraud, illegal or illicit activities  (including terrorism), or referrals from whistleblowers, or certain other IRS  divisions. Other referrals are labeled as high, medium or lower priority,  based on potential for revocation or significant tax assessments.

Appendix V: Status of Contractor Recommendations on Exempt Organizations Examination Selection

The Tax Exempt and Government Entities division (TE/GE) hired a  contractor to review the effectiveness of its Form 990 examination  selection model. The contractor prepared two reports. The first, delivered  in January 2018, makes recommendations on the model process, the  computing environment, and performance measures. The second,  delivered in July 2018, makes recommendations to more effectively and  efficiently identify returns for examination, such as through the model.  Within the two reports, the contractor made 17 recommendations. Table  12 lists the recommendations and the status of each.

In September 2019, TE/GE initiated another study, anticipated to be  completed in September 2020. The study focuses on developing  alternatives to enhance the models. The study will explore architectures  and alternative designs for the model and propose alternative compliance  actions to examinations and recommend measures to monitor their  effectiveness.

Appendix VI: Comments from Internal Revenue Service

Appendix VII: GAO Contact and Staff Acknowledgments

In addition to the contact named above, Tom Short (Assistant Director),  Lindsay Swenson (Analyst-in-Charge), Ann Czapiewski, George  Guttman, Amalia Konstas, Krista Loose, Alan Rozzi, Cynthia Saunders,  Andrew J. Stephens, and Sonya Vartivarian made key contributions to  this report.

Tax-exempt organizations often provide charitable services or membership benefits—and they generally don’t pay federal income taxes. However, they do file an annual return to report their tax-exempt activities.

In 2016, IRS started using analytical models as part of an approach to use data to select which of these returns to review for compliance with tax-exempt laws. The models score each return on the likelihood that it isn’t in compliance.

But the scores may not be reliable because of problems with the models, documentation errors, and incomplete procedures.