Late last summer, the New York City Department of Health began releasing the results of new restaurant inspections, and assigning letter grades based on compliance with the city Health Code and the state Sanitary Code. The letter grades have become ubiquitous in restaurant windows throughout the city. But what do they really mean, and how seriously should we take them in choosing a place to eat?
The grades take into account three categories of violations that may be observed during a restaurant’s annual inspection:
— Public health violations, with the potential to cause an immediate health threat. Inspectors are authorized to shut the place down immediately if the restaurant is unable to address these.
— Critical violations, which do not pose an immediate public health threat but are generally unappetizing, like the presence of live roaches, evidence of rats or mice, and food held at improper temperatures.
— General violations. The list of these is long, and may include improper thawing techniques for frozen meats, improper trash receptacles and unsanitary bathroom facilities.
(The complete list of violations is available in the city health department’s Blue Book for restaurants; it is not for the faint of heart.)
Restaurants are assigned points for each violation the inspectors find. The rules call for assigning at least 7 for a public health violation, 5 for a critical violation, 2 for a general violation, and the guidelines in the Blue Book give inspectors authority to assign more than that based on the severity of the violation. These points are added up to yield a letter grade: 13 points or less earns an A, 14 to 27 gets a B, and 28 points or more means the restaurant gets a C (and should probably be avoided at all costs).
Since the Department of Health began using the new system last July, more than 12,000 restaurants (of the estimated 24,000 restaurants in the city) have been graded. More than half (about 56 percent) received an A; slightly fewer than one-third (31 percent) received a B; and about one out of eight (12 percent) got a C grade:
Providing consumers with easy-to-read, easy-to-access information on their choices is valuable, both in restaurant window signs and through the health department’s website, which allows users to search for specific restaurants and access information about their inspections.
As it currently stands, though, the grading system is too coarse, because it masks wide variations in the quality of restaurants receiving the same grade, making it hard to say quite what it means to be an A-rated restaurant in New York City. Closer inspection of the underlying data reveals a suspicious distribution of restaurants near the cut-off point between an A and a B.
A restaurant receiving any score from 0 to 13 points gets an A, but the difference from one end of that range to the other is substantial. A zero score means that inspectors found no violations at all, while 13 points means they found a host of concerns. A (hypothetical) restaurant where hot food items were not being held above the required temperature of 140 degrees, toxic chemicals were improperly labeled or stored in a way that contamination of the food might occur, and the restroom had no toilet paper or trash can would get 12 points, according to the inspection system outlined in the Blue Book — and qualify for an A.
The graph below shows the distribution of A and B-rated restaurants inspected between July 28, 2010, when the grading began, and Jan. 3, 2011, when we compiled the data for this post. The horizontal axis tracks the number of violation points, and the vertical axis tracks the number of restaurants scoring in that category. The blue bars are A-rated restaurants, and the green bars are B-rated.
During this period, only 381 restaurants scored a perfect zero, while 1,012 scored 12 points and 721 scored 13 — just barely squeaking in to the A category. The data suggests that when you visit an A-rated restaurant, the odds are that it barely made the grade.
A more pressing statistical problem arises from the sharp break in the 