What generally happens if a department is unable to attract students? Say the math department used to teach 200 students, but because each year the number of students interested in math has decreased, now it only teaches 100 students. What are the usual practical consequences?

This clearly depends on the internal policy of the institute, but I wonder if some generalizations are possible, e.g. whether the department will be forced to downsize or whether people outside the department will intervene and try to help the department get more students.
As comments already indicate, this is hugely dependent on the country and university. 

When I was in the Netherlands funding of the university was directly related to the number of students and the number of students who finished. This dependency was passed on by the universities to the departments. Typically, there was some sort of delay build in, so the budget would not become too unpredictable. 

Usually fluctuations are not as dramatic as in your scenario, and smaller decreases can usually be covered by reducing the budget for conference visits, inviting speakers, etc.

However if the number of students halfs then cuts in personel are pretty much unavoidable, as personel is by far the largest post in the budget. However, firing people on a permanent contract is hard. It is much much easier to not prolong a temporary contract. The department usualy consisted of a small number of professors (on permanent contract) plus a larger number of lecturers (some on a permanent contract and some on a temporary contract). So the professors and lecturers on a permament contract are safe, but they will have a smaller department.

So this is a case where there was an explicit policy to add such a feedback. However, this description is specific to the Netherlands (and may not even be current). Other countries are organized differently. Even within countries there can be a difference between universities or types of universities.