Why don't we subtract out / ignore the leap year every 128 years? When we add the extra day to the calendar every 4 years it's to "correct" the calendar under the assumption that the year's length is 365.25 days. Every year, you're .25 days off, so after 4 years, you're 1 day too far ahead, so add in an extra day to resync.
But in reality it's more like 365 days, 5 hours, 48 minutes, 45 seconds. In other words it's not 365.25 days exactly, we're over-stating it by 675 seconds.
Every 4 years when we add that day back we're actually adding 2700 seconds too much!
After doing this 32 times though (after 128 years), we'd have added 2700 * 32 = 86400 seconds too much. But this is exactly the number of seconds in a day, (24 * 60 * 60).
So why aren't we subtracting a day back out every 128 years? Instead we're subtracting the day out after 400 years instead.

You asked:

So why aren't we subtracting a day back out every 128 years? Instead we're subtracting the day out after 400 years instead.

The rule in the Gregorian calendar subtracts out the leap day 3 times in 400 years. (The leap days in years 1700, 1800, and 1900 were skipped; 2000 was not skipped, and 2100 will be skipped again.)
So we subtract out a day, on average, every $\frac{400}3$ years, and guess what, $\frac{400}3≈133.3$, which is quite close to the 128 you suggested.
The remaining discrepancy can be explained:


*

*At the time the Gregorian calendar was devised (1582 or thereabouts) the length of the year wasn't known with such accuracy.

*The length of the year varies slightly, so there's no point in fine-tuning the correction past a certain point.

*It's more important to have a simple rule than a perfectly accurate one. Whether the calendar loses a day every 3000 years or every 5000 years is a minor issue compared with the trouble of retraining everyone.


If you do want to fine-tune the calendar, I recommend 8 leap days every 33 years.
