For each size a city gains it receives a square of influence (by your choice) where you can build improvements and an additional space for a unit inside the city itself.
Each improvement also has a unit capacity (usually rather small) and depending on the type or amount of units there it might give a different sort of benefit.