Does doing a PhD in a small/simple university make sense after getting a Master's from a very prestigious school? I got a Master's degree in Electrical Engineering from a school which is among the top ranked of Europe. Nevertheless, this is a pure coincidence, I did it here because it was the closest to my home.

For purely personal reasons I am tired of living in a place where there is such high population density, and having to suffer city life, crowded trains, traffic (even though I do not drive, I suffer the noise and pollution a lot), and spending so much time every day in queues is very annoying. My dream would be to return living in the countryside in a small village where people know each other, like when I was a kid (the village where I grew up is now today part of a suburb and mostly lost its soul).

I'm considering applying for a PhD position in a very small university in a medium sized town, in order to relocate and live in a rural village near the town. That university would be quite small and barely large enough to offer PhD programs, and definitely does not have much international reputation or anything. The exact opposite to the university where I got a Master's, which is high ranked and has amazing international reputation.

What I'd be doing would be the exact opposite of the mainstream, where people are always more willing to live in cities despite the disadvantages, and also are wanting to have their PhD from a school that is prestigious. I am afraid to kill my academic career because of a caprice of mine ("live in the countryside") would have taken up all the other advantages over staying here in the same School for a PhD.
I guess the first question is What do you want to do with your PhD?

If you want to stay in academia, you will have the same problem after graduating all over again (and, speaking from personal experience, as a young postdoc/faculty, one does not generally have the luxury of choosing where one goes next). Further, going to a (presumably) under-funded local university with weak research programme is going to severely reduce the probability of you doing world-class research. While Moriaty is correct in saying that "It's the quality of your individual work that counts.", said quality is definitely strongly correlated to the excellence of your advisors and fellow students and collaborators. Another problem may be funding (e.g., for going to conferences), which may not be plentiful in a small university.

What I am also wondering is whether you actually need to switch university, or should just live off-campus? I work in Zurich, and I have one student who used to live between Zurich and Bern (about a one-hour car drive away), in an area that was pretty much as rural as it gets. He just did a lot of home office, and took the train approximately 3 times a week. Maybe an arrangement like that helps you more than throwing out the baby with the bath water?