I'm interested in how browser layout engines like gecko, webkit and trident are architected from a high level. what are the key abstractions? how are they layered? what are the inputs/outputs for the different abstractions? Is there a diagram or article that explains this particularly well?I am sure the implementations vary, but I'm generally curious about how all the pieces fit together. I realize webkit is open source and I can just look at the code, but I wouldn't know where to start. 
high level design of a browser layout engine?
architecture;browser
A few papers that will help you get a better understating of browser architecture: A Reference Architecture for Web BrowsersArchitecture and evolution of the modern web browserConceptual Architecture of FirefoxThe Security Architecture of the Chromium Browser Be warned, the links are to pdf files. The last one is not exactly what you are looking for, but it's a good read. As for specific layout engines:GeckoMozilla's Layout Engine, presentation by David BaronGecko Overview, in the Mozilla WikiWebkitYou can find some basic architectural diagrams in this presentation on Developing hybrid Web GTK+ applications by Alp Toker.TridentInternet Explorer Architecture, on MSDN