The power of cache in a slow world
CC Attribution - ShareAlike 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this
May, Michael
Most of us have some form of cache anxiety. We’re good at caching static content like images and scripts, but taking the next step - caching dynamic and user-specific content - is confusing and often requires a leap of faith. In this talk you’ll learn new and old strategies for caching dynamic content, HTTP performance rules to live by, and a better understanding of how to accelerate any application. This is just what the doctor ordered: a prescription for cache money.
 My name is Michael May, and so in this talk we're going to cover a lot of things. Hopefully it will be too fast, and hopefully you'll learn something. So the first thing we'll talk about is just cool things about the Internet. So like routing some background on TCP, DNS, things that make the Internet work, but that also is real developers might not touch every day, but it is good to know something about these things. We'll talk about HTTP, what HTTP provides to us for caching, and then how to handle things like cookies. And finally the moment of the talk we'll be around actually doing acceleration with apps. I'll talk a lot about varnished cache and some caching strategies for caching dynamic content. So before we begin, I just want to point out that I work at a company called Vasli. We are a content delivery network. We're all about real time, so whether that's stats, log-streaming, you are instant purging in 150 milliseconds, we're all about being very fast. So we are actually built on top of a fourth of varnished cache, and we allow you to upload something called VCL, it stands for varnished config language. Now so does anyone in the room know about varnished cache? Have you used it, heard of it? Cool. What about VCL, varnished config language? Okay, cool. So that's what it is hiring, if you're interested, come find me after and let's chat. So I'd like to show a couple of these pictures. Vasli, we brought our own infrastructure, so we stick tons and tons of SSDs and RAM into all our cache nodes. Now, I'm going to talk about internet routing in just a moment, so if you find that really interesting, there's a talk by a colleague of mine called Scaling Networks Through Software, and that's from SRE-Con 2015, just a couple of weeks ago. Check that out. So, things about the internet. The first thing we know is that the internet is made up of many different autonomous systems. And autonomous systems, that just means that it's a network controlled by a single person or individual. So for example, Comcast has an autonomous system. Local 3, Vasli, we operate our own network, so we have an autonomous system. The real key thing here with autonomous systems is that there is an ASA, an autonomous system number, and that's just a unique identifier that identifies that network as something distinct from another network. And so, these ASAs are actually really critical in internet routing. So there's, say, called the Border Gate Protocol, and it's still BGB, and this is used to find the best path through the internet from point A to point B. So the way this works is when you connect up a new router to various ISPs that you may connect to, typically you'll can do this when you have more than one. So all of us in the room might be not doing this, like our Fusil provider is probably doing this for us. So when you connect a new router, it pulls all of the internet routes from those ISPs, it then analyzes those routes and finds something that's called the shortest AS path. And all that means is the path through the internet that traverses the least amount of these autonomous systems. So once you find that path, you then store the next top of that path in your routing table. And this is important because there are these things called period agreements and period. So period is when two autonomous systems get together and exchange this routing information. And so with that comes these agreements called period agreements, and you define how much traffic can be sent through their network, how much it costs, things like that. And these typically happen at physical points called internet exchanges. So these are very big, you know, doing like hundreds of teramits a second. And so like Amsterdam, AMSI, it is one of the largest. Now there's also just like a fun tidbit of information, there's this group called NANOGRAPH. And these, this is a gathering of all the network operators in North America. They come together for these period agreements at NANOGRAPH. So if you think that nobody actually controls the internet, NANOG totally does because we use a routing. Right, so if you ever hear someone tell you that nobody controls the internet, people totally do. And so if you go up like in a Mac, you can see the routing table, it's not terribly interesting, but you can do that with like a netstat.nr on the Mac or on Linux, it's just route. So it's like kind of basic internet routing in a nutshell. So let's just talk about TCP a little bit. And we should all be familiar that to create a new TCP connection, there's this thing called the three-way handshake. And that takes three round trips to turn your client to initiate a new connection. And the reason we care is because HTTP runs on top of TCP. So if every time that we have to send data to a client, we initialize one of the new connections, it's going to be very slow because it takes three round trips from server to client. Now there's this thing called the TCP window size, and that just controls how much data the server could send to the client at one time. And just one of the optimizations that you can do with TCP, not necessarily that you're going to do as Braille developers, but that you might expect your CDNs or your provider to do for you, it's what you call slow start. And that's just where you intuitively increase the window size so that you can send more data at one time from a server to a client. Now DNS, we use it every day, it's just the human name mapping to an IP address. So DNS has a TTL, it stands for Time to Live, and that justifies how long you can catch these DNS records. Now, has anyone used a dig or heard of a man called Dig? Also, so Dig is really handy. For example, if you go to digonfacil.com, you'll see that the question that we ask is, hey, fastly.com, give me the A record for this. And A record, that's just the actual IP address. So in this case, we came back with four different IPs. I'm not exactly sure why there's four here, but some of them might be unicast routes, some of them might be anycast routes, and then a couple of them are probably for redundancy. So if we do another one, we can see that this one has a seeding, digapi.facil.com, and we'll see that that is actually a seeding to global SSL, and that itself is another seeding to fold back to a global. And then finally, foldback.global results to two different A records. And the reason there's two A records here is just for redundancy purposes. If one of those IP addresses fails for some reason, then there's another one there that you can go to. And also notice there's some 10s, there's some 29s here. Those numbers are the TTL. And these are pretty short TTLs for DNS. And the reason we do this at fastly is that we update these very often so that we can always determine the closest pop to where the DNS resolver is. So that's just some background on the internet. Now, like, why we care? You probably don't necessarily care about that stuff, but I just think it's kind of cool. Now, the rest of this talk is about making things fast. And the reason that we care about making things fast is because it takes time to move data around the world, right? So for example, we're here in Atlanta. Say we have some clients that we need to talk to in Sydney, and that's about 10,000 miles, it's a pretty long distance. So just ideal at the speed of light, that might take 80 milliseconds for a round trip from here in Atlanta to Sydney all through Atlanta. Now, in reality, that might take a little longer, and it's actually going to take a lot longer. So you can find this out by doing a ping. And so here's one of our cash nodes in Sydney that I ping. And you'll see that for a round trip, it takes about upwards of 250, 350 milliseconds. So when you start thinking about that, and if you have to open a TCP connection to clients out in Sydney every time you're sending data, this is going to be a very long time, you know, 3 times 300, that's like kind of 3 milliseconds. So your client's already waiting almost a second just so that you can begin sending information. A little side note, I also did this from airplane walk-by, and that was taking like 700, 800 milliseconds. So like, everything in my play is terrible, but we already knew that. And I was curious, like where these packets were actually going, why it was taking so long. And so you just see here that like bounces around the like planes when for a while, and then like finally it resolved down here in Denver. Maybe I was above Denver at the time, I don't really know, but Trace out was a fun little tool. Now, instead of waiting for that 250, we have to live second round trip time. We can do like some cool things and decrease that, right? So here, in this case, Atlanta, Sydney, it's only taking 20, 30 milliseconds. So like what's going on here? Did we just eat the speed of light? Right, like if we did, people like Stephen Hawkins probably want to know, because they've been working on really hard problems like this for a long time. So did we just eat the speed of light? No, right? And actually, we cheated. So like my fourth grade teacher that told me cheating never gets anywhere, like take that, like on the internet, cheating can do lots of places, much, much faster. And the way that we cheated was with cashier. So with that, performance round trip time is something in mind. Let's talk about HTTP. And some of the things that HTTP provides us for cashier, cash control is HTTP header. We've probably all seen it, use it, love it, you know it. I just want to point out a couple directives for the cash control header that you may be less familiar with. And one of these is s-max age. And so this behaves exactly like max age, except it only applies to surveyed cashiers. Surveyed cashiers are being your CDN. So you can do some cool stuff where you define different cashier policies for the browser or for your back-end CDN cashiers. There's also some pretty new stuff that's interesting, referring to grace code. So there's the scale while revalidating, scale if error directives. And this tells the cash that it's okay to serve stale content while it does something in the background. So I revalidate, the cash is going to go fetch the fresh content in the background while it's still serving stale. Or an error, same type of view, if the content is stale, it makes a request to origin. Origin comes back with a 500 maybe it's down or something. So this way the cash will still serve content out to users and not serve those errors, which in my view is a much better user experience. I'd rather have my user see stale content than errors. Now the variant HTTP header is another header that can affect caching. Now a good use of the variant header would be variant on something like accepting coding. And this checks out because accepting coding has a pretty small set of permutations that can be made in a response. So there's like gzip, deflate, and non-cache versions. So all of those get cached but there's not too many permutations. So for example a bad use of the variant header would be variant on user agent. Please do not do this because there are thousands of different user agents. So if you're trying to cache responses and there's a user agent header in there, then you're really going to just eliminate any benefit that caching might provide you because there's so many different permutations. Now just to clarify things on cookies, the set cookie header is sent by the server, the cookie header is sent by the client. And in general proxy caches like CDNs will not cache anything that has a cookie header in it. And intuitively this makes sense and it should because generally when we have cookies in responses, there means there's some sort of private data in there that we don't want to cache in a public cache. So now I want to talk a little bit about varnish cache. So varnish is a hdpu-reversed proxy cache. It works just like a reverse proxy would. So it sits between your user and your server and all the requests flow through that. So generally when you see these things out in the wild in production, you'll see name your top level domain like example.com to the host name of your cache server or load balancer so that all of those requests to your top level domain flow through that cache. And just to give you an idea of what this actually looks like, here's the output of the curl to app.facil. So I'll just point out a couple things here. First is you see this via header. So all this does is tell you that this request was actually proxied. And in this case it was proxied through varnish. It could be proxied through something else like squid or any other proxy cache that you might be using. Now some other headers that are interesting are these x-served by xcash hits. And so any htp header you see that's prefixed by x- means that it's not an actual official htp header. It doesn't exist in the HTTP spec. It basically just means that someone made it up. So in this case these are kind of actually relevant to us because the x-served by tells us which cache this actually came out of. So here I'm in Atlanta, came out of the Atlanta cache. And this was also a miss because this request happened to include cookies. And because it was a cache miss there's no cache hits. Finally the thing to note is that we are using connection keep alive here. So curl in this case did not actually close the TCP connection. It left it open so that we can reuse it and not do that expensive three-round-the-trip TCP handshake every time we make a request. So varnish comes with something called varnish config language or VCL. And this is just a domain-specific language for interacting with the traffic that comes through your cache. And this VCL is translated to C and endowed to by code. So it actually has a pretty negligible effect on performance as opposed to if you were to do something similar with like middleware where that could affect performance, doing that in VCL doesn't do that. So varnish and VCL run on something called the varnish state machine. So the way this works is when varnish processes a request, that request goes through a bunch of different states. And at each point, each one of those different states, VCL allows you to inject logic at that state so that you can do some sort of processing. So for example, here's a couple of different states. Reqv is just the first routine that gets called deliver if the last routine is called fetch gets called after a cache miss and hit after a cache hit. And the error is an interesting one. You can do kind of interesting things with control flow there. So a quick example, this is pretty straightforward, say that we want to block our requests to an additive point. And if those requests don't have an authorization pattern, we're just going to forward one then. So this code right here, dcl, Reqv, remember that's the first sub routine that gets executed. So right immediately when this request comes in, we're going to forward one it. This request is going to pass back to our application server. So we don't even need logic there to do that in our application server to do that since we can do it out here. And then remember, feedback to those pings, doing this is a difference between 300 milliseconds and 30 milliseconds. Because with varnish now, if we're just setting these errors from varnish, those requests never take that other hundreds of milliseconds to go all the way back to our application server. So varnish has something really interesting also called synthetic responses. And this is just best shown by example. So say we have a page and this page has a like button with a counter. It counts the number of likes that this video has or whatever. So typically how we might cast this using a traditional reverse proxy would be, maybe there's a piece of JavaScript in the page. When you flip that button, it fires off a post request. That post request goes back to your origin server. It does a dv update and then comes back with a 200. And then when the JavaScript on a page sees that 200, it'll increase that counter. Now again, think if you're going back and forth from here to Sydney, that's going to take a really long time. So what synthetic responses allow you to do is completely bypass making that round trip to origin. So the way that this could work is that JavaScript on a page makes a post that post gets to your cache. And then you can have a synthetic response with like just say it returns 200 okay. And that 200 gets back to the client very quickly so that the client, or the browser in our case, gets that feedback immediately and the user is just not sitting there, clicking the button going like, hey, what's going on? Like I'm clicking you, you're not updating. And then the way that you would eventually update the database is you can send a log out from your cache and it has some background job that's processing those logs in real time. It's watching for updates and then answering them in the background. So the way that this actually looks in BCL is pretty straightforward. So we have a rep meeting, the first function that gets executed. We're saying if the URL matches like some like input and it's a post, we're going to throw an error. And here we're throwing error 666. Now that's not a real error, but that's okay. It doesn't matter because what that's going to do is going to break the Vargas state machine. It's going to break out of that and immediately go to this BCL error. Now BCL, all of these BCL functions are global and so they operate on all requests. So what we have to do in this error function or sub module is check, okay, we say it's a status 666. If it's 666, we know that this is going to be a synthetic response. So we're going to set the status to 200. This is going to come to the type and then our actual request body is just okay. And then we deliver it. So that's how you can do this and potentially this would save hundreds of milliseconds on that round thread back to your origin. And this works well for things that eventually will consist of C as okay. If you have things that actually need to be consistent, this won't work out for you, right? So let's move on and with some of that varnish, BCL knowledge, let's talk about strategies for caching dynamic content. Now first let's talk about what type of dynamic content is actually caching. So pure, truly dynamic content isn't caching. And that's stuff like real-time statistics or like credit card numbers, things that actually we don't want to cache. But generally when we think of dynamic content, such for example as like a JSON API, that's actually pretty static and it's actually pretty cached. The only problem with caching is that it changes unpredictable. So we need some form or some way to be able to end balance that. Now there's a lot of different strategies for caching these dynamic types of things. So one of them is short TTLs, there's a thing called cached-cited-cludes. You can do Ajax and then there's the API caching which is then driven purges. So the way that short TTLs work is pretty straightforward. Basically in your cache control you set your max age to something very small. So for example you can set it to one second. So what this is going to do is your caches are going to cache this thing for one second. All requests that come in will get that cache content. But then basically what you're doing here is flatlining your requests back to origin. Because your origin is only going to see one request a second from the cache. So it's going to absorb all the other load. So it's pretty cool. And you know this is a pretty easy strategy. You don't have to think a whole lot about doing this. And this works out pretty well. If your cache doesn't have some sort of invalidation mechanism, that's fast enough to keep up with dynamic content. Now edge-cited-cludes. As you've heard of edge-cited-cludes, they've been around for a really long time. Now so the way to think about these is kind of like Russian dollar caching. The way these work is templates are generated and cached on the actual edge cache. And then for pieces that are actually dynamic, such as personalized user information or things like that, you can actually fetch that content back to origin, get it back, and the template is actually generated at the cache and sent out from there. So when this actually looks in real life, say we have a nav bar, there's some links in it, and if the user's logged in, we want to be able to show their avatar and maybe their name there to give them a personalized look and feel. So the way this works, you'll see the ESI included. That is a type of keyword here that the cache will look through when it parses through HTML. And then to actually insert the content in there, it's going to make a request back to your application server that might look something like this curl. It'll be a cookie there, it'll be to the path that's defined in your source. And then here, you know, we're returning the name and an avatar. And so this doesn't just work for HTML. You can do this with any type of content, whether that's JSON, XML, whatever. And you can template these things as at will. So those are edge-side includes. Now, another way to cache dynamic content and make your applications faster is to extract all of that dynamic content into APIs and then fetch that asynchronously from the client. And this is good because we can get in and get our HTML to the client faster. Maybe we won't spend so much time building it on the application side. Now, when we have these APIs that Ajax is calling into, we can actually cache them. And so we do this by setting a cache control with some reasonably long TTL. Maybe it's a couple of days. We don't really know. We don't really care. It's something that works out for you. And each response will tag with a cache key, much like in Rails-Russian null caching, you cache with a cache key. Except in this case, we're going to use an HTTP header called Sergi key. Sergi key really just means cache key. Like, use it like that. And then, so once that content is cached, it's tagged with those Sergi keys, when it changes, we can perform a purge on that Sergi key so that content is removed from the cache and it gets updated on the next request. And so the way this might look in a Rails app is, you know, in our Git methods like index, we can set the Sergi key. Here we're using just a table key from the active-regarded model. And then, say we have an update that maps to an HTTP put. When this content updates, we can issue the purge and then render the new content. So, there's actually a Rails plugin that exposes some of these helpers to you. So check that out if that's maybe something you're interested in doing. Now, Rails also helps us out because it injects this CFRF meta tag token into all of our layouts. And so this is good and this is like, we don't want to eliminate this, but we can cache that base to HTML when there's a unique CFRF token in each page. So one of the ways that we can extract that to make it more catchable, that base content more catchable, we can use an edge side of boot to include that CFRF at CFRF time. We could maybe add that CFRF token into the cookie and client side extract that cookie somehow. Or my favorite one of these is to extract that CFRF token into a private API endpoint and then you can age-ax that into the page asynchronously. So we're approaching the end, this is kind of the last strategy I'm going to talk about, and this is caching things when your responses have cookies. So the way that you do this is when a response, when a request with a cookie comes in, you strip that cookie out of the request and save it to a temporary variable. Then you'll do a normal cache load up like you would, and because you strip that cookie out, that would be part of the cache load up. And then once you fetch that content from the cache, you can set that cookie back on the response from the temporary variable. Now the way that this might look in VCL, something like this, here we have rectV. Remember it's the first sub-module that's subroutine that's executed whenever a request comes in the barge. So in the queue we can say, if there's a cookie named this cookie that I'm looking for, take that cookie, store it in a temporary variable, and un-set the cookie header. And then similarly, VCL deliver, this is the last subroutine that executes before you deliver your content out to your client. Here we can say, if we set this temporary cookie, set cookie header to the value of that cookie that we temporarily store. And so this is how it works when there's a cache hit. If there's a cache miss, it's slightly different because you will have to fetch that content from origin. But you can do that. VCL fetch is the subroutine that executes right after the response comes back from your back-end application server. So here what we're saying is, if there's a set cookie header in that response, store it to this temporary variable, rip it out of the response from your application server so that the cache can actually cache that. And then back here, when we deliver, we'll just go ahead and set it again. And so that's how you can cache content that has cookies in it. You basically fake it. And so that's about all. We're pretty much done here. So these final thoughts, tools like DIG and TracerOut are super helpful to you with deep-bub and network problems. Two of your cache control headers, you can get very, very fine-grained control from your cache by using this. And then I didn't really touch on this too much, but try replacing some of your middleware with logic in VCL at the edge and see what happens in performance. Finally, we talked about a couple of different caching strategies. None of them are wrong. They're all right, but pick one that is most comfortable to you and use it will be easiest for you. And finally, we talked about stripping cookies out to cache what would otherwise be considered private content and unpatchable. So that's about all that I have. I think you have some time for questions if anyone has any. Yes? Do you varnish something? No, I think it's fine. So the question is, is varnish something that would replace your existing reverse proxy like Intunex? And the answer is it doesn't have to. So you can totally put varnish behind or in front of Intunex, however you want to do it. And I mentioned that those via headers, if you have a teacher reverse proxy, it's just going to depend on each proxy that goes through into that header. So you can be able to turn to that. It could, absolutely. Sure. Any other questions? All right. Oh, in the back. For the synthesized responses, is there a way to handle errors? Sure. So the question is, in these synthetic responses, is there a way to handle errors there if you get one from your origin server? Yes. And the answer is yes. I didn't cover it because it's a little bit of an advanced topic, but absolutely. You can define some logic there that says, oh, if I get a five third back, do this instead. So absolutely. Thank you. Anyone? Yeah, there are significant amounts of the internet that don't respect these cache primal. The question is, is there any significant part of the internet that doesn't respect these cache TTLs? And the answer is probably not unless you're using a really terrible cache. In general, as long as the person who implemented your cache was reasonably sane, they should be able to handle these TTLs. Yes. A lot of ISPs, thank you for the point of view of the end user, a lot of ISPs implement a source of edge caches also. How do they play well with that? They talk special, but generally with people themselves. So that if an edge cache closest to the user, how would it know to expire sooner than its default time? Right. So the question is, various ISPs may have their own edge caches. So how can you guarantee invalidation of that content? Is that along the direct line? Let's say you have a kind of edge cache really close to you, so the first thing you're going to hit, and it has a ridiculously long TTL line, what do you think it is? But let's say a year, a year, because it's ridiculously long with the edge cache time. And then somebody goes from that cache to your cache. Sure. So the likely to be warm in the localist cache? Sure. So how do you fire a question? I'm just trying to keep that up. But anyway, the two caches are talking to each other so that you end up with what's realistically recent content. Right. So I mean, if you invalidate a piece of content on your cache, and there's another cache sitting behind that, or ahead of it, right, so there's something cache in front of the cache that you actually own, then you really don't have any control over that. And it's not, like, you can't even say that they maybe support purging or invalidation. You don't even know. So really the only thing that you know about is the cache that you control. I hope that answers your question. Sorry, so it's not what I wanted to hear, but it's thanks to the caches. Right. Yes. For the synthetic requests, do you play them in the background? The question is, for the synthetic requests, does it just replay that for the background job? Right. There's like Q&A around the same thing. Okay, so my example actually where I was doing this, I totally left out the log statement. So I apologize for that. But there is a VCL subroutine called VCL log. And so then in that, you can check for those error codes, and that's where you can define how your log looks and where it is and how. But that happens in the flow of the processing tab requests response. It's not tuned up or anything. It's immediately sent out in real time to wherever you want it to go. Last one, I think. For example, I have short TTL. Does that one have a slow down client? Do they have to do a lot of DNS? Ah, good question. So the question was, short TTLs, does that end up slowing down the client since they have to do more DNS lookups to press the content? And the answer is not necessarily because if you could define something in that cash control, you could find a directive that specifies part to be okay for it to be stale. And so that way, you don't have your clients necessarily requesting this content every second or every time. So you could put like some grace window on there, maybe 30 seconds, one minute, if your content allows for that. And I think that's about all we have time for. If you have more questions, feel free to leave a comment and chat with me. Thank you.
 My name is Michael May, and so in this talk we're going to cover a lot of things.  Hopefully it will be too fast, and hopefully you'll learn something.  So the first thing we'll talk about is just cool things about the Internet.  So like routing some background on TCP, DNS, things that make the Internet work,  but that also is real developers might not touch every day,  but it is good to know something about these things.  We'll talk about HTTP, what HTTP provides to us for caching,  and then how to handle things like cookies.  And finally the moment of the talk we'll be around actually doing acceleration with apps.  I'll talk a lot about varnished cache and some caching strategies for caching dynamic content.  So before we begin, I just want to point out that I work at a company called Vasli.  We are a content delivery network.  We're all about real time, so whether that's stats, log-streaming, you are instant purging in 150 milliseconds,  we're all about being very fast.  So we are actually built on top of a fourth of varnished cache,  and we allow you to upload something called VCL, it stands for varnished config language.  Now so does anyone in the room know about varnished cache?  Have you used it, heard of it?  Cool. What about VCL, varnished config language?  Okay, cool.  So that's what it is hiring, if you're interested, come find me after and let's chat.  So I'd like to show a couple of these pictures.  Vasli, we brought our own infrastructure, so we stick tons and tons of SSDs and RAM into all our cache nodes.  Now, I'm going to talk about internet routing in just a moment, so if you find that really interesting,  there's a talk by a colleague of mine called Scaling Networks Through Software,  and that's from SRE-Con 2015, just a couple of weeks ago.  Check that out.  So, things about the internet. The first thing we know is that the internet is made up of many different autonomous systems.  And autonomous systems, that just means that it's a network controlled by a single person or individual.  So for example, Comcast has an autonomous system.  Local 3, Vasli, we operate our own network, so we have an autonomous system.  The real key thing here with autonomous systems is that there is an ASA, an autonomous system number,  and that's just a unique identifier that identifies that network as something distinct from another network.  And so, these ASAs are actually really critical in internet routing.  So there's, say, called the Border Gate Protocol, and it's still BGB,  and this is used to find the best path through the internet from point A to point B.  So the way this works is when you connect up a new router to various ISPs that you may connect to,  typically you'll can do this when you have more than one.  So all of us in the room might be not doing this, like our Fusil provider is probably doing this for us.  So when you connect a new router, it pulls all of the internet routes from those ISPs,  it then analyzes those routes and finds something that's called the shortest AS path.  And all that means is the path through the internet that traverses the least amount of these autonomous systems.  So once you find that path, you then store the next top of that path in your routing table.  And this is important because there are these things called period agreements and period.  So period is when two autonomous systems get together and exchange this routing information.  And so with that comes these agreements called period agreements,  and you define how much traffic can be sent through their network, how much it costs, things like that.  And these typically happen at physical points called internet exchanges.  So these are very big, you know, doing like hundreds of teramits a second.  And so like Amsterdam, AMSI, it is one of the largest.  Now there's also just like a fun tidbit of information, there's this group called NANOGRAPH.  And these, this is a gathering of all the network operators in North America.  They come together for these period agreements at NANOGRAPH.  So if you think that nobody actually controls the internet, NANOG totally does because we use a routing.  Right, so if you ever hear someone tell you that nobody controls the internet, people totally do.  And so if you go up like in a Mac, you can see the routing table, it's not terribly interesting,  but you can do that with like a netstat.nr on the Mac or on Linux, it's just route.  So it's like kind of basic internet routing in a nutshell.  So let's just talk about TCP a little bit.  And we should all be familiar that to create a new TCP connection, there's this thing called the three-way handshake.  And that takes three round trips to turn your client to initiate a new connection.  And the reason we care is because HTTP runs on top of TCP.  So if every time that we have to send data to a client, we initialize one of the new connections,  it's going to be very slow because it takes three round trips from server to client.  Now there's this thing called the TCP window size, and that just controls how much data the server could send to the client at one time.  And just one of the optimizations that you can do with TCP, not necessarily that you're going to do as Braille developers,  but that you might expect your CDNs or your provider to do for you, it's what you call slow start.  And that's just where you intuitively increase the window size so that you can send more data at one time from a server to a client.  Now DNS, we use it every day, it's just the human name mapping to an IP address.  So DNS has a TTL, it stands for Time to Live, and that justifies how long you can catch these DNS records.  Now, has anyone used a dig or heard of a man called Dig? Also, so Dig is really handy.  For example, if you go to digonfacil.com, you'll see that the question that we ask is,  hey, fastly.com, give me the A record for this.  And A record, that's just the actual IP address.  So in this case, we came back with four different IPs.  I'm not exactly sure why there's four here, but some of them might be unicast routes, some of them might be anycast routes,  and then a couple of them are probably for redundancy.  So if we do another one, we can see that this one has a seeding, digapi.facil.com,  and we'll see that that is actually a seeding to global SSL, and that itself is another seeding to fold back to a global.  And then finally, foldback.global results to two different A records.  And the reason there's two A records here is just for redundancy purposes.  If one of those IP addresses fails for some reason, then there's another one there that you can go to.  And also notice there's some 10s, there's some 29s here.  Those numbers are the TTL.  And these are pretty short TTLs for DNS.  And the reason we do this at fastly is that we update these very often  so that we can always determine the closest pop to where the DNS resolver is.  So that's just some background on the internet.  Now, like, why we care? You probably don't necessarily care about that stuff, but I just think it's kind of cool.  Now, the rest of this talk is about making things fast.  And the reason that we care about making things fast is because it takes time to move data around the world, right?  So for example, we're here in Atlanta.  Say we have some clients that we need to talk to in Sydney,  and that's about 10,000 miles, it's a pretty long distance.  So just ideal at the speed of light, that might take 80 milliseconds for a round trip from here in Atlanta to Sydney all through Atlanta.  Now, in reality, that might take a little longer, and it's actually going to take a lot longer.  So you can find this out by doing a ping.  And so here's one of our cash nodes in Sydney that I ping.  And you'll see that for a round trip, it takes about upwards of 250, 350 milliseconds.  So when you start thinking about that, and if you have to open a TCP connection to clients out in Sydney every time you're sending data,  this is going to be a very long time, you know, 3 times 300, that's like kind of 3 milliseconds.  So your client's already waiting almost a second just so that you can begin sending information.  A little side note, I also did this from airplane walk-by, and that was taking like 700, 800 milliseconds.  So like, everything in my play is terrible, but we already knew that.  And I was curious, like where these packets were actually going, why it was taking so long.  And so you just see here that like bounces around the like planes when for a while,  and then like finally it resolved down here in Denver.  Maybe I was above Denver at the time, I don't really know, but Trace out was a fun little tool.  Now, instead of waiting for that 250, we have to live second round trip time.  We can do like some cool things and decrease that, right?  So here, in this case, Atlanta, Sydney, it's only taking 20, 30 milliseconds.  So like what's going on here?  Did we just eat the speed of light?  Right, like if we did, people like Stephen Hawkins probably want to know,  because they've been working on really hard problems like this for a long time.  So did we just eat the speed of light? No, right?  And actually, we cheated.  So like my fourth grade teacher that told me cheating never gets anywhere, like take that, like on the internet,  cheating can do lots of places, much, much faster.  And the way that we cheated was with cashier.  So with that, performance round trip time is something in mind.  Let's talk about HTTP.  And some of the things that HTTP provides us for cashier, cash control is HTTP header.  We've probably all seen it, use it, love it, you know it.  I just want to point out a couple directives for the cash control header that you may be less familiar with.  And one of these is s-max age.  And so this behaves exactly like max age, except it only applies to surveyed cashiers.  Surveyed cashiers are being your CDN.  So you can do some cool stuff where you define different cashier policies for the browser or for your back-end CDN cashiers.  There's also some pretty new stuff that's interesting, referring to grace code.  So there's the scale while revalidating, scale if error directives.  And this tells the cash that it's okay to serve stale content while it does something in the background.  So I revalidate, the cash is going to go fetch the fresh content in the background while it's still serving stale.  Or an error, same type of view, if the content is stale, it makes a request to origin.  Origin comes back with a 500 maybe it's down or something.  So this way the cash will still serve content out to users and not serve those errors,  which in my view is a much better user experience.  I'd rather have my user see stale content than errors.  Now the variant HTTP header is another header that can affect caching.  Now a good use of the variant header would be variant on something like accepting coding.  And this checks out because accepting coding has a pretty small set of permutations that can be made in a response.  So there's like gzip, deflate, and non-cache versions.  So all of those get cached but there's not too many permutations.  So for example a bad use of the variant header would be variant on user agent.  Please do not do this because there are thousands of different user agents.  So if you're trying to cache responses and there's a user agent header in there,  then you're really going to just eliminate any benefit that caching might provide you  because there's so many different permutations.  Now just to clarify things on cookies, the set cookie header is sent by the server,  the cookie header is sent by the client.  And in general proxy caches like CDNs will not cache anything that has a cookie header in it.  And intuitively this makes sense and it should because generally when we have cookies in responses,  there means there's some sort of private data in there that we don't want to cache in a public cache.  So now I want to talk a little bit about varnish cache.  So varnish is a hdpu-reversed proxy cache.  It works just like a reverse proxy would.  So it sits between your user and your server and all the requests flow through that.  So generally when you see these things out in the wild in production,  you'll see name your top level domain like example.com to the host name of your cache server or load balancer  so that all of those requests to your top level domain flow through that cache.  And just to give you an idea of what this actually looks like,  here's the output of the curl to app.facil.  So I'll just point out a couple things here.  First is you see this via header.  So all this does is tell you that this request was actually proxied.  And in this case it was proxied through varnish.  It could be proxied through something else like squid or any other proxy cache that you might be using.  Now some other headers that are interesting are these x-served by xcash hits.  And so any htp header you see that's prefixed by x- means that it's not an actual official htp header.  It doesn't exist in the HTTP spec.  It basically just means that someone made it up.  So in this case these are kind of actually relevant to us because the x-served by tells us which cache this actually came out of.  So here I'm in Atlanta, came out of the Atlanta cache.  And this was also a miss because this request happened to include cookies.  And because it was a cache miss there's no cache hits.  Finally the thing to note is that we are using connection keep alive here.  So curl in this case did not actually close the TCP connection.  It left it open so that we can reuse it and not do that expensive three-round-the-trip TCP handshake every time we make a request.  So varnish comes with something called varnish config language or VCL.  And this is just a domain-specific language for interacting with the traffic that comes through your cache.  And this VCL is translated to C and endowed to by code.  So it actually has a pretty negligible effect on performance as opposed to if you were to do something similar with like middleware  where that could affect performance, doing that in VCL doesn't do that.  So varnish and VCL run on something called the varnish state machine.  So the way this works is when varnish processes a request, that request goes through a bunch of different states.  And at each point, each one of those different states, VCL allows you to inject logic at that state so that you can do some sort of processing.  So for example, here's a couple of different states.  Reqv is just the first routine that gets called deliver if the last routine is called fetch gets called after a cache miss and hit after a cache hit.  And the error is an interesting one.  You can do kind of interesting things with control flow there.  So a quick example, this is pretty straightforward, say that we want to block our requests to an additive point.  And if those requests don't have an authorization pattern, we're just going to forward one then.  So this code right here, dcl, Reqv, remember that's the first sub routine that gets executed.  So right immediately when this request comes in, we're going to forward one it.  This request is going to pass back to our application server.  So we don't even need logic there to do that in our application server to do that since we can do it out here.  And then remember, feedback to those pings, doing this is a difference between 300 milliseconds and 30 milliseconds.  Because with varnish now, if we're just setting these errors from varnish, those requests never take that other hundreds of milliseconds to go all the way back to our application server.  So varnish has something really interesting also called synthetic responses.  And this is just best shown by example.  So say we have a page and this page has a like button with a counter.  It counts the number of likes that this video has or whatever.  So typically how we might cast this using a traditional reverse proxy would be,  maybe there's a piece of JavaScript in the page.  When you flip that button, it fires off a post request.  That post request goes back to your origin server.  It does a dv update and then comes back with a 200.  And then when the JavaScript on a page sees that 200, it'll increase that counter.  Now again, think if you're going back and forth from here to Sydney, that's going to take a really long time.  So what synthetic responses allow you to do is completely bypass making that round trip to origin.  So the way that this could work is that JavaScript on a page makes a post that post gets to your cache.  And then you can have a synthetic response with like just say it returns 200 okay.  And that 200 gets back to the client very quickly so that the client, or the browser in our case,  gets that feedback immediately and the user is just not sitting there,  clicking the button going like, hey, what's going on?  Like I'm clicking you, you're not updating.  And then the way that you would eventually update the database is you can send a log out from your cache  and it has some background job that's processing those logs in real time.  It's watching for updates and then answering them in the background.  So the way that this actually looks in BCL is pretty straightforward.  So we have a rep meeting, the first function that gets executed.  We're saying if the URL matches like some like input and it's a post, we're going to throw an error.  And here we're throwing error 666.  Now that's not a real error, but that's okay.  It doesn't matter because what that's going to do is going to break the Vargas state machine.  It's going to break out of that and immediately go to this BCL error.  Now BCL, all of these BCL functions are global and so they operate on all requests.  So what we have to do in this error function or sub module is check, okay, we say it's a status 666.  If it's 666, we know that this is going to be a synthetic response.  So we're going to set the status to 200.  This is going to come to the type and then our actual request body is just okay.  And then we deliver it.  So that's how you can do this and potentially this would save hundreds of milliseconds on that round thread back to your origin.  And this works well for things that eventually will consist of C as okay.  If you have things that actually need to be consistent, this won't work out for you, right?  So let's move on and with some of that varnish, BCL knowledge,  let's talk about strategies for caching dynamic content.  Now first let's talk about what type of dynamic content is actually caching.  So pure, truly dynamic content isn't caching.  And that's stuff like real-time statistics or like credit card numbers,  things that actually we don't want to cache.  But generally when we think of dynamic content, such for example as like a JSON API,  that's actually pretty static and it's actually pretty cached.  The only problem with caching is that it changes unpredictable.  So we need some form or some way to be able to end balance that.  Now there's a lot of different strategies for caching these dynamic types of things.  So one of them is short TTLs, there's a thing called cached-cited-cludes.  You can do Ajax and then there's the API caching which is then driven purges.  So the way that short TTLs work is pretty straightforward.  Basically in your cache control you set your max age to something very small.  So for example you can set it to one second.  So what this is going to do is your caches are going to cache this thing for one second.  All requests that come in will get that cache content.  But then basically what you're doing here is flatlining your requests back to origin.  Because your origin is only going to see one request a second from the cache.  So it's going to absorb all the other load.  So it's pretty cool.  And you know this is a pretty easy strategy.  You don't have to think a whole lot about doing this.  And this works out pretty well.  If your cache doesn't have some sort of invalidation mechanism,  that's fast enough to keep up with dynamic content.  Now edge-cited-cludes.  As you've heard of edge-cited-cludes, they've been around for a really long time.  Now so the way to think about these is kind of like Russian dollar caching.  The way these work is templates are generated and cached on the actual edge cache.  And then for pieces that are actually dynamic, such as personalized user information or things like that,  you can actually fetch that content back to origin, get it back,  and the template is actually generated at the cache and sent out from there.  So when this actually looks in real life, say we have a nav bar, there's some links in it,  and if the user's logged in, we want to be able to show their avatar and maybe their name there  to give them a personalized look and feel.  So the way this works, you'll see the ESI included.  That is a type of keyword here that the cache will look through when it parses through HTML.  And then to actually insert the content in there,  it's going to make a request back to your application server that might look something like this curl.  It'll be a cookie there, it'll be to the path that's defined in your source.  And then here, you know, we're returning the name and an avatar.  And so this doesn't just work for HTML.  You can do this with any type of content, whether that's JSON, XML, whatever.  And you can template these things as at will.  So those are edge-side includes.  Now, another way to cache dynamic content and make your applications faster  is to extract all of that dynamic content into APIs and then fetch that asynchronously from the client.  And this is good because we can get in and get our HTML to the client faster.  Maybe we won't spend so much time building it on the application side.  Now, when we have these APIs that Ajax is calling into,  we can actually cache them.  And so we do this by setting a cache control with some reasonably long TTL.  Maybe it's a couple of days. We don't really know. We don't really care.  It's something that works out for you.  And each response will tag with a cache key, much like in Rails-Russian null caching,  you cache with a cache key.  Except in this case, we're going to use an HTTP header called Sergi key.  Sergi key really just means cache key. Like, use it like that.  And then, so once that content is cached, it's tagged with those Sergi keys,  when it changes, we can perform a purge on that Sergi key  so that content is removed from the cache and it gets updated on the next request.  And so the way this might look in a Rails app is, you know, in our Git methods like index,  we can set the Sergi key. Here we're using just a table key from the active-regarded model.  And then, say we have an update that maps to an HTTP put.  When this content updates, we can issue the purge and then render the new content.  So, there's actually a Rails plugin that exposes some of these helpers to you.  So check that out if that's maybe something you're interested in doing.  Now, Rails also helps us out because it injects this CFRF meta tag token into all of our layouts.  And so this is good and this is like, we don't want to eliminate this,  but we can cache that base to HTML when there's a unique CFRF token in each page.  So one of the ways that we can extract that to make it more catchable,  that base content more catchable, we can use an edge side of boot to include that CFRF at CFRF time.  We could maybe add that CFRF token into the cookie and client side extract that cookie somehow.  Or my favorite one of these is to extract that CFRF token into a private API endpoint  and then you can age-ax that into the page asynchronously.  So we're approaching the end, this is kind of the last strategy I'm going to talk about,  and this is caching things when your responses have cookies.  So the way that you do this is when a response, when a request with a cookie comes in,  you strip that cookie out of the request and save it to a temporary variable.  Then you'll do a normal cache load up like you would, and because you strip that cookie out,  that would be part of the cache load up.  And then once you fetch that content from the cache, you can set that cookie back on the response from the temporary variable.  Now the way that this might look in VCL, something like this, here we have rectV.  Remember it's the first sub-module that's subroutine that's executed whenever a request comes in the barge.  So in the queue we can say, if there's a cookie named this cookie that I'm looking for,  take that cookie, store it in a temporary variable, and un-set the cookie header.  And then similarly, VCL deliver, this is the last subroutine that executes  before you deliver your content out to your client.  Here we can say, if we set this temporary cookie, set cookie header to the value of that cookie that we temporarily store.  And so this is how it works when there's a cache hit.  If there's a cache miss, it's slightly different because you will have to fetch that content from origin.  But you can do that.  VCL fetch is the subroutine that executes right after the response comes back from your back-end application server.  So here what we're saying is, if there's a set cookie header in that response, store it to this temporary variable,  rip it out of the response from your application server so that the cache can actually cache that.  And then back here, when we deliver, we'll just go ahead and set it again.  And so that's how you can cache content that has cookies in it.  You basically fake it.  And so that's about all.  We're pretty much done here.  So these final thoughts, tools like DIG and TracerOut are super helpful to you with deep-bub and network problems.  Two of your cache control headers, you can get very, very fine-grained control from your cache by using this.  And then I didn't really touch on this too much, but try replacing some of your middleware with logic in VCL at the edge and see what happens in performance.  Finally, we talked about a couple of different caching strategies.  None of them are wrong.  They're all right, but pick one that is most comfortable to you and use it will be easiest for you.  And finally, we talked about stripping cookies out to cache what would otherwise be considered private content and unpatchable.  So that's about all that I have.  I think you have some time for questions if anyone has any.  Yes?  Do you varnish something?  No, I think it's fine.  So the question is, is varnish something that would replace your existing reverse proxy like Intunex?  And the answer is it doesn't have to.  So you can totally put varnish behind or in front of Intunex, however you want to do it.  And I mentioned that those via headers, if you have a teacher reverse proxy, it's just going to depend on each proxy that goes through into that header.  So you can be able to turn to that.  It could, absolutely.  Sure.  Any other questions?  All right.  Oh, in the back.  For the synthesized responses, is there a way to handle errors?  Sure.  So the question is, in these synthetic responses, is there a way to handle errors there if you get one from your origin server?  Yes.  And the answer is yes.  I didn't cover it because it's a little bit of an advanced topic, but absolutely.  You can define some logic there that says, oh, if I get a five third back, do this instead.  So absolutely.  Thank you.  Anyone?  Yeah, there are significant amounts of the internet that don't respect these cache primal.  The question is, is there any significant part of the internet that doesn't respect these cache TTLs?  And the answer is probably not unless you're using a really terrible cache.  In general, as long as the person who implemented your cache was reasonably sane, they should be able to handle these TTLs.  Yes.  A lot of ISPs, thank you for the point of view of the end user, a lot of ISPs implement a source of edge caches also.  How do they play well with that?  They talk special, but generally with people themselves.  So that if an edge cache closest to the user, how would it know to expire sooner than its default time?  Right.  So the question is, various ISPs may have their own edge caches.  So how can you guarantee invalidation of that content?  Is that along the direct line?  Let's say you have a kind of edge cache really close to you, so the first thing you're going to hit, and it has a ridiculously long TTL line,  what do you think it is?  But let's say a year, a year, because it's ridiculously long with the edge cache time.  And then somebody goes from that cache to your cache.  Sure.  So the likely to be warm in the localist cache?  Sure.  So how do you fire a question?  I'm just trying to keep that up.  But anyway, the two caches are talking to each other so that you end up with what's realistically recent content.  Right.  So I mean, if you invalidate a piece of content on your cache, and there's another cache sitting behind that,  or ahead of it, right, so there's something cache in front of the cache that you actually own,  then you really don't have any control over that.  And it's not, like, you can't even say that they maybe support purging or invalidation.  You don't even know.  So really the only thing that you know about is the cache that you control.  I hope that answers your question.  Sorry, so it's not what I wanted to hear, but it's thanks to the caches.  Right.  Yes.  For the synthetic requests, do you play them in the background?  The question is, for the synthetic requests, does it just replay that for the background job?  Right.  There's like Q&A around the same thing.  Okay, so my example actually where I was doing this, I totally left out the log statement.  So I apologize for that.  But there is a VCL subroutine called VCL log.  And so then in that, you can check for those error codes, and that's where you can define how your log looks and where it is and how.  But that happens in the flow of the processing tab requests response.  It's not tuned up or anything.  It's immediately sent out in real time to wherever you want it to go.  Last one, I think.  For example, I have short TTL.  Does that one have a slow down client?  Do they have to do a lot of DNS?  Ah, good question.  So the question was, short TTLs, does that end up slowing down the client since they have to do more DNS lookups to press the content?  And the answer is not necessarily because if you could define something in that cash control, you could find a directive that specifies part to be okay for it to be stale.  And so that way, you don't have your clients necessarily requesting this content every second or every time.  So you could put like some grace window on there, maybe 30 seconds, one minute, if your content allows for that.  And I think that's about all we have time for.  If you have more questions, feel free to leave a comment and chat with me.  Thank you.
The_power_of_cache_in_a_slow_world_0_47.png The_power_of_cache_in_a_slow_world_48_155.png The_power_of_cache_in_a_slow_world_156_299.png The_power_of_cache_in_a_slow_world_300_371.png The_power_of_cache_in_a_slow_world_372_1535.png The_power_of_cache_in_a_slow_world_1536_1619.png The_power_of_cache_in_a_slow_world_1620_2651.png The_power_of_cache_in_a_slow_world_2652_3335.png The_power_of_cache_in_a_slow_world_3336_3407.png The_power_of_cache_in_a_slow_world_3408_4379.png The_power_of_cache_in_a_slow_world_4380_5831.png The_power_of_cache_in_a_slow_world_5832_8747.png The_power_of_cache_in_a_slow_world_8748_10943.png The_power_of_cache_in_a_slow_world_10944_12959.png The_power_of_cache_in_a_slow_world_12960_13055.png The_power_of_cache_in_a_slow_world_13056_13103.png The_power_of_cache_in_a_slow_world_13104_13283.png The_power_of_cache_in_a_slow_world_13284_14075.png The_power_of_cache_in_a_slow_world_14076_15251.png The_power_of_cache_in_a_slow_world_15252_16595.png The_power_of_cache_in_a_slow_world_16596_19475.png The_power_of_cache_in_a_slow_world_19476_20867.png The_power_of_cache_in_a_slow_world_20868_21791.png The_power_of_cache_in_a_slow_world_21792_22547.png The_power_of_cache_in_a_slow_world_22548_22691.png The_power_of_cache_in_a_slow_world_22692_23123.png The_power_of_cache_in_a_slow_world_23124_25571.png The_power_of_cache_in_a_slow_world_25572_26675.png The_power_of_cache_in_a_slow_world_26676_28367.png The_power_of_cache_in_a_slow_world_28368_28739.png The_power_of_cache_in_a_slow_world_28740_32771.png The_power_of_cache_in_a_slow_world_32772_33035.png The_power_of_cache_in_a_slow_world_33036_33875.png The_power_of_cache_in_a_slow_world_33876_34271.png The_power_of_cache_in_a_slow_world_34272_36491.png The_power_of_cache_in_a_slow_world_36492_38675.png The_power_of_cache_in_a_slow_world_38676_39791.png The_power_of_cache_in_a_slow_world_39792_42035.png The_power_of_cache_in_a_slow_world_42036_43055.png The_power_of_cache_in_a_slow_world_43056_45167.png The_power_of_cache_in_a_slow_world_45168_46607.png The_power_of_cache_in_a_slow_world_46608_56147.png The_power_of_cache_in_a_slow_world_56148_56231.png The_power_of_cache_in_a_slow_world_56232_56291.png The_power_of_cache_in_a_slow_world_56292_56461.png