How Disqus is using Django as the basis of our Service Oriented Architecture
CC Attribution 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Hitchcock, Adam
adam - How Disqus is using Django as the basis of our Service Oriented Architecture Disqus maintains the largest Django app out there. And we love it! It has, however, grown rather large and unwieldy. In the last year Disqus has had an increasing number of smaller services cropping up based on several different platforms. So this talk will be about how we do continuous deployment with our emerging service-based infrastructure.
 All right. The fast version, because we're 10 minutes in, apparently Max don't work every single time. Hi, my name is Adam Hitchcock. I work at Discussed as you just heard. You can follow me on Twitter or something. Shit. Okay, cool. Sweet. Let's just start off the bat. We're hiring, but we're also American. So if you want to leave Europe, come talk to me. But that would be crazy. So what am I going to talk about? First I'm going to address the issue of why did I lie to you last year? Then I say, what's a SOA? Why should you SOA? What are different data patterns in SOA? How does Discuss do it with both a legacy and Greenfield's example? And does that work for us? So last year my talk was entitled, How Discuss Does It When It Isn't Jango? And it was a place of order for services. It was a talk about a very similar topic, but saying specifically, hey, you don't have to use Jango for everything. It's not what the cool kids are doing. But why do I sit on Throne of Lies? That's the main question. So when I got back from EuroPython last year, my CTO basically said, hey, we're going to make a technology choice. And that technology choice is Jango. We're going to redouble our investment because we are a huge Jango shop. So we already are really good at it. And there's some really great reasons for it. Using standards across a company that are also known outside the company makes it super easy to hire. And there's a huge community. So we want to leverage all of that in both our legacy application and the platform we are moving to. So I kind of had to challenge my thinking about Jango and find some new ways to use it. So here is Take Two, is How Discuss Does It? Jango Edition. So if I ask you to raise your hands, who will raise their hands? All right. So about 60%. Cool. Who already knows what SOA stands for? Cool. That's like 120% because more of you raised. Were you correct? I guess. So it stands for Service Oriented Architecture, or if you're a British, Service Oriented Architecture. And I'm going to define what I think that means right now. So a SOA is something that I like to pronounce, TLAs, or TULAs, three letter acronyms. So a SOA is a system that you've architected in a specific way. That system should have discrete software components. Those are basically running programs. And those are called services. Those running programs should have simple, well-defined APIs or interfaces. And those are the APIs through which they're going to communicate. Then you can loosely coordinate or loosely federate these services to achieve some sort of business goal. In a SOA, there's two primary roles. There's a service provider and consumers. These are providers and consumers of data. And you can totally do both. That's cool. We're not going to mark you points for that. It's not a new idea at all. Doug McIlroy, who I think he worked at Bell Labs a long time ago, said this of the UNIX philosophy. The UNIX philosophy is write programs that do one thing and do it well. Write programs to work together and write programs to handle text streams, because that is a universal interface. And he was talking about UNIX pipes and programs talking over file descriptors on the same system. He was talking about network pipes and programs talking over file descriptors over the network. So they're really the exact same philosophy. And I think this still holds true today. It's also not necessarily a new idea in web systems. It's been people have been doing this for many years. You're probably already using services in your system. How many of you are Django app developers day to day? Sweet, a lot of you. Excellent. You're probably using Postgres if you're doing it right or MySQL if you're doing it also right. Or Redis. You're probably using queues in your system. Rabbit or Kafka. Kafka is not technically a queue, but it's basically a queue, so I put on a queues list. Or you're using external API. Who knows what Kafka is? Sweet. I will talk about that later. It's awesome. If someone asks me, it reminds me. So external APIs are also services. Everything is a service. Single page Java apps are great examples of consumer services usually. But if they do a post command back to your server, that's now producing data. So why should you do a SOA? This is basically my report card for later in the talk. I think that if you've done it right, you can have services written in any language. It doesn't matter what machines those services run on. And if those services have a strong API, it doesn't matter how they store or abstract that data internally. It's just the API that matters. Services should scale independently of each other. And they should also provide you easier testability, easier deployment. And because you've split your system up into well-defined chunks, it should be easier for individuals to maintain conceptual integrity or understanding of the system. So hopefully you can learn how to do all this things in the stock. I think that in a SOA, it's really important to not like when you're saying, okay, sweet, I want to make a SOA. Well, how? So I think that you have to understand the data patterns in your system. And I see two primary data patterns. There's kind of transactional data and asynchronous data. Transactional further is divided by into two areas, I think. There's are you doing transactional data against basically database models, some sort of straight representation of an object in your system, like a user? Then REST works really well. It's crud semantics against a model. If you need to do something where you're combining data and math or logic to say, hey, recommend something to me or can I do something, then RPC is a model that works really well. And that's a more procedural thing. If your response is the same every time, then REST might work well. That's kind of the way I think about it. Then asynchronous data maps really well to queuing systems or pub subsystems. It's really good when you have high CPU or long running workloads that you can't do in a web request cycle. So you also have to pick your APIs. There's two parts here. You need to pick what are the bytes that I'm going to send to the other person or other service and how am I going to send those bytes to the other service? How many people know what protobufs or thrifts or message pack or Avro is? So all of those are examples of non-JSON encodings of how do you put an object across the network in a format that other languages can read? So picking something from that list is super important because in order to have a heterogeneous environment, you need to have encoding that multiple languages can work. Pickle is not on this list. It just spent the last week removing pickle from something. So you also have to pick your transportation protocol. So HTTP, who knows what that is? Literally every person in this world. That was the lowest number of hands. So HTTP stands for Hypertext Transport Protocol since no one raised their hands. Or thrift provides a network binding as well. I like HTTP and JSON because they're super easy to do. Django does them really well out of the box. And HTTP is great if you actually read the spec or don't even read the spec. Read the Wikipedia page on the spec. You'll learn a lot. Things like the accept header. So if you can start with HTTP plus JSON and if your server respects the accept header, you can support multiple encodings when you go forward. You can say I accept application JSON or text JSON. In the future, you can say I accept application protobuf or protobuf version 2, whatever. And then HTTP works great too. It has things like keep alive as a connection option. Negotiating that initial connection a little expensive. But when you're going forward with HTTP, it's like if you're keeping the connection alive, it costs about the same as any other protocol. If you need to save the eight bytes you get from HTTP at thrift, I mean it's sweet or Facebook. But if not, it was. So transactional data stuff, REST, Django does it really well out of the box. That's basically what it's been doing for the last, I don't know how, like million years. But in recent years, we used to use Django forms and those suck and other Django things that are easy to make bad practices in. Django REST framework is a relatively recent framework that makes it super easy to keep your code organized and clean while providing RESTful access to your database models. It's really cool. Check it out. There's a kick to our campaign. It's awesome. Or you can roll your own API. There were definitely, we hit lots of performance tags with REST framework, but we also got around them very easily. Come talk to me about how I do awesome caching things and verging in that framework. I'm going to probably read a blog post about it. Cool. RPC is also, is the other transactional. Like I said, logic heavy APIs, recommendation services, authorization, authentication. I don't like RPC as a first stop because it's prone to over-specialization when you're building APIs. You make APIs that are really good at doing one thing. And if you're using, when you're building a platform, you want to make APIs that can support lots of things to be built on top of them. Sometimes you need a specialized API, but if you can do it in a generalized way, I think that's better. In other systems like Thrift and Zero RPC, which are super easy to use, that's their downfall. You can abstract away the fact that you're actually doing a remote procedure call. So I've seen code that it's lots of Thrift requests in a, like, seriously, like 50 of them. And they're like, why is it slow? Like because you just did 50 network requests and you waited for each one to finish before starting the next. So yeah, it's slow. HTTP, you usually know you're using HTTP because you imported HTTP lib or requests or something. So asynchronous, good for a high CPU, long running tasks. There's a couple of ways to do this. I think Django is a great entry point that our management commands, these are highly underutilized on the server. We have a pattern of while true, do work in our management command. And so we can basically do something forever. So in our ads system, we have to rank ads basically forever. It's just constant CPU. And so, and we never want to not do that. So Cron jobs don't really work very well because there's like a minute in between the jobs. Celery, how many people know what Celery is? Excellent. I don't have to explain this then. So yeah, Celery is a tasking system, great pattern for asynchronous work, post-save hook, and a Celery task. Celery can also use JSON, which is supporting our heterogeneous requirement. So we wrote something called Go Celery, our DevOps lead Matt Robinal did that, because we need to parse Celery tasks and go. So if you're doing that in pickle, you can't do that because nothing else can read pickle. So use JSON, something platform independent. And Celery is great for periodic tasks that you need to do like data import. Django also has been incredibly easy for us to run. It has incredibly well understood IO loop, which is wait for a command or outsource the IO loop itself to something else like UWISG. It has multiple entry points. WISG, it can be run in like engine X directly, UWISG, Apache, modWISG, and G unicorn, and a million other things. So it's great because those things handle the IO loop and WISG just has to respond to here's a dictionary of specific information. Management commands also, you're in control of the IO loop there. Celery tasks and Celery beat the same. They're just listening on cues to respond to information. So it's really great because basically each of these is a message passing interface at the end of the day. They either generate or respond to messages. So how do we do this at discuss? First we'll look at discuss web, which is a legacy product. It's a monolithic Django product. It is this many lines of code, 183,000. And that's just because we deleted a lot of code recently. It was great, like 67,000 lines or something. So it's over seven years old and we made a lot of bad decisions along the way. But some good ones too. When we're deploying this in our service oriented way, we deploy the entire code base. We decide it's impossible at its current state for us to break it up into multiple code bases. So we have to deploy the entire thing and that means we end up treating it like a library. We cluster machines by purpose. So at the end of the day, those will be our service. And by doing this, we kind of see the CPU patterns emerge. Makes scaling easier at the end of the day. You say, oh, we need a different kind of computer, high CPU for one thing and high memory for a different purpose. And then we route to these services or clusters based on host name. So we use varnish and HA proxy in order to route requests and then further based on the path to get them to the right machine. That plays into our data transparency point. And then when we're deploying, we do a three phase deploy. So there are multiple versions of the services out at any given time. It goes old version, old and new version, just new version. So we always have to make sure that whenever we upgrade anything, it will play nice in that route. To change the entry points on these clusters, we use different settings.py files. I haven't seen this done a ton in the wild, so I wanted to highlight it. But basically by using multiple settings.py files, we're able to drastically change the behavior of the discuss library. So we can have different URLs, different middlewares, different template request context. It's something interesting. Template request context, I don't think they're lazy loaded. So if you put something that you want in one of your templates in a global request context, if you're setting that up in a middleware or something, that's going to execute every time. And so even if you're using it like 80% of the time, 20% of the time, that's wasted cycles, wasted I.O. So having a, being able to separate that out per service. URL resolution is also O of N. So Django will go down your URLs list doing, does this match this regular expression? Does it match this regular expression? And so if you're most commonly used URLs are at the bottom of that list, you're going to have a bad time. So we found actually in our API service by reordering those, we found a 15% CPU saving because our most used URL was at the bottom. It was literally the last one. And simply putting it in about 15 others out of a list of several hundred to the top, we got huge savings. So do that. So I'm going to make a cool project that does that automatically. So two examples of services, we have our public API, which if you discuss.com. It's how we use our own products in the wild. And it's also how anyone that wants to integrate with it uses it. It has a ton of middleware. I think dozens of them. It has over 300 URL routes. And it has several things that automatically get loaded for templates. Our internal objects API. So this is an internal to our data center, very raw model representation. It has no middleware. It only has one URL route. And it has no, it basically has nothing. It's settings up high, it's like 12 lines long. So there's an enormous difference in speed between these two setups. Unfortunately, I do not have a graph to show that, but I can probably tweet one later. So did it work for us? Yeah, you know, it kind of works fine. It was still a large code base, and we got all of the problems of having a large code base with it still. Version conflicts still very problematic, whether it's a function or method that you're changing, you've got to integrate that over the entire code base still, or an external package upgrade. We've had lots of problems with Zookeeper, Kizu over the last two years because there were several incompatibilities there. And people were using different versions. So the conceptual integrity is basically still hard of this entire thing because you still have almost 200,000 lines of code. So on our report card, we did okay. We got our heterogeneous environment, and we utilized that. We have the go-sellerie tasks, data location transparency, independent scalability, and easier deployments. Something I didn't say about those deployments, we can actually deploy those services, the service clusters separately. So if we need to deploy just something for one of them, in an emergency, we can totally do that. We usually deploy the entire thing at the same time, though. So now the ad server. So this is a product that's about a year and a half old. And it was started as a Flask application, as I talked about in my previous talk last year, and other custom stuff. So in the last year, we've ported it from that to Django, and we made some decisions when we did that. We want to make sure that we use Django apps very well. Django best practices are best practices for a reason. They really help you architect your system in a good way. We want to leverage Django beyond WISGI. We want to use all those entry points, like the management commands and salary tasks. And we want to split up our code base very intentionally. We want to have one code base that can access the database. And I'm talking about model access, like using the ORM. And the other code base could only access that sort of information via the strong API. So that's just, we separate the services in that regard. But we do have multiple services that can access the database. So this is just some of the services that go into the ad system. We have our data API. This maps really well to REST framework. And it has some minimal RPC endpoints, because we need to do mass data export for other services. We basically just need an endpoint that says, everything, give me everything. And so then we have our ad serving API. This is basically a recommendations API, and it's an RPC endpoint. Then we have our asynchronous tasks, scoring and cache warming, keeping caches hot with recent information. And these are run using management commands in that while true do something. And then the ads data import service, which is salary and salary beat. These are either responding to new objects getting created and having to collect more information about them, or looking for what's changed in other systems on a batch process, like every five minutes, what was created in a different system, and import that information for scoring and other purposes. And this is how we organized it. There's the two on top have the ads stuff. Because we're leveraging lots of frameworks, we get to only have 11,000 lines of code compared to 180,000. And then on the bottom, those are the warming services and such. That's also, they're very close in lines of code, about 11,000. And this is what it looks like with boxes and arrows. So on the far left, you have JavaScript land. Then there's the internet. Then you kind of have a web serving, oh, I'm supposed to use the mouse. Okay. Boom. Now you have like a web serving layer here. And then you have all of our back end services. And these are again coded by, colored by code base, just so you can see how that goes. And here are the backing technologies for that. So backbone on the front end, we have our Django, Uwizgi, with Nginx layer, and then our backing stuff. So did this work? I think it was kind of amazing. Ah, it didn't play more than once. Ah, it's so good. Why is it not looping? Okay. So did it work? Okay. I think it worked really well. The only downside is the inverse downside. Well we had problems that small code bases have. It's hard to share code when you have lots of code bases. I hate things like Git subtree and stuff. Those are just going to hurt you. So we ended up making a third code base for sharing stuff. We use pip internally. So we run our own she shop. So we just deploy a package out there that is the ads core package for shared information. And that actually even helps us make even better code. Django best practices helped us a lot in the long term. I wish we actually were more strict on those. And we definitely made it easy to understand the entire system. Because it's everything is, there's so few lines of code and everything's encapsulated so well, it's easy to quickly add new things as long as they're being added as an individual item. Easy to test everything. Integration tests are more important because you need to integrate with other processes, not just other libraries. So you need to do a really good job of testing your external APIs. So the Django test client becomes way more important than I like it to be because it's slow. Then service APIs live a long time. So you're going to need to learn how to version them and support them for exam periods of time. I definitely have a couple of days every couple of months that's like, okay, I'm going to go find everyone else's code, where are they using the version of the API I wanted to delete and fix it for them. We get fast deployments. Our deployments are about two minutes. And then our testing is like a couple seconds right now per service. So it's pretty fast in general. And yeah, they scale independently. So we got all of our checkmarks on our report card. Is it a success for discuss? It's a knockout success. That was really bad. Okay. Yeah, it's easier to run overall. It's easier to understand and build new systems in this environment. We can hire an employee that never has to see legacy code and I'm jealous of them. But and it's easier to not break existing systems because you're not touching them. You can't accidentally break something that you're not committing code to. Well, okay, you can. But it's harder. So as a roundup, do one thing and do it well. You want to examine your data patterns upfront and then look based on that data pattern, you need to make some decisions. Those are going to be API decisions around protocol, transport, and what methodology you're using to access that data. And then Django has multiple entry points. You should definitely use them. It's not just a whiskey machine. And then do one thing and do it well. That is the one line from the talk you should remember. Here's some links. Go support Django REST framework. It's got a Kickstarter. It's got a ton of money right now. I don't even, I don't know if you even want more money. But you know, it's a great thing. Support it. Here's our go-selleries stuff. Django best practices. I like Lincoln Loop. They're out of Portland, I think. There are some awesome people. And they are really good thought leaders in just how to do Django development in general. And the Unix philosophy. It's a great Wikipedia page. So if that was interesting to you and you hate Europe, come to San Francisco. Yeah. Questions which I already have one of. About three phases deployment. Oh yeah. In case of the service which uses a database directly, that one you mentioned. Oh yeah. How do you manage that? Because actually database is only at one version, I guess. Yeah. So we basically roll forward. So when we want to do something to upgrade our database, we change the code. We change our Django models. We make the SQL that will do that upgrade. We run that upgrade. When you're adding a new column type, that's okay. Like if you're adding a new column, sweet. Django will run with additional columns. It will ignore that. If you're changing a column type, well, you can't do that. We exceeded the ID variable for one of the size of ID in one of our tables. So now we're all like big ID or big ID, whatever it is. Was it big sequential? Some post-credits thing. I don't know how to SQL. But so the point is you can't do that. You cannot change your models. It's very expensive and basically impossible. But you can totally add to them. So it goes to data planning. But also because it's behind the API, you can always abstract that. So we have models that are, it has one model. It has its tables. And then it kind of has adjunct models that are meta information for that. And as a consumer of the API, I don't even care because I didn't even know that that exists. It's up to that service to manage all of that. Right, just a question. What do you do for logging and monitoring? Because you already have lots of things in your SOA and lots of external dependencies. And I dreads when something happens in Celery or RabbitMQ and I have to go to the manual and start to read again. And I have no idea what's going on. So what do you do for that? So I'm not on the ops team. But I have a little bit of insight into how that works because we use the same infrastructure to manage our applications. And we basically use statsD for everything. Who knows what statsD is? We're still in the participation phase. Okay, cool. StatsD is basically a daemon that runs locally or anywhere because it's a service. And you can talk UDP to it and it will basically over a period aggregate information and then beacon that out to somewhere else. What's usually called a collection service. And then that collection service will take everything as collected and tell Graphite or something about that. And so we use Graphite. We use... Oh, I forgot the thing that actually does the alerting. Nagios? No, can't remember. But we basically have Graphite stats. We alert on thresholds on those stats and we have them for hardware and software. And so we have thousands and thousands and thousands of different metrics being tracked. And by aggregating the correct ones, you can have a pretty good intuition on the system. I want to answer one question I was asked last night because I thought it was particularly interesting, which I was talking to Tom Christie and he was saying, what was the most challenging part about designing your new system? And I was surprised when I found it was actually just the software architecture. It was making sure that we don't write that spaghetti code because when you're taking an idea and implementing it, you can spread that idea across your entire application or you can encapsulate it. So using Django apps in a Django project as their own mini services with their own APIs at a software level, that's like doing that well was our hardest part. And his response was really, I thought I would say not design it so it doesn't go horribly, horribly wrong when it breaks. And that was actually relatively easy because it's everything is message passing. And something breaks, it stops sending messages. So that means everything turns off instead of blows up, which is, I think, it's still a failure mode, which is bad, but it's a better failure mode. It's not the, oh, we need to bring more hardware online to handle the load. It's just kind of a load dies. So those are the two things. I also have questions for you, which is how do you guys like RPC? And we can talk about this outside later. But do you realize how do you do maintainable RPC? How do you guys do service discovery? I like DNS because I also like HTTP. And what do you guys think on the many code bases versus one code base thing? I know large companies like Google and Facebook love it and I'm split. So I'm just curious, like, what are your thoughts? Okay, I'm so real at the time. Thank you again, Adam. Thank you. Thank you.
 All right.  The fast version, because we're 10 minutes in, apparently Max don't work every single  time.  Hi, my name is Adam Hitchcock.  I work at Discussed as you just heard.  You can follow me on Twitter or something.  Shit.  Okay, cool.  Sweet.  Let's just start off the bat.  We're hiring, but we're also American.  So if you want to leave Europe, come talk to me.  But that would be crazy.  So what am I going to talk about?  First I'm going to address the issue of why did I lie to you last year?  Then I say, what's a SOA?  Why should you SOA?  What are different data patterns in SOA?  How does Discuss do it with both a legacy and Greenfield's example?  And does that work for us?  So last year my talk was entitled, How Discuss Does It When It Isn't Jango?  And it was a place of order for services.  It was a talk about a very similar topic, but saying specifically, hey, you don't have  to use Jango for everything.  It's not what the cool kids are doing.  But why do I sit on Throne of Lies?  That's the main question.  So when I got back from EuroPython last year, my CTO basically said, hey, we're going to  make a technology choice.  And that technology choice is Jango.  We're going to redouble our investment because we are a huge Jango shop.  So we already are really good at it.  And there's some really great reasons for it.  Using standards across a company that are also known outside the company makes it super  easy to hire.  And there's a huge community.  So we want to leverage all of that in both our legacy application and the platform we  are moving to.  So I kind of had to challenge my thinking about Jango and find some new ways to use it.  So here is Take Two, is How Discuss Does It?  Jango Edition.  So if I ask you to raise your hands, who will raise their hands?  All right.  So about 60%.  Cool.  Who already knows what SOA stands for?  Cool.  That's like 120% because more of you raised.  Were you correct?  I guess.  So it stands for Service Oriented Architecture, or if you're a British, Service Oriented  Architecture.  And I'm going to define what I think that means right now.  So a SOA is something that I like to pronounce, TLAs, or TULAs, three letter acronyms.  So a SOA is a system that you've architected in a specific way.  That system should have discrete software components.  Those are basically running programs.  And those are called services.  Those running programs should have simple, well-defined APIs or interfaces.  And those are the APIs through which they're going to communicate.  Then you can loosely coordinate or loosely federate these services to achieve some sort  of business goal.  In a SOA, there's two primary roles.  There's a service provider and consumers.  These are providers and consumers of data.  And you can totally do both.  That's cool.  We're not going to mark you points for that.  It's not a new idea at all.  Doug McIlroy, who I think he worked at Bell Labs a long time ago, said this of the UNIX  philosophy.  The UNIX philosophy is write programs that do one thing and do it well.  Write programs to work together and write programs to handle text streams, because that  is a universal interface.  And he was talking about UNIX pipes and programs talking over file descriptors on the same  system.  He was talking about network pipes and programs talking over file descriptors over the network.  So they're really the exact same philosophy.  And I think this still holds true today.  It's also not necessarily a new idea in web systems.  It's been people have been doing this for many years.  You're probably already using services in your system.  How many of you are Django app developers day to day?  Sweet, a lot of you.  Excellent.  You're probably using Postgres if you're doing it right or MySQL if you're doing it  also right.  Or Redis.  You're probably using queues in your system.  Rabbit or Kafka.  Kafka is not technically a queue, but it's basically a queue, so I put on a queues list.  Or you're using external API.  Who knows what Kafka is?  Sweet.  I will talk about that later.  It's awesome.  If someone asks me, it reminds me.  So external APIs are also services.  Everything is a service.  Single page Java apps are great examples of consumer services usually.  But if they do a post command back to your server, that's now producing data.  So why should you do a SOA?  This is basically my report card for later in the talk.  I think that if you've done it right, you can have services written in any language.  It doesn't matter what machines those services run on.  And if those services have a strong API, it doesn't matter how they store or abstract  that data internally.  It's just the API that matters.  Services should scale independently of each other.  And they should also provide you easier testability, easier deployment.  And because you've split your system up into well-defined chunks, it should be easier for  individuals to maintain conceptual integrity or understanding of the system.  So hopefully you can learn how to do all this things in the stock.  I think that in a SOA, it's really important to not like when you're saying, okay, sweet,  I want to make a SOA.  Well, how?  So I think that you have to understand the data patterns in your system.  And I see two primary data patterns.  There's kind of transactional data and asynchronous data.  Transactional further is divided by into two areas, I think.  There's are you doing transactional data against basically database models, some sort of straight  representation of an object in your system, like a user?  Then REST works really well.  It's crud semantics against a model.  If you need to do something where you're combining data and math or logic to say, hey, recommend  something to me or can I do something, then RPC is a model that works really well.  And that's a more procedural thing.  If your response is the same every time, then REST might work well.  That's kind of the way I think about it.  Then asynchronous data maps really well to queuing systems or pub subsystems.  It's really good when you have high CPU or long running workloads that you can't do in  a web request cycle.  So you also have to pick your APIs.  There's two parts here.  You need to pick what are the bytes that I'm going to send to the other person or other  service and how am I going to send those bytes to the other service?  How many people know what protobufs or thrifts or message pack or Avro is?  So all of those are examples of non-JSON encodings of how do you put an object across the network  in a format that other languages can read?  So picking something from that list is super important because in order to have a heterogeneous  environment, you need to have encoding that multiple languages can work.  Pickle is not on this list.  It just spent the last week removing pickle from something.  So you also have to pick your transportation protocol.  So HTTP, who knows what that is?  Literally every person in this world.  That was the lowest number of hands.  So HTTP stands for Hypertext Transport Protocol since no one raised their hands.  Or thrift provides a network binding as well.  I like HTTP and JSON because they're super easy to do.  Django does them really well out of the box.  And HTTP is great if you actually read the spec or don't even read the spec.  Read the Wikipedia page on the spec.  You'll learn a lot.  Things like the accept header.  So if you can start with HTTP plus JSON and if your server respects the accept header,  you can support multiple encodings when you go forward.  You can say I accept application JSON or text JSON.  In the future, you can say I accept application protobuf or protobuf version 2, whatever.  And then HTTP works great too.  It has things like keep alive as a connection option.  Negotiating that initial connection a little expensive.  But when you're going forward with HTTP, it's like if you're keeping the connection alive,  it costs about the same as any other protocol.  If you need to save the eight bytes you get from HTTP at thrift, I mean it's sweet or  Facebook.  But if not, it was.  So transactional data stuff, REST, Django does it really well out of the box.  That's basically what it's been doing for the last, I don't know how, like million years.  But in recent years, we used to use Django forms and those suck and other Django things  that are easy to make bad practices in.  Django REST framework is a relatively recent framework that makes it super easy to keep  your code organized and clean while providing RESTful access to your database models.  It's really cool.  Check it out.  There's a kick to our campaign.  It's awesome.  Or you can roll your own API.  There were definitely, we hit lots of performance tags with REST framework, but we also got  around them very easily.  Come talk to me about how I do awesome caching things and verging in that framework.  I'm going to probably read a blog post about it.  Cool.  RPC is also, is the other transactional.  Like I said, logic heavy APIs, recommendation services, authorization, authentication.  I don't like RPC as a first stop because it's prone to over-specialization when you're  building APIs.  You make APIs that are really good at doing one thing.  And if you're using, when you're building a platform, you want to make APIs that can  support lots of things to be built on top of them.  Sometimes you need a specialized API, but if you can do it in a generalized way, I think  that's better.  In other systems like Thrift and Zero RPC, which are super easy to use, that's their  downfall.  You can abstract away the fact that you're actually doing a remote procedure call.  So I've seen code that it's lots of Thrift requests in a, like, seriously, like 50 of  them.  And they're like, why is it slow?  Like because you just did 50 network requests and you waited for each one to finish before  starting the next.  So yeah, it's slow.  HTTP, you usually know you're using HTTP because you imported HTTP lib or requests or something.  So asynchronous, good for a high CPU, long running tasks.  There's a couple of ways to do this.  I think Django is a great entry point that our management commands, these are highly  underutilized on the server.  We have a pattern of while true, do work in our management command.  And so we can basically do something forever.  So in our ads system, we have to rank ads basically forever.  It's just constant CPU.  And so, and we never want to not do that.  So Cron jobs don't really work very well because there's like a minute in between the jobs.  Celery, how many people know what Celery is?  Excellent.  I don't have to explain this then.  So yeah, Celery is a tasking system, great pattern for asynchronous work, post-save hook,  and a Celery task.  Celery can also use JSON, which is supporting our heterogeneous requirement.  So we wrote something called Go Celery, our DevOps lead Matt Robinal did that, because  we need to parse Celery tasks and go.  So if you're doing that in pickle, you can't do that because nothing else can read pickle.  So use JSON, something platform independent.  And Celery is great for periodic tasks that you need to do like data import.  Django also has been incredibly easy for us to run.  It has incredibly well understood IO loop, which is wait for a command or outsource the  IO loop itself to something else like UWISG.  It has multiple entry points.  WISG, it can be run in like engine X directly, UWISG, Apache, modWISG, and G unicorn, and  a million other things.  So it's great because those things handle the IO loop and WISG just has to respond to  here's a dictionary of specific information.  Management commands also, you're in control of the IO loop there.  Celery tasks and Celery beat the same.  They're just listening on cues to respond to information.  So it's really great because basically each of these is a message passing interface at  the end of the day.  They either generate or respond to messages.  So how do we do this at discuss?  First we'll look at discuss web, which is a legacy product.  It's a monolithic Django product.  It is this many lines of code, 183,000.  And that's just because we deleted a lot of code recently.  It was great, like 67,000 lines or something.  So it's over seven years old and we made a lot of bad decisions along the way.  But some good ones too.  When we're deploying this in our service oriented way, we deploy the entire code base.  We decide it's impossible at its current state for us to break it up into multiple code  bases.  So we have to deploy the entire thing and that means we end up treating it like a library.  We cluster machines by purpose.  So at the end of the day, those will be our service.  And by doing this, we kind of see the CPU patterns emerge.  Makes scaling easier at the end of the day.  You say, oh, we need a different kind of computer, high CPU for one thing and high memory for  a different purpose.  And then we route to these services or clusters based on host name.  So we use varnish and HA proxy in order to route requests and then further based on the  path to get them to the right machine.  That plays into our data transparency point.  And then when we're deploying, we do a three phase deploy.  So there are multiple versions of the services out at any given time.  It goes old version, old and new version, just new version.  So we always have to make sure that whenever we upgrade anything, it will play nice in  that route.  To change the entry points on these clusters, we use different settings.py files.  I haven't seen this done a ton in the wild, so I wanted to highlight it.  But basically by using multiple settings.py files, we're able to drastically change the  behavior of the discuss library.  So we can have different URLs, different middlewares, different template request context.  It's something interesting.  Template request context, I don't think they're lazy loaded.  So if you put something that you want in one of your templates in a global request context,  if you're setting that up in a middleware or something, that's going to execute every  time.  And so even if you're using it like 80% of the time, 20% of the time, that's wasted cycles,  wasted I.O.  So having a, being able to separate that out per service.  URL resolution is also O of N. So Django will go down your URLs list doing, does this match  this regular expression?  Does it match this regular expression?  And so if you're most commonly used URLs are at the bottom of that list, you're going to  have a bad time.  So we found actually in our API service by reordering those, we found a 15% CPU saving  because our most used URL was at the bottom.  It was literally the last one.  And simply putting it in about 15 others out of a list of several hundred to the top, we  got huge savings.  So do that.  So I'm going to make a cool project that does that automatically.  So two examples of services, we have our public API, which if you discuss.com.  It's how we use our own products in the wild.  And it's also how anyone that wants to integrate with it uses it.  It has a ton of middleware.  I think dozens of them.  It has over 300 URL routes.  And it has several things that automatically get loaded for templates.  Our internal objects API.  So this is an internal to our data center, very raw model representation.  It has no middleware.  It only has one URL route.  And it has no, it basically has nothing.  It's settings up high, it's like 12 lines long.  So there's an enormous difference in speed between these two setups.  Unfortunately, I do not have a graph to show that, but I can probably tweet one later.  So did it work for us?  Yeah, you know, it kind of works fine.  It was still a large code base, and we got all of the problems of having a large code  base with it still.  Version conflicts still very problematic, whether it's a function or method that you're  changing, you've got to integrate that over the entire code base still, or an external  package upgrade.  We've had lots of problems with Zookeeper, Kizu over the last two years because there  were several incompatibilities there.  And people were using different versions.  So the conceptual integrity is basically still hard of this entire thing because you still  have almost 200,000 lines of code.  So on our report card, we did okay.  We got our heterogeneous environment, and we utilized that.  We have the go-sellerie tasks, data location transparency, independent scalability, and  easier deployments.  Something I didn't say about those deployments, we can actually deploy those services, the  service clusters separately.  So if we need to deploy just something for one of them, in an emergency, we can totally  do that.  We usually deploy the entire thing at the same time, though.  So now the ad server.  So this is a product that's about a year and a half old.  And it was started as a Flask application, as I talked about in my previous talk last  year, and other custom stuff.  So in the last year, we've ported it from that to Django, and we made some decisions  when we did that.  We want to make sure that we use Django apps very well.  Django best practices are best practices for a reason.  They really help you architect your system in a good way.  We want to leverage Django beyond WISGI.  We want to use all those entry points, like the management commands and salary tasks.  And we want to split up our code base very intentionally.  We want to have one code base that can access the database.  And I'm talking about model access, like using the ORM.  And the other code base could only access that sort of information via the strong API.  So that's just, we separate the services in that regard.  But we do have multiple services that can access the database.  So this is just some of the services that go into the ad system.  We have our data API.  This maps really well to REST framework.  And it has some minimal RPC endpoints, because we need to do mass data export for other services.  We basically just need an endpoint that says, everything, give me everything.  And so then we have our ad serving API.  This is basically a recommendations API, and it's an RPC endpoint.  Then we have our asynchronous tasks, scoring and cache warming, keeping caches hot with  recent information.  And these are run using management commands in that while true do something.  And then the ads data import service, which is salary and salary beat.  These are either responding to new objects getting created and having to collect more  information about them, or looking for what's changed in other systems on a batch process,  like every five minutes, what was created in a different system, and import that information  for scoring and other purposes.  And this is how we organized it.  There's the two on top have the ads stuff.  Because we're leveraging lots of frameworks, we get to only have 11,000 lines of code compared  to 180,000.  And then on the bottom, those are the warming services and such.  That's also, they're very close in lines of code, about 11,000.  And this is what it looks like with boxes and arrows.  So on the far left, you have JavaScript land.  Then there's the internet.  Then you kind of have a web serving, oh, I'm supposed to use the mouse.  Okay.  Boom.  Now you have like a web serving layer here.  And then you have all of our back end services.  And these are again coded by, colored by code base, just so you can see how that goes.  And here are the backing technologies for that.  So backbone on the front end, we have our Django, Uwizgi, with Nginx layer, and then  our backing stuff.  So did this work?  I think it was kind of amazing.  Ah, it didn't play more than once.  Ah, it's so good.  Why is it not looping?  Okay.  So did it work?  Okay.  I think it worked really well.  The only downside is the inverse downside.  Well we had problems that small code bases have.  It's hard to share code when you have lots of code bases.  I hate things like Git subtree and stuff.  Those are just going to hurt you.  So we ended up making a third code base for sharing stuff.  We use pip internally.  So we run our own she shop.  So we just deploy a package out there that is the ads core package for shared information.  And that actually even helps us make even better code.  Django best practices helped us a lot in the long term.  I wish we actually were more strict on those.  And we definitely made it easy to understand the entire system.  Because it's everything is, there's so few lines of code and everything's encapsulated  so well, it's easy to quickly add new things as long as they're being added as an individual  item.  Easy to test everything.  Integration tests are more important because you need to integrate with other processes,  not just other libraries.  So you need to do a really good job of testing your external APIs.  So the Django test client becomes way more important than I like it to be because it's  slow.  Then service APIs live a long time.  So you're going to need to learn how to version them and support them for exam periods of time.  I definitely have a couple of days every couple of months that's like, okay, I'm going to  go find everyone else's code, where are they using the version of the API I wanted to delete  and fix it for them.  We get fast deployments.  Our deployments are about two minutes.  And then our testing is like a couple seconds right now per service.  So it's pretty fast in general.  And yeah, they scale independently.  So we got all of our checkmarks on our report card.  Is it a success for discuss?  It's a knockout success.  That was really bad.  Okay.  Yeah, it's easier to run overall.  It's easier to understand and build new systems in this environment.  We can hire an employee that never has to see legacy code and I'm jealous of them.  But and it's easier to not break existing systems because you're not touching them.  You can't accidentally break something that you're not committing code to.  Well, okay, you can.  But it's harder.  So as a roundup, do one thing and do it well.  You want to examine your data patterns upfront and then look based on that data pattern,  you need to make some decisions.  Those are going to be API decisions around protocol, transport, and what methodology  you're using to access that data.  And then Django has multiple entry points.  You should definitely use them.  It's not just a whiskey machine.  And then do one thing and do it well.  That is the one line from the talk you should remember.  Here's some links.  Go support Django REST framework.  It's got a Kickstarter.  It's got a ton of money right now.  I don't even, I don't know if you even want more money.  But you know, it's a great thing.  Support it.  Here's our go-selleries stuff.  Django best practices.  I like Lincoln Loop.  They're out of Portland, I think.  There are some awesome people.  And they are really good thought leaders in just how to do Django development in general.  And the Unix philosophy.  It's a great Wikipedia page.  So if that was interesting to you and you hate Europe, come to San Francisco.  Yeah.  Questions which I already have one of.  About three phases deployment.  Oh yeah.  In case of the service which uses a database directly, that one you mentioned.  Oh yeah.  How do you manage that?  Because actually database is only at one version, I guess.  Yeah.  So we basically roll forward.  So when we want to do something to upgrade our database, we change the code.  We change our Django models.  We make the SQL that will do that upgrade.  We run that upgrade.  When you're adding a new column type, that's okay.  Like if you're adding a new column, sweet.  Django will run with additional columns.  It will ignore that.  If you're changing a column type, well, you can't do that.  We exceeded the ID variable for one of the size of ID in one of our tables.  So now we're all like big ID or big ID, whatever it is.  Was it big sequential?  Some post-credits thing.  I don't know how to SQL.  But so the point is you can't do that.  You cannot change your models.  It's very expensive and basically impossible.  But you can totally add to them.  So it goes to data planning.  But also because it's behind the API, you can always abstract that.  So we have models that are, it has one model.  It has its tables.  And then it kind of has adjunct models that are meta information for that.  And as a consumer of the API, I don't even care because I didn't even know that that  exists.  It's up to that service to manage all of that.  Right, just a question.  What do you do for logging and monitoring?  Because you already have lots of things in your SOA and lots of external dependencies.  And I dreads when something happens in Celery or RabbitMQ and I have to go to the manual  and start to read again.  And I have no idea what's going on.  So what do you do for that?  So I'm not on the ops team.  But I have a little bit of insight into how that works because we use the same infrastructure  to manage our applications.  And we basically use statsD for everything.  Who knows what statsD is?  We're still in the participation phase.  Okay, cool.  StatsD is basically a daemon that runs locally or anywhere because it's a service.  And you can talk UDP to it and it will basically over a period aggregate information and then  beacon that out to somewhere else.  What's usually called a collection service.  And then that collection service will take everything as collected and tell Graphite  or something about that.  And so we use Graphite.  We use...  Oh, I forgot the thing that actually does the alerting.  Nagios?  No, can't remember.  But we basically have Graphite stats.  We alert on thresholds on those stats and we have them for hardware and software.  And so we have thousands and thousands and thousands of different metrics being tracked.  And by aggregating the correct ones, you can have a pretty good intuition on the system.  I want to answer one question I was asked last night because I thought it was particularly  interesting, which I was talking to Tom Christie and he was saying, what was the most challenging  part about designing your new system?  And I was surprised when I found it was actually just the software architecture.  It was making sure that we don't write that spaghetti code because when you're taking  an idea and implementing it, you can spread that idea across your entire application or  you can encapsulate it.  So using Django apps in a Django project as their own mini services with their own APIs  at a software level, that's like doing that well was our hardest part.  And his response was really, I thought I would say not design it so it doesn't go horribly,  horribly wrong when it breaks.  And that was actually relatively easy because it's everything is message passing.  And something breaks, it stops sending messages.  So that means everything turns off instead of blows up, which is, I think, it's still  a failure mode, which is bad, but it's a better failure mode.  It's not the, oh, we need to bring more hardware online to handle the load.  It's just kind of a load dies.  So those are the two things.  I also have questions for you, which is how do you guys like RPC?  And we can talk about this outside later.  But do you realize how do you do maintainable RPC?  How do you guys do service discovery?  I like DNS because I also like HTTP.  And what do you guys think on the many code bases versus one code base thing?  I know large companies like Google and Facebook love it and I'm split.  So I'm just curious, like, what are your thoughts?  Okay, I'm so real at the time.  Thank you again, Adam.  Thank you.  Thank you.
adam_-_How_Disqus_is_using_Django_as_the_basis_of_our_Service_Oriented_Architecture_0_1019.png adam_-_How_Disqus_is_using_Django_as_the_basis_of_our_Service_Oriented_Architecture_1020_1067.png adam_-_How_Disqus_is_using_Django_as_the_basis_of_our_Service_Oriented_Architecture_1068_1547.png adam_-_How_Disqus_is_using_Django_as_the_basis_of_our_Service_Oriented_Architecture_1548_1955.png adam_-_How_Disqus_is_using_Django_as_the_basis_of_our_Service_Oriented_Architecture_1956_3167.png adam_-_How_Disqus_is_using_Django_as_the_basis_of_our_Service_Oriented_Architecture_3168_3587.png adam_-_How_Disqus_is_using_Django_as_the_basis_of_our_Service_Oriented_Architecture_3588_3635.png adam_-_How_Disqus_is_using_Django_as_the_basis_of_our_Service_Oriented_Architecture_3636_3851.png adam_-_How_Disqus_is_using_Django_as_the_basis_of_our_Service_Oriented_Architecture_3852_5855.png adam_-_How_Disqus_is_using_Django_as_the_basis_of_our_Service_Oriented_Architecture_5856_6347.png adam_-_How_Disqus_is_using_Django_as_the_basis_of_our_Service_Oriented_Architecture_6348_8063.png adam_-_How_Disqus_is_using_Django_as_the_basis_of_our_Service_Oriented_Architecture_8064_8435.png adam_-_How_Disqus_is_using_Django_as_the_basis_of_our_Service_Oriented_Architecture_8436_8555.png adam_-_How_Disqus_is_using_Django_as_the_basis_of_our_Service_Oriented_Architecture_8556_8723.png adam_-_How_Disqus_is_using_Django_as_the_basis_of_our_Service_Oriented_Architecture_8724_13883.png adam_-_How_Disqus_is_using_Django_as_the_basis_of_our_Service_Oriented_Architecture_13884_13943.png adam_-_How_Disqus_is_using_Django_as_the_basis_of_our_Service_Oriented_Architecture_13944_14003.png adam_-_How_Disqus_is_using_Django_as_the_basis_of_our_Service_Oriented_Architecture_14004_14123.png adam_-_How_Disqus_is_using_Django_as_the_basis_of_our_Service_Oriented_Architecture_14124_14687.png adam_-_How_Disqus_is_using_Django_as_the_basis_of_our_Service_Oriented_Architecture_14688_16703.png adam_-_How_Disqus_is_using_Django_as_the_basis_of_our_Service_Oriented_Architecture_16704_19499.png adam_-_How_Disqus_is_using_Django_as_the_basis_of_our_Service_Oriented_Architecture_19500_22391.png adam_-_How_Disqus_is_using_Django_as_the_basis_of_our_Service_Oriented_Architecture_22392_24335.png adam_-_How_Disqus_is_using_Django_as_the_basis_of_our_Service_Oriented_Architecture_24336_26183.png adam_-_How_Disqus_is_using_Django_as_the_basis_of_our_Service_Oriented_Architecture_26184_26315.png adam_-_How_Disqus_is_using_Django_as_the_basis_of_our_Service_Oriented_Architecture_26316_27071.png adam_-_How_Disqus_is_using_Django_as_the_basis_of_our_Service_Oriented_Architecture_27072_27191.png adam_-_How_Disqus_is_using_Django_as_the_basis_of_our_Service_Oriented_Architecture_27192_31931.png adam_-_How_Disqus_is_using_Django_as_the_basis_of_our_Service_Oriented_Architecture_31932_32687.png adam_-_How_Disqus_is_using_Django_as_the_basis_of_our_Service_Oriented_Architecture_32688_35951.png adam_-_How_Disqus_is_using_Django_as_the_basis_of_our_Service_Oriented_Architecture_35952_36035.png adam_-_How_Disqus_is_using_Django_as_the_basis_of_our_Service_Oriented_Architecture_36036_36323.png adam_-_How_Disqus_is_using_Django_as_the_basis_of_our_Service_Oriented_Architecture_36324_36947.png adam_-_How_Disqus_is_using_Django_as_the_basis_of_our_Service_Oriented_Architecture_36948_38507.png adam_-_How_Disqus_is_using_Django_as_the_basis_of_our_Service_Oriented_Architecture_38508_38603.png adam_-_How_Disqus_is_using_Django_as_the_basis_of_our_Service_Oriented_Architecture_38604_38723.png adam_-_How_Disqus_is_using_Django_as_the_basis_of_our_Service_Oriented_Architecture_38724_39803.png adam_-_How_Disqus_is_using_Django_as_the_basis_of_our_Service_Oriented_Architecture_39804_43919.png adam_-_How_Disqus_is_using_Django_as_the_basis_of_our_Service_Oriented_Architecture_43920_45887.png adam_-_How_Disqus_is_using_Django_as_the_basis_of_our_Service_Oriented_Architecture_45888_46459.png