But how do you know your mock is valid? Verified fakes of web services
CC Attribution - NonCommercial - ShareAlike 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this
Dangoor, Adam
But how do you know your mock is valid? Verified fakes of web services If your code calls a third party service then you may want to test that your code works but you don't want to call the service in your tests. It may be expensive, slow or impossible to call that service. For example, if you are making a Slack bot, you want to create tests which don't make calls across the network to Slack. One approach is to create a mock of that service. Our tests can now run quickly, cheaply and reliably. But if we copy the service incorrectly, or if the service changes, our tests will pass while our code does not work. Verified fakes solve this problem. You can write tests which confirm that your mock is an accurate representation of the service being mocked. Those tests can be a small subset of your test suite and they can be run periodically, to verify the validity of the many tests which use the mock. This talk will follow the example of VWS-Python, a verified fake for a proprietary web service. It will discuss the practicalities of creating such a fake and it will focus on the trade-offs, tooling and approaches involved. By the end of this talk the audience will understand how to tie together pytest, Travis CI, requests and Responses to create a verified fake. The talk is aimed at people who have an interest in writing correct software. It is assumed that the audience is familiar with basic testing techniques
 Thank you and thank you everyone for coming. So my name is Adam Dengore. I work at a company called Misesphere, building an operating system for data centers. But last year I was working on something quite different. I was working on the back end of an iPhone app. Now what you do is as a user you would take a photo of a wine label with your phone and the app would tell you all kinds of details about that wine. Or at least it was something like that. I'm going to protect my NDA here today. And our app was a Flask app. Now if you don't know Flask it's a really simple web framework. And it looks something like this. Now a really cool thing about Flask is that it provides, let me give a second, a Verkzeug test client. I hope I got that right. And what that means is that you can make requests against an in-memory application and you can get response objects which you can inspect. So if we look at this test here, it kind of looks like we've made an HTTP request. But actually everything's being done in memory in our test. Now in our wine recognition app we use something as well called Vaforia web services. And basically what Vaforia web services is, is it's a tool that lets you upload a whole bunch of images, let's say in our case images of wine labels. And then when a user uploaded a photo to us, well we could send that image to Vaforia and then Vaforia would tell us which one of our previously uploaded images, their photo most closely matched. And then what we could do is we could fetch details about that wine from our database and tell the user's details about the wine, like how much it should cost, how well rated it was from, exactly where it's from, that kind of thing. But when we built our prototype, well we kept finding loads of problems, loads of bugs. And in particular those bugs came from assumptions that we'd made about Vaforia which weren't quite right. Often that came from reading their documentation and trusting that it was truthful and full and you can't always make those assumptions. And so what we wanted to do, we wanted to add tests for our matching workflow. And that matching workflow of course used Vaforia and we wanted those tests to be in our existing test suite. Now Vaforia here was accessed over HTTP. And that's what I'm going to focus on today. But the general ideas really are on specific to HTTP because you might want to test code, let's say, that uses a database for local storage. Or you might want to test a deployment workflow which uses Docker. Or maybe you even want to test code which uses Amazon S3 or some other cloud storage as a storage backend. Now we were lucky we had a very clear idea of what we wanted our first test to be. I know this is quite a lot of code to have on a slide. But simply what we wanted to test was that if a user uploaded a photo of a wine label which matched a photo that we had already added, well then they would get details about that wine. So I wrote a test that looked a little bit like this. I had two wines here. Add wine, let's say it adds it to our database, but it also uploads it to Vaforia. And then I check that I get the right one back when I query the match function. That much function uses Vaforia on the backend. Now with some third party tools, maybe even some of the ones I mentioned like Docker, you might be totally fine, totally cool to call that real tool in your test suite. But when we call Vaforia in our tests, we actually hit some problems. Now first of all, we were at the mercy of the network. And what that meant is when our CRE system had a little network glitch, well then our whole test suite would fail. Because our tests made HTTP requests against the Internet. And we didn't know if those failures were because of the network failure or because there was some kind of flakiness in our code. But also we were at the mercy of Vaforia. So similarly, when Vaforia went down or went down temporarily, our test suite would fail. It really does slow down development if you're constantly worrying have you made a mistake or is it on their end. Now say you're using a real service like S3. S3 might be pretty stable, probably even more stable than your software. So you might not have to worry too much about flakiness. But S3 charges you per megabyte use. So if you want to use it in your test suite, it might actually become really expensive to run your tests. You might have to pay per megabyte and just spend quite a lot of money. And another problem that you might run into is resource limits. This is definitely something that I've hit. A lot of services have resource limits, a certain number of requests that your account can make. And so if you call something in your test suite very heavily, especially if you're doing performance benchmarking, you're making a load of calls, well then you might hit those resource limits and you can't run your tests anymore and you're pretty much blocked on development. And even when those things weren't problems, everything was really slow. So the 4A is quite advanced software. It does a lot of processing magic so that it can do the image matching. And that means that after you've uploaded an image, well, it takes a few minutes until that image can be matched. That's totally reasonable. I don't think that I could really expect them to do it instantly. But in our test suite, well, I didn't really want to have to wait a few minutes to know if our get match code worked. So we call these tests like the one that I showed you before integration tests because, well, they tested the integration of our software with Vaforius. I think a lot of people get confused about the terminology. Some people call these things acceptance tests or end-to-end tests. But I think we can agree that they're high-level tests. And they were definitely useful. They really did help us track down some bugs. But we also wanted unit tests because unit tests give us a lot of benefits over integration tests. And in particular, they tell us if our code calls Vaforia correctly in this case, even when Vaforia is down. And unit tests are also really small in scope. And what that means is, well, let's say one fails, not all the time, but often you know exactly which part of your code failed. And if you change that bit of your code to make the unit test pass, well, that can be a small isolated change. And when you've got unit tests that run quickly and are small, well, you can even use some tools maybe like Hypothesis to generate a whole bunch of unit tests. So what we want to do, we want to turn a code base which can currently be tested only by integration tests into one which can also be tested with unit tests. And one way that some people achieve this is by using mocks. Now, roughly, a mock is some code which provides the same interface as something that your code calls, but it reduces or it removes some cost. And in this case, the main costs that we cared about, like I mentioned, were time we cared about those slow tests and flakiness. But again, you might want to avoid financial costs, resource limits, or all kinds of other costs that can come into your test suite. So my goal was that wherever code on the test made a request of a for you, at least in our unit test suite, the tests would make sure that that request, that HTTP request, was actually handled by a mock function rather than going over the web. Now, we were very fortunate we were using the requests library that I'm sure some of you at least are familiar with. And there are a few ways with Python to get requests which are made with the requests library to point to some mock code. And the tool I chose is this one. It's called requests mock. I know there's also another one by the folks who make sentry called responses. There's also something called HTTP get if you're on Python 2 and maybe you're not using the requests library. Now the simple requests mock example is this one. So what you can say is here, when I make a get request to test.com, return the string that says data. And that's pretty simple. And at the same time as using requests mock, I'm sorry, a person who tried to take a photo, the slides will be online. And at the same time as using requests mock, we were also using PyTest. Now what PyTest is, it's a test runner which gives you a really neat way to do set up and tear down for test requirements. Now that feature is called fixtures and we have a fixture right here. And what this one says is, hey, if I use this fixture, then requests in this test will be handled by mock code. You can see we yield when we're in the context manager. But I'm sure that if you're using a more traditional test framework, you can use just the normal kind of set up and tear down methods. Now what I wanted, I didn't want to return the string data or something like that. I wanted some quite advanced features in my mock. And in particular, I wanted to have a stateful mock. And that would allow me to give different responses based on previous requests. So I could give a different match response if someone had already uploaded to the mock a picture of a matching label. So I use a requests mock feature which let me use a callable instead of a predefined response. And that callable takes a request-like object that gives me almost details of the request. So we created a whole bunch of small mock functions for every endpoint we used. And at this point, we pretty much achieved our goal. We could test our code without touching the real Vaforia. But then we hit some more problems, problems when we were using that mock. And I actually think that these are problems that a lot of mocks face. And sometimes we found that we'd copied the interface correctly. It can be pretty hard. There are lots of edge cases. What if the image is too big? Do we give the right error back? That kind of thing. And humans make mistakes even with code review. And so we found that we'd copied a lot of things incorrectly. But then even when we were extra careful, we found that the mock quickly became outdated whenever Vaforia changed. Now, if they sent out a really nice change log, we could change our mock to match it. But that's not always the case, especially for very minor things. And this isn't a Python library where you can even inspect the code changes. This is a web service. Now when you have an outdated mock, you have quite a serious problem. Or at least what was serious for us, which is our tests pass, but our software is actually failing in production. And when you've got that, you can have a real difficult time tracking down exactly why your code is broken. Because everything looks like it should be working. And you have to find, oh, actually, my mock is wrong. Where is it wrong? Trying to basically remake those manual requests to check your mock, it's very tedious. So that was a contract gig. And that contract ended. And I kind of felt like I'd built an OK solution. It was working all right for the client. But I really felt like the problem could be tackled in a better way and that I could have provided a better solution if I had more time. In particular because we kept hitting those issues of Vaforia changing and of human error. And at the same time, I really believed that Vaforia, and I still do, could be a genuinely useful tool for a bunch of people. And it could be especially useful if it was easy to develop against. So I set out to make VWS Python, which is basically an open source library for using the Vaforia web services with Python. It's in progress, hopefully coming very soon to PyPI. But I also had another goal. I started testing it with an open source mock part of that library. But I realized that the mock itself is very useful, whether or not you're using my library. And I wanted to ship that mock to people so that if they were writing code which used Vaforia, well, then they could have the mock for their own tests. So I wrote some integration tests for the library, and I wrote some unit tests for that library which used the mock. And I put the test suite on Travis CI because I knew it, and because it was free for open source projects. And one really cool feature of Travis, I'm sure a lot of other CI systems share it, though, is that I can give it the credentials for Vaforia. And I don't have to have those credentials show up in the code base where someone can abuse them. But I also don't have to have them show up in the logs. So I could really use the real service even from a CI system. And every time I made a change to the library, the tests were run, and those integration tests ran against the real Vaforia. But if you remember the goal I set, I wanted people to be able to use my mock to test their code whether or not they were using my library. And there's a cool way to let even people who use different programming languages, not just Python, to use your mock. While you're still keeping the interface really nice and pleasant, if you remember we had a PyTest fixture or if you're not using PyTest, just a context manager or decorator. So you want to keep that for Python users, but you want to let other people use your code as well. And the way that I did this is, well, I built the mock in a way that meant it could be run as a standalone server. And what that meant is ditching the request mock syntax that we had before. But at the same time, well, no, I'll move on. So I wrote this little bit of code. I'm not going to get into it too deeply because maybe I'm a little bit embarrassed. It's a bit of a hairy hack. But really, it let me rewrite the mock as a flask app and keep using it with request mock. So that means that I've got a flask app that I can just run as a standalone server. But if I use this code, it ties it into request mock. So let's say what it does is it translates those request objects from request mock into something that can be used by the, I guess, the Verkzoic test client again. But then you also translate responses from that test client into something that request mock can use. All this code will be online later. So if you're not using Python, then what you can do is you can spin up a flask app, let's say, in a Docker container for every test, and then you can route your request to that container using whatever kind of request mock alternative your language has. And that can be particularly useful, especially even if you're on an old Python version that doesn't support my mocks code. So I'd say this, if you're in an organization and you're writing a mock, and you want that mock to be used across your organization, even if people use different languages, this is a really cool way to do it. So back to writing the mock, this time around, the mock was definitely part of my product. So I didn't want to just do it in an ad hoc manner. I wanted to test it thoroughly. And I wanted to write those tests that confirmed it was doing what I wanted. So if you think about it at this point, I'm kind of probably duplicating a lot of the work that the people at Vaforia did, right? I'm rewriting a bit of their service, and I'm also thinking about edge cases for it. And what I'm doing is very manual. I'm making requests to their servers with those kind of edge cases that I'm thinking about. Then I'm noting the responses down in tests, and then I'm making sure that that test passes for my mock. And I test things, especially that aren't mentioned in the documentation. So let's say one example is they take a width for the image in centimeters. Well, what happens if you give it a negative width? Well, I did it. I found that they gave an error. I copied that exact error into my mock. And then the library, which is the kind of the main product, handles that error and raises an appropriate Python, a nice Python exception for that error. So at this point, I have three sets of tests. So I have a few integration tests, which use the test library with the real Vaforia. I have a whole bunch of unit tests for the library, maybe hundreds, thousands if you can, those which are generated by hypothesis. And those use the mock. And then I have some unit tests for the mock itself. But I'm still vulnerable to those problems that I mentioned earlier, copying incorrectly and Vaforia changing, which will render my mock inaccurate and now my library possibly even broken. So they're turning a mock into a verified fake, which is the title of this talk, is all about avoiding those problems. Now what a verified fake is, roughly, is it's a fake implementation, which is verified against the subset, a subset of the same test suite as the real implementation. Now I don't have the Vaforia code, and I definitely don't have their test suite if they've even got one. So if I wanted to make a verified fake, which I did, I needed to have my own test suite. So turning the mock into a verified fake really meant making a test suite which ran both against the mock and the real thing. So if you recall that simple PyTest fixture from before, well, I expanded it. So PyTest has this really cool feature called parameterization. And you can parameterize fixtures so that tests which use those fixtures are run once with each parameter option. So here I've got a simple true false, and I map that to use real Vaforia or not. And so any test which uses this fixture is run twice. So it's run once with the real Vaforia and then once with the mock Vaforia. So these are the test results. They look something like this. You can see each test runs twice. And fortunately, I already had at least the start of a test suite for that mock. So the first thing I did was I applied this to those tests so they ran against the mock and the real thing. And of course, I found that I'd made a whole bunch of mistakes. So now we've got a verified fake. And we have a test suite which runs against both the fake implementation and the real implementation. Now because the mock's been turned into a verified fake, we actually trust that it's representative of the real Vaforia. So we have loads of confidence in those hundreds of tests that we had for the library. And we know that they don't just rely on an unrealistic mock. But we also had another problem, if you remember. We were worried that Vaforia would change and that that would make our mock inaccurate. Well now, whenever these tests pass, I know that the mock is still a faithful representation of Vaforia. And we only incur the cost of running 100 tests against Vaforia, but we get almost the whole benefit of running thousands of tests against Vaforia. So we lessen that kind of cost of flakiness and slow tests. But at this point, our tests only run when we make a change to the code, which might not be that often, especially once it's quite mature. So we want to know what happens if Vaforia changes at that point. Well, a cool feature of Travis, and I'm sure a lot of other build systems, is that you can actually set tests to run on a schedule. So there's this trade-off. If you run them all the time, you find out problems quickly, but you hit those costs. If you run them very rarely, it takes you a long time to find out the problems. So the trade-off that I chose was to trigger them every night. But you can do them every release, every week, just whatever works for your particular situation. Now back to that width example. In the wine application I talked about at the beginning, we really didn't care about the physical width of a wine label. It wasn't a differentiating factor. But also it was actually really hard to get. That's why we didn't care about it that much. We told Vaforia all the time that the width was zero. Didn't matter to us. And that always worked, and our mock supported it. And when we get to the verified fake now, a few months later, the verified fake also supports it and has a test that a width of zero is okay. No error is returned. The image is added. But one morning I get an email from Travis, and it looks something like this. And it tells me that the build failed. So I look at the logs, and I see that we actually have a very precise data point of exactly what's changed in Vaforia. So the mock passes for this test, but the real implementation fails. And the test is, well, what if I add an image with a width of zero? So now what I do, I just change the mock function and the test so that the behavior, new behavior is represented by the mock, and that's very easy. But now if you remember the library's tests, they themselves depended on the mock. So now the library expects that a width of zero is valid, but it's invalid. So as soon as I change the mock, well, then the library's tests immediately started failing. So I could change the library to give a nice Python exception when you use a width of zero. And what that really demonstrates is that really within a few hours, Vaforia made an undocumented change, and that introduced an incompatibility with my library, and then this incompatibility was fixed without any real complex debugging. And to me, that shows the value of having a verified fake to any developer, really, who's using, who's writing code, which integrates with third-party software. So now you can imagine that building a verified fake when you have the original source code is much simpler than when you don't. And a lot of the fake can share, well, it can share code with the real implementation. And hardly any open, web services are open source, so this can be really valuable if you're shipping software to people. If you're shipping software to people, which they might want to call in tests, well, you can actually add tremendous value to that software by shipping your own verified fake. And it might even cause someone like me to choose to use your software over a competitor's. And if you make a verified fake as the author of the software, well, it's much easier because, well, because you can get told before merging any changes that it would make the fake unrealistic. So you know when to make changes to your code without the need for that once per day test run. So I'm hoping that maybe in the future, having an API which is easily tested against will become kind of table stakes. And one cool thing about making a verified fake, well, you don't really have to ship your secret source. You can just ship something that does the bare minimum of your API interface. Let's say you're making something like Vaforia, you can just have a really rubbishy kind of image-matching thing. That's the core of your business. You don't need to ship that to people. So I hope now that you have a rough idea at least of what a verified fake is, why it might be useful, and how you can start making one for yourself and for your users maybe. So thank you very much. That was my talk. And I'm happy to take any questions. Thank you. Okay. Some questions. Hi, very great talk and 80% overlapping with the one I gave two talks ago. But you've got a case study, which is great. I don't. I had the general discussion. And I think your ending is exactly where it should be. It's like there is no justification for releasing a component without a fake. The terminology, I'm trying to use Martin's. So the distinction between fake and mock. So for example, one thing left says is that essentially that a fake should be a spy. That's not in the original. A mock should be a spy. A fake should provide an introspection API perhaps. So in this case, I didn't worry about it so much because the API itself provides introspection abilities. Right. The big thing that's missing here in my view is the ability to simulate error. The example I give is CPU on fire. You don't really want to be there with a lighter to give fire to your CPU to check that your code is handling it. They mock, fake, whatever should be programmable to raise an error. So actually, I've got a response to that. So first of all, it's very difficult for the on fire case to verify it. Because how do you have a test that checks against, and thank you for your question as well, your comment, how do you have a test that tests that when you say this is going to give a 500, it will give a 500 just like when their servers are down because their servers aren't down right now. Actually, if you check out the source code for VWS Python, it takes a state object. And so I have various states like on fire but not quite. And so you can say, just like I had this verify mock fixture or verify Vaforia fixture or verify Vaforia context manager, you can give that a parameter which says broken, inactive, slow. And then you can see if your tests work even when there is a five minute delay in the matching ability. I hope that gives you a little bit of insight into how I've dealt with that issue. There's another question. I really appreciate the talk. But I was wondering, in this kind of service you were mocking, basically the response was depending on the data you put before. How would you go about mocking a service for which you don't have the ability to specify the data? For example, if I want to know the events in a specific location, they change every day, they're not in my control. And how can I write tests against this data which I don't have? Sure. So you can imagine that that API, that event consuming API that you're doing, let's say I think Event Bright is one of those companies or meetup.com, that they also have an API ad event. But that API might not be public to you. So what you've got to do is act as if you are the meetup.com person, you're the meetup.com servers and you just make some ability to add an event, even if it doesn't have a mocked API that will be exactly like theirs. And then you can know, okay, given that I've already added an event, it works in the same structure. Now, if you want to verify it, well, what you can do, you can have a test account that has an event in a particular location with a particular image, and then you can make your test run against that test account and have uploaded that kind of event into your mock already. And then you can say, okay, I want to check this event and check that the response is exactly the same. I hope that roughly answers your question, but you're right, it's not a solved issue. It's not always that easy, and it is context specific. Thank you. Next question. Yes, great. Is there any way that you could integrate this with fuzzing to find out the API responses that you may not be able to think of the applications not using? Sure. So I mentioned hypothesis before, that's the closest tool that I've personally used to fuzzing. If anyone doesn't know it, it's a property-based testing tool, and it, like I said, generates a lot of tests, which is kind of what fuzzing is. I haven't actually done it for this because the request limits were so slow, it was so low, and the request took so long. Actually the point of doing this for me was so that I could add fuzzing to my code. But you can imagine that if those problems weren't the case, well, you could say, hey, hypothesis or my fuzzing tool, please run random requests against my mark and the real implementation and check that they either are exactly the same in response or share some properties like they have the same keys. That would be ideal, but it really wasn't suitable in this case. Okay, another question? No? Oh, yes. So after just the lunchtime. Hi. A really nice talk. Thank you. I wonder, you have libraries like VCR or Betamax, which is ported from Ruby, right? And you can record a response, and it's recorded in JSON. And I wonder why you wouldn't use just for day-to-day testing like that and then at midnight or once a day, just disable the cache and see if the test passed then. So yeah, VCR tools are definitely something that I've used a bit. How do you know that the... I'll put it this way, maybe. You have a very similar case, right? That the API can change, and then when you disable the cache, then you have to update your VCR responses. And then you've kind of got a very similar thing, but you might not have the add component. If I want to hear add an image, what do I do in a VCR system? I kind of have... Sorry, I don't have a great answer for that. I'm going to pass on to the next one. This is an alternative, I guess, to a VCR system. No, I think that people use VCR to VCR some other service. I've certainly very briefly contributed to PyGitHub, a GitHub API, and what they do is they record responses from VCR. Maybe I tried to avoid it because it came with its own set of problems, and it was more painful for me to use than the system. No, I think it's too... It's cool. Thank you, Adam. Thank you to you.
 Thank you and thank you everyone for coming.  So my name is Adam Dengore.  I work at a company called Misesphere, building an operating system for data centers.  But last year I was working on something quite different.  I was working on the back end of an iPhone app.  Now what you do is as a user you would take a photo of a wine label with your phone and  the app would tell you all kinds of details about that wine.  Or at least it was something like that.  I'm going to protect my NDA here today.  And our app was a Flask app.  Now if you don't know Flask it's a really simple web framework.  And it looks something like this.  Now a really cool thing about Flask is that it provides, let me give a second, a Verkzeug  test client.  I hope I got that right.  And what that means is that you can make requests against an in-memory application and you  can get response objects which you can inspect.  So if we look at this test here, it kind of looks like we've made an HTTP request.  But actually everything's being done in memory in our test.  Now in our wine recognition app we use something as well called Vaforia web services.  And basically what Vaforia web services is, is it's a tool that lets you upload a whole  bunch of images, let's say in our case images of wine labels.  And then when a user uploaded a photo to us, well we could send that image to Vaforia and  then Vaforia would tell us which one of our previously uploaded images, their photo most  closely matched.  And then what we could do is we could fetch details about that wine from our database  and tell the user's details about the wine, like how much it should cost, how well rated  it was from, exactly where it's from, that kind of thing.  But when we built our prototype, well we kept finding loads of problems, loads of bugs.  And in particular those bugs came from assumptions that we'd made about Vaforia which weren't  quite right.  Often that came from reading their documentation and trusting that it was truthful and full  and you can't always make those assumptions.  And so what we wanted to do, we wanted to add tests for our matching workflow.  And that matching workflow of course used Vaforia and we wanted those tests to be in  our existing test suite.  Now Vaforia here was accessed over HTTP.  And that's what I'm going to focus on today.  But the general ideas really are on specific to HTTP because you might want to test code,  let's say, that uses a database for local storage.  Or you might want to test a deployment workflow which uses Docker.  Or maybe you even want to test code which uses Amazon S3 or some other cloud storage  as a storage backend.  Now we were lucky we had a very clear idea of what we wanted our first test to be.  I know this is quite a lot of code to have on a slide.  But simply what we wanted to test was that if a user uploaded a photo of a wine label  which matched a photo that we had already added, well then they would get details about  that wine.  So I wrote a test that looked a little bit like this.  I had two wines here.  Add wine, let's say it adds it to our database, but it also uploads it to Vaforia.  And then I check that I get the right one back when I query the match function.  That much function uses Vaforia on the backend.  Now with some third party tools, maybe even some of the ones I mentioned like Docker, you  might be totally fine, totally cool to call that real tool in your test suite.  But when we call Vaforia in our tests, we actually hit some problems.  Now first of all, we were at the mercy of the network.  And what that meant is when our CRE system had a little network glitch, well then our  whole test suite would fail.  Because our tests made HTTP requests against the Internet.  And we didn't know if those failures were because of the network failure or because  there was some kind of flakiness in our code.  But also we were at the mercy of Vaforia.  So similarly, when Vaforia went down or went down temporarily, our test suite would fail.  It really does slow down development if you're constantly worrying have you made a mistake  or is it on their end.  Now say you're using a real service like S3.  S3 might be pretty stable, probably even more stable than your software.  So you might not have to worry too much about flakiness.  But S3 charges you per megabyte use.  So if you want to use it in your test suite, it might actually become really expensive to  run your tests.  You might have to pay per megabyte and just spend quite a lot of money.  And another problem that you might run into is resource limits.  This is definitely something that I've hit.  A lot of services have resource limits, a certain number of requests that your account  can make.  And so if you call something in your test suite very heavily, especially if you're doing  performance benchmarking, you're making a load of calls, well then you might hit those resource  limits and you can't run your tests anymore and you're pretty much blocked on development.  And even when those things weren't problems, everything was really slow.  So the 4A is quite advanced software.  It does a lot of processing magic so that it can do the image matching.  And that means that after you've uploaded an image, well, it takes a few minutes until  that image can be matched.  That's totally reasonable.  I don't think that I could really expect them to do it instantly.  But in our test suite, well, I didn't really want to have to wait a few minutes to know  if our get match code worked.  So we call these tests like the one that I showed you before integration tests because,  well, they tested the integration of our software with Vaforius.  I think a lot of people get confused about the terminology.  Some people call these things acceptance tests or end-to-end tests.  But I think we can agree that they're high-level tests.  And they were definitely useful.  They really did help us track down some bugs.  But we also wanted unit tests because unit tests give us a lot of benefits over integration  tests.  And in particular, they tell us if our code calls Vaforia correctly in this case, even  when Vaforia is down.  And unit tests are also really small in scope.  And what that means is, well, let's say one fails, not all the time, but often you know  exactly which part of your code failed.  And if you change that bit of your code to make the unit test pass, well, that can be  a small isolated change.  And when you've got unit tests that run quickly and are small, well, you can even use some  tools maybe like Hypothesis to generate a whole bunch of unit tests.  So what we want to do, we want to turn a code base which can currently be tested only by  integration tests into one which can also be tested with unit tests.  And one way that some people achieve this is by using mocks.  Now, roughly, a mock is some code which provides the same interface as something that your  code calls, but it reduces or it removes some cost.  And in this case, the main costs that we cared about, like I mentioned, were time we cared  about those slow tests and flakiness.  But again, you might want to avoid financial costs, resource limits, or all kinds of other  costs that can come into your test suite.  So my goal was that wherever code on the test made a request of a for you, at least in our  unit test suite, the tests would make sure that that request, that HTTP request, was  actually handled by a mock function rather than going over the web.  Now, we were very fortunate we were using the requests library that I'm sure some of  you at least are familiar with.  And there are a few ways with Python to get requests which are made with the requests  library to point to some mock code.  And the tool I chose is this one.  It's called requests mock.  I know there's also another one by the folks who make sentry called responses.  There's also something called HTTP get if you're on Python 2 and maybe you're not using  the requests library.  Now the simple requests mock example is this one.  So what you can say is here, when I make a get request to test.com, return the string  that says data.  And that's pretty simple.  And at the same time as using requests mock, I'm sorry, a person who tried to take a photo,  the slides will be online.  And at the same time as using requests mock, we were also using PyTest.  Now what PyTest is, it's a test runner which gives you a really neat way to do set up and  tear down for test requirements.  Now that feature is called fixtures and we have a fixture right here.  And what this one says is, hey, if I use this fixture, then requests in this test will be  handled by mock code.  You can see we yield when we're in the context manager.  But I'm sure that if you're using a more traditional test framework, you can use just the normal  kind of set up and tear down methods.  Now what I wanted, I didn't want to return the string data or something like that.  I wanted some quite advanced features in my mock.  And in particular, I wanted to have a stateful mock.  And that would allow me to give different responses based on previous requests.  So I could give a different match response if someone had already uploaded to the mock  a picture of a matching label.  So I use a requests mock feature which let me use a callable instead of a predefined  response.  And that callable takes a request-like object that gives me almost details of the request.  So we created a whole bunch of small mock functions for every endpoint we used.  And at this point, we pretty much achieved our goal.  We could test our code without touching the real Vaforia.  But then we hit some more problems, problems when we were using that mock.  And I actually think that these are problems that a lot of mocks face.  And sometimes we found that we'd copied the interface correctly.  It can be pretty hard.  There are lots of edge cases.  What if the image is too big?  Do we give the right error back?  That kind of thing.  And humans make mistakes even with code review.  And so we found that we'd copied a lot of things incorrectly.  But then even when we were extra careful, we found that the mock quickly became outdated  whenever Vaforia changed.  Now, if they sent out a really nice change log, we could change our mock to match it.  But that's not always the case, especially for very minor things.  And this isn't a Python library where you can even inspect the code changes.  This is a web service.  Now when you have an outdated mock, you have quite a serious problem.  Or at least what was serious for us, which is our tests pass, but our software is actually  failing in production.  And when you've got that, you can have a real difficult time tracking down exactly why your  code is broken.  Because everything looks like it should be working.  And you have to find, oh, actually, my mock is wrong.  Where is it wrong?  Trying to basically remake those manual requests to check your mock, it's very tedious.  So that was a contract gig.  And that contract ended.  And I kind of felt like I'd built an OK solution.  It was working all right for the client.  But I really felt like the problem could be tackled in a better way and that I could have  provided a better solution if I had more time.  In particular because we kept hitting those issues of Vaforia changing and of human error.  And at the same time, I really believed that Vaforia, and I still do, could be a genuinely  useful tool for a bunch of people.  And it could be especially useful if it was easy to develop against.  So I set out to make VWS Python, which is basically an open source library for using  the Vaforia web services with Python.  It's in progress, hopefully coming very soon to PyPI.  But I also had another goal.  I started testing it with an open source mock part of that library.  But I realized that the mock itself is very useful, whether or not you're using my library.  And I wanted to ship that mock to people so that if they were writing code which used  Vaforia, well, then they could have the mock for their own tests.  So I wrote some integration tests for the library, and I wrote some unit tests for that  library which used the mock.  And I put the test suite on Travis CI because I knew it, and because it was free for open  source projects.  And one really cool feature of Travis, I'm sure a lot of other CI systems share it, though,  is that I can give it the credentials for Vaforia.  And I don't have to have those credentials show up in the code base where someone can  abuse them.  But I also don't have to have them show up in the logs.  So I could really use the real service even from a CI system.  And every time I made a change to the library, the tests were run, and those integration  tests ran against the real Vaforia.  But if you remember the goal I set, I wanted people to be able to use my mock to test their  code whether or not they were using my library.  And there's a cool way to let even people who use different programming languages, not  just Python, to use your mock.  While you're still keeping the interface really nice and pleasant, if you remember we had  a PyTest fixture or if you're not using PyTest, just a context manager or decorator.  So you want to keep that for Python users, but you want to let other people use your  code as well.  And the way that I did this is, well, I built the mock in a way that meant it could be run  as a standalone server.  And what that meant is ditching the request mock syntax that we had before.  But at the same time, well, no, I'll move on.  So I wrote this little bit of code.  I'm not going to get into it too deeply because maybe I'm a little bit embarrassed.  It's a bit of a hairy hack.  But really, it let me rewrite the mock as a flask app and keep using it with request  mock.  So that means that I've got a flask app that I can just run as a standalone server.  But if I use this code, it ties it into request mock.  So let's say what it does is it translates those request objects from request mock into  something that can be used by the, I guess, the Verkzoic test client again.  But then you also translate responses from that test client into something that request  mock can use.  All this code will be online later.  So if you're not using Python, then what you can do is you can spin up a flask app,  let's say, in a Docker container for every test, and then you can route your request  to that container using whatever kind of request mock alternative your language has.  And that can be particularly useful, especially even if you're on an old Python version that  doesn't support my mocks code.  So I'd say this, if you're in an organization and you're writing a mock, and you want that  mock to be used across your organization, even if people use different languages, this  is a really cool way to do it.  So back to writing the mock, this time around, the mock was definitely part of my product.  So I didn't want to just do it in an ad hoc manner.  I wanted to test it thoroughly.  And I wanted to write those tests that confirmed it was doing what I wanted.  So if you think about it at this point, I'm kind of probably duplicating a lot of the  work that the people at Vaforia did, right?  I'm rewriting a bit of their service, and I'm also thinking about edge cases for it.  And what I'm doing is very manual.  I'm making requests to their servers with those kind of edge cases that I'm thinking  about.  Then I'm noting the responses down in tests, and then I'm making sure that that test passes  for my mock.  And I test things, especially that aren't mentioned in the documentation.  So let's say one example is they take a width for the image in centimeters.  Well, what happens if you give it a negative width?  Well, I did it.  I found that they gave an error.  I copied that exact error into my mock.  And then the library, which is the kind of the main product, handles that error and raises  an appropriate Python, a nice Python exception for that error.  So at this point, I have three sets of tests.  So I have a few integration tests, which use the test library with the real Vaforia.  I have a whole bunch of unit tests for the library, maybe hundreds, thousands if you  can, those which are generated by hypothesis.  And those use the mock.  And then I have some unit tests for the mock itself.  But I'm still vulnerable to those problems that I mentioned earlier, copying incorrectly  and Vaforia changing, which will render my mock inaccurate and now my library possibly  even broken.  So they're turning a mock into a verified fake, which is the title of this talk, is  all about avoiding those problems.  Now what a verified fake is, roughly, is it's a fake implementation, which is verified  against the subset, a subset of the same test suite as the real implementation.  Now I don't have the Vaforia code, and I definitely don't have their test suite if they've even  got one.  So if I wanted to make a verified fake, which I did, I needed to have my own test suite.  So turning the mock into a verified fake really meant making a test suite which ran  both against the mock and the real thing.  So if you recall that simple PyTest fixture from before, well, I expanded it.  So PyTest has this really cool feature called parameterization.  And you can parameterize fixtures so that tests which use those fixtures are run once  with each parameter option.  So here I've got a simple true false, and I map that to use real Vaforia or not.  And so any test which uses this fixture is run twice.  So it's run once with the real Vaforia and then once with the mock Vaforia.  So these are the test results.  They look something like this.  You can see each test runs twice.  And fortunately, I already had at least the start of a test suite for that mock.  So the first thing I did was I applied this to those tests so they ran against the mock  and the real thing.  And of course, I found that I'd made a whole bunch of mistakes.  So now we've got a verified fake.  And we have a test suite which runs against both the fake implementation and the real  implementation.  Now because the mock's been turned into a verified fake, we actually trust that it's  representative of the real Vaforia.  So we have loads of confidence in those hundreds of tests that we had for the library.  And we know that they don't just rely on an unrealistic mock.  But we also had another problem, if you remember.  We were worried that Vaforia would change and that that would make our mock inaccurate.  Well now, whenever these tests pass, I know that the mock is still a faithful representation  of Vaforia.  And we only incur the cost of running 100 tests against Vaforia, but we get almost the  whole benefit of running thousands of tests against Vaforia.  So we lessen that kind of cost of flakiness and slow tests.  But at this point, our tests only run when we make a change to the code, which might  not be that often, especially once it's quite mature.  So we want to know what happens if Vaforia changes at that point.  Well, a cool feature of Travis, and I'm sure a lot of other build systems, is that you  can actually set tests to run on a schedule.  So there's this trade-off.  If you run them all the time, you find out problems quickly, but you hit those costs.  If you run them very rarely, it takes you a long time to find out the problems.  So the trade-off that I chose was to trigger them every night.  But you can do them every release, every week, just whatever works for your particular situation.  Now back to that width example.  In the wine application I talked about at the beginning, we really didn't care about  the physical width of a wine label.  It wasn't a differentiating factor.  But also it was actually really hard to get.  That's why we didn't care about it that much.  We told Vaforia all the time that the width was zero.  Didn't matter to us.  And that always worked, and our mock supported it.  And when we get to the verified fake now, a few months later, the verified fake also  supports it and has a test that a width of zero is okay.  No error is returned.  The image is added.  But one morning I get an email from Travis, and it looks something like this.  And it tells me that the build failed.  So I look at the logs, and I see that we actually have a very precise data point of exactly  what's changed in Vaforia.  So the mock passes for this test, but the real implementation fails.  And the test is, well, what if I add an image with a width of zero?  So now what I do, I just change the mock function and the test so that the behavior, new behavior  is represented by the mock, and that's very easy.  But now if you remember the library's tests, they themselves depended on the mock.  So now the library expects that a width of zero is valid, but it's invalid.  So as soon as I change the mock, well, then the library's tests immediately started failing.  So I could change the library to give a nice Python exception when you use a width of zero.  And what that really demonstrates is that really within a few hours, Vaforia made an  undocumented change, and that introduced an incompatibility with my library, and then  this incompatibility was fixed without any real complex debugging.  And to me, that shows the value of having a verified fake to any developer, really,  who's using, who's writing code, which integrates with third-party software.  So now you can imagine that building a verified fake when you have the original source code  is much simpler than when you don't.  And a lot of the fake can share, well, it can share code with the real implementation.  And hardly any open, web services are open source, so this can be really valuable if  you're shipping software to people.  If you're shipping software to people, which they might want to call in tests, well, you  can actually add tremendous value to that software by shipping your own verified fake.  And it might even cause someone like me to choose to use your software over a competitor's.  And if you make a verified fake as the author of the software, well, it's much easier because,  well, because you can get told before merging any changes that it would make the fake unrealistic.  So you know when to make changes to your code without the need for that once per day test  run.  So I'm hoping that maybe in the future, having an API which is easily tested against will  become kind of table stakes.  And one cool thing about making a verified fake, well, you don't really have to ship  your secret source.  You can just ship something that does the bare minimum of your API interface.  Let's say you're making something like Vaforia, you can just have a really rubbishy kind of  image-matching thing.  That's the core of your business.  You don't need to ship that to people.  So I hope now that you have a rough idea at least of what a verified fake is, why it might  be useful, and how you can start making one for yourself and for your users maybe.  So thank you very much.  That was my talk.  And I'm happy to take any questions.  Thank you.  Okay.  Some questions.  Hi, very great talk and 80% overlapping with the one I gave two talks ago.  But you've got a case study, which is great.  I don't.  I had the general discussion.  And I think your ending is exactly where it should be.  It's like there is no justification for releasing a component without a fake.  The terminology, I'm trying to use Martin's.  So the distinction between fake and mock.  So for example, one thing left says is that essentially that a fake should be a spy.  That's not in the original.  A mock should be a spy.  A fake should provide an introspection API perhaps.  So in this case, I didn't worry about it so much because the API itself provides introspection  abilities.  Right.  The big thing that's missing here in my view is the ability to simulate error.  The example I give is CPU on fire.  You don't really want to be there with a lighter to give fire to your CPU to check that your  code is handling it.  They mock, fake, whatever should be programmable to raise an error.  So actually, I've got a response to that.  So first of all, it's very difficult for the on fire case to verify it.  Because how do you have a test that checks against, and thank you for your question as  well, your comment, how do you have a test that tests that when you say this is going  to give a 500, it will give a 500 just like when their servers are down because their  servers aren't down right now.  Actually, if you check out the source code for VWS Python, it takes a state object.  And so I have various states like on fire but not quite.  And so you can say, just like I had this verify mock fixture or verify Vaforia fixture or  verify Vaforia context manager, you can give that a parameter which says broken, inactive,  slow.  And then you can see if your tests work even when there is a five minute delay in the matching  ability.  I hope that gives you a little bit of insight into how I've dealt with that issue.  There's another question.  I really appreciate the talk.  But I was wondering, in this kind of service you were mocking, basically the response was  depending on the data you put before.  How would you go about mocking a service for which you don't have the ability to specify  the data?  For example, if I want to know the events in a specific location, they change every day,  they're not in my control.  And how can I write tests against this data which I don't have?  Sure.  So you can imagine that that API, that event consuming API that you're doing, let's say  I think Event Bright is one of those companies or meetup.com, that they also have an API  ad event.  But that API might not be public to you.  So what you've got to do is act as if you are the meetup.com person, you're the meetup.com  servers and you just make some ability to add an event, even if it doesn't have a mocked  API that will be exactly like theirs.  And then you can know, okay, given that I've already added an event, it works in the same  structure.  Now, if you want to verify it, well, what you can do, you can have a test account that  has an event in a particular location with a particular image, and then you can make  your test run against that test account and have uploaded that kind of event into your  mock already.  And then you can say, okay, I want to check this event and check that the response is  exactly the same.  I hope that roughly answers your question, but you're right, it's not a solved issue.  It's not always that easy, and it is context specific.  Thank you.  Next question.  Yes, great.  Is there any way that you could integrate this with fuzzing to find out the API responses  that you may not be able to think of the applications not using?  Sure.  So I mentioned hypothesis before, that's the closest tool that I've personally used to  fuzzing.  If anyone doesn't know it, it's a property-based testing tool, and it, like I said, generates  a lot of tests, which is kind of what fuzzing is.  I haven't actually done it for this because the request limits were so slow, it was so  low, and the request took so long.  Actually the point of doing this for me was so that I could add fuzzing to my code.  But you can imagine that if those problems weren't the case, well, you could say, hey,  hypothesis or my fuzzing tool, please run random requests against my mark and the real  implementation and check that they either are exactly the same in response or share  some properties like they have the same keys.  That would be ideal, but it really wasn't suitable in this case.  Okay, another question?  No?  Oh, yes.  So after just the lunchtime.  Hi.  A really nice talk.  Thank you.  I wonder, you have libraries like VCR or Betamax, which is ported from Ruby, right?  And you can record a response, and it's recorded in JSON.  And I wonder why you wouldn't use just for day-to-day testing like that and then at midnight  or once a day, just disable the cache and see if the test passed then.  So yeah, VCR tools are definitely something that I've used a bit.  How do you know that the...  I'll put it this way, maybe.  You have a very similar case, right?  That the API can change, and then when you disable the cache, then you have to update  your VCR responses.  And then you've kind of got a very similar thing, but you might not have the add component.  If I want to hear add an image, what do I do in a VCR system?  I kind of have...  Sorry, I don't have a great answer for that.  I'm going to pass on to the next one.  This is an alternative, I guess, to a VCR system.  No, I think that people use VCR to VCR some other service.  I've certainly very briefly contributed to PyGitHub, a GitHub API, and what they do is  they record responses from VCR.  Maybe I tried to avoid it because it came with its own set of problems, and it was more  painful for me to use than the system.  No, I think it's too...  It's cool.  Thank you, Adam.  Thank you to you.
Adam_Dangoor_-_But_how_do_you_know_your_mock_is_valid__Verified_fakes_of_web_services_bYtA1WK7aLM_0_323.png Adam_Dangoor_-_But_how_do_you_know_your_mock_is_valid__Verified_fakes_of_web_services_bYtA1WK7aLM_324_9731.png Adam_Dangoor_-_But_how_do_you_know_your_mock_is_valid__Verified_fakes_of_web_services_bYtA1WK7aLM_9732_9839.png Adam_Dangoor_-_But_how_do_you_know_your_mock_is_valid__Verified_fakes_of_web_services_bYtA1WK7aLM_9840_37463.png Adam_Dangoor_-_But_how_do_you_know_your_mock_is_valid__Verified_fakes_of_web_services_bYtA1WK7aLM_37464_37547.png Adam_Dangoor_-_But_how_do_you_know_your_mock_is_valid__Verified_fakes_of_web_services_bYtA1WK7aLM_37548_38591.png Adam_Dangoor_-_But_how_do_you_know_your_mock_is_valid__Verified_fakes_of_web_services_bYtA1WK7aLM_38592_39095.png Adam_Dangoor_-_But_how_do_you_know_your_mock_is_valid__Verified_fakes_of_web_services_bYtA1WK7aLM_39096_39323.png Adam_Dangoor_-_But_how_do_you_know_your_mock_is_valid__Verified_fakes_of_web_services_bYtA1WK7aLM_39324_39467.png Adam_Dangoor_-_But_how_do_you_know_your_mock_is_valid__Verified_fakes_of_web_services_bYtA1WK7aLM_39468_41471.png Adam_Dangoor_-_But_how_do_you_know_your_mock_is_valid__Verified_fakes_of_web_services_bYtA1WK7aLM_41472_41519.png Adam_Dangoor_-_But_how_do_you_know_your_mock_is_valid__Verified_fakes_of_web_services_bYtA1WK7aLM_41520_41711.png Adam_Dangoor_-_But_how_do_you_know_your_mock_is_valid__Verified_fakes_of_web_services_bYtA1WK7aLM_41712_42227.png Adam_Dangoor_-_But_how_do_you_know_your_mock_is_valid__Verified_fakes_of_web_services_bYtA1WK7aLM_42228_42275.png Adam_Dangoor_-_But_how_do_you_know_your_mock_is_valid__Verified_fakes_of_web_services_bYtA1WK7aLM_42276_42335.png Adam_Dangoor_-_But_how_do_you_know_your_mock_is_valid__Verified_fakes_of_web_services_bYtA1WK7aLM_42336_42383.png Adam_Dangoor_-_But_how_do_you_know_your_mock_is_valid__Verified_fakes_of_web_services_bYtA1WK7aLM_42384_42443.png Adam_Dangoor_-_But_how_do_you_know_your_mock_is_valid__Verified_fakes_of_web_services_bYtA1WK7aLM_42444_43679.png Adam_Dangoor_-_But_how_do_you_know_your_mock_is_valid__Verified_fakes_of_web_services_bYtA1WK7aLM_43680_44387.png Adam_Dangoor_-_But_how_do_you_know_your_mock_is_valid__Verified_fakes_of_web_services_bYtA1WK7aLM_44388_44447.png Adam_Dangoor_-_But_how_do_you_know_your_mock_is_valid__Verified_fakes_of_web_services_bYtA1WK7aLM_44448_44591.png Adam_Dangoor_-_But_how_do_you_know_your_mock_is_valid__Verified_fakes_of_web_services_bYtA1WK7aLM_44592_44639.png Adam_Dangoor_-_But_how_do_you_know_your_mock_is_valid__Verified_fakes_of_web_services_bYtA1WK7aLM_44640_44723.png Adam_Dangoor_-_But_how_do_you_know_your_mock_is_valid__Verified_fakes_of_web_services_bYtA1WK7aLM_44724_44807.png Adam_Dangoor_-_But_how_do_you_know_your_mock_is_valid__Verified_fakes_of_web_services_bYtA1WK7aLM_44808_44891.png Adam_Dangoor_-_But_how_do_you_know_your_mock_is_valid__Verified_fakes_of_web_services_bYtA1WK7aLM_44892_46043.png Adam_Dangoor_-_But_how_do_you_know_your_mock_is_valid__Verified_fakes_of_web_services_bYtA1WK7aLM_46044_46103.png Adam_Dangoor_-_But_how_do_you_know_your_mock_is_valid__Verified_fakes_of_web_services_bYtA1WK7aLM_46104_46163.png Adam_Dangoor_-_But_how_do_you_know_your_mock_is_valid__Verified_fakes_of_web_services_bYtA1WK7aLM_46164_46211.png Adam_Dangoor_-_But_how_do_you_know_your_mock_is_valid__Verified_fakes_of_web_services_bYtA1WK7aLM_46212_46979.png Adam_Dangoor_-_But_how_do_you_know_your_mock_is_valid__Verified_fakes_of_web_services_bYtA1WK7aLM_46980_47039.png Adam_Dangoor_-_But_how_do_you_know_your_mock_is_valid__Verified_fakes_of_web_services_bYtA1WK7aLM_47040_47207.png Adam_Dangoor_-_But_how_do_you_know_your_mock_is_valid__Verified_fakes_of_web_services_bYtA1WK7aLM_47208_47507.png Adam_Dangoor_-_But_how_do_you_know_your_mock_is_valid__Verified_fakes_of_web_services_bYtA1WK7aLM_47508_47939.png Adam_Dangoor_-_But_how_do_you_know_your_mock_is_valid__Verified_fakes_of_web_services_bYtA1WK7aLM_47940_49283.png Adam_Dangoor_-_But_how_do_you_know_your_mock_is_valid__Verified_fakes_of_web_services_bYtA1WK7aLM_49284_49406.png