Effectively test your webapp with Python and Selenium
CC Attribution - NonCommercial - ShareAlike 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this
Coman, Andrei
Andrei Coman - Effectively test your webapp with Python and Selenium How often do you run your Selenium test suite? How fast do you get a result from it? Would you like to answer with: "Whenever I feel like it" and "Well, about the time it takes me to finish a coffee" ? This talk will try to get you closer to these answers. We will have a look at the lessons learned and the challenges my team faced maintaining a Selenium test suite against a long-lived Django web application. We will go over the pros and cons of: - test design approaches - technologies we used (nose, py.test, LiveServerTestCase) - reporting tools
 Okay, so let's welcome Andrej Komann and he'll talk to us about testing the web application with Selenium. Okay. That's working? Yep. Cool. So effectively testing, effectively test your web, Python, and Selenium. That's a mouthful, right? So what's this talk actually about? I've been working on a project for a pretty long time now, and I got to see, like, from its beginnings how the project itself evolved, like the code base, and how the tests around it evolved, because we reached some conclusions at some points in time, and we tried to improve what we had. And that's what I want to share with you guys, like, what we learned from that, and hoping that you don't make the same mistakes that we did. That's cool. Okay. So my name is Andrej Komann. I'm from Timishwara. That's a city in Romania, in the west of Romania. I work at 3peller for PBS. That's the public broadcast system in the US. You can find me at friends in my code on GitHub or follow me on Twitter. That's cool. How about you guys? How many of you have been working with Selenium? Well, quite a lot. How many of you are, like, QA, software engineering tests, something? Okay, a lot less. That's not a bad thing. So, well, let's get cracking then. Right? We kind of have, like, I like to think of it like the way the tests evolved in three phases, and we're going to take a look at each phase and see what we did right in each phase, what we did not so good, and how we try to iterate over that in the next phase. Okay? Cool. So, this is how a test looked like in, like, the first phase. Bear in mind that this has been, like, in the beginnings of the project where the QA team didn't have that much support for development because we were, like, crunching out to get features out. So this is more like what the tests look like. This created resource, thinkable resource, like a blog post or something. It gave it a title, right? Pretty innocent. And then we had a test that kind of used that particular resource. Nothing out of the ordinary till now. So let's take a look at how the page object model for Selenium looked like. So we were good kids, and we separated the Selenium interaction from the test itself. So we have, like, this page object model classes that model the page. And at the time we were using XPath. So to get the title, we were doing, like, getting the div and going by a specific ID, then, you know, go a level up and starts to get a little bit confusing, right? Cool. So what this test was hiding, actually, that at the time we were using NOS tests as our test runner, and NOS tests when it starts collecting your tests by default, it runs them in alphabetical order. So if C goes before V, you can use your test kind of like means to create the fixture and the state for the test on the server, and then test that in a different test. You can see how that kind of doesn't scale and contradicts some good practices of keeping tests separated and independent. And yep, this is the kind of the problem with the way we use Selenium. So we used the page object model. That was cool. But we had this really long XPath that makes my head hurt when I try to read it. And whenever a developer came in unknowingly and changed the way the HTML was structured for this specific page, well, that XPath, that test got broken inadvertently. So all these tests were running, like, on environment. We were giving it, hey, run this on production, run this on QA, run this on staging. But yeah, you can imagine that if the tests were failing like midway, it's not that cool that you left leftovers on production. Okay. So yeah, as I said, we wanted something better. We wanted to stop using tests as fixture generators. We want to move to something more robust in terms of identifying a Selenium element and interacting with it, getting its value or interacting with that element like it's a button click on it or something. So I think at the time we upgraded to Django 2014. And then we had live server test case. So we said, yeah, let's give that a shot. Why not? And then all the problems with setting up fixtures kind of magically went away because we were using Django ORN to create, like, the state we were going to test. And using tests to just do a targeted test to see, yeah, this is the title of the page where I can click on a specific element and the video plays or stuff like that. And we weren't stuck in that end-to-end test where you need to create something, maybe using the admin and then test it using the published state. So this was like a better world for us. We can run tests independently. We can parallelize, which we couldn't do without many headaches before. So this is good, right? So we also took note of how we were handling Selenium integration. And we started moving to something more specific. So we started hooking into IDs or CSS selectors. And that gave us also more readability into the code. Like, you know, it's going to be an ID called title. And it was also less brittle because once you moved, like, sections of HTML in your templates, it's not as likely to break the tests with that. Cool. So what's the problem with this kind of approach? It's not testing the real environment. Now, when you get that email in the middle of the day saying that, hey, we are a team that you're a client of, so like we have a team that manages RR image, ingestion, and resizing, right? And if they make a deploy, they kind of send us an email, hey, run some tests to make sure your production environment still works. Well, you can't really use these tests for that kind of thing because these tests spin up like a really bare bones environment. You just create just enough data to do your test in them. And afterwards, you throw them away. And they're not really designed to test the real environment. Whereas in the previous approach, we were actually testing like QA production and so on. Cool. This is a good thing that in the page object model, we started using IDs. But then again, we could do a bit better here, too. Okay. So what we had, what we wanted to do better after all these two phases were like separating long running tests from short tests. You still like to have end-to-end tests that go maybe to multiple parts of your application and test the workflow. But maybe something like if you have a blog and you just check that the specific page works, that's good enough for like a smoke test. Google does this. So in Google test software, they have like a section where they kind of chunk tests in medium or small, large, extra large. And small tests are the unit test. The developer runs on his machine. The large tests are more like these integration tests. And the next or large would be like an end-to-end test. So it's important to put a distinction between these and put time boxes in which a test with like all my large tests must run under five minutes. Running them independently, that's kind of a conflict between the first phase and second phase. We couldn't run them independently, but we could use an environment. In the second phase, we ran them independently, but we weren't hitting any real environment. So we started looking around. I mentioned we were using those tests, but PyTest was like the new kid on a block and made sense to try to look at what that has to offer. We put a limit on our Selenium suite. It has to run in under five minutes, five or 10 minutes. Previously our whole Selenium was running in, I think, 45 minutes. In the first iteration, so in the first phase, it was really hard to debug a test. So if you wanted to, if it failed in the view part, you had to go through all the creation and it was a very cumbersome and hard effort to debug. So also what we wanted to make an emphasis on is decoupling the test from the HTML structure of the page. And for this, we went a step further from hooking, so identifying elements from existing CSS selectors or IDs in the page to setting up a convention between the development team and the QA team, saying, hey, this is a prefix which all the identifiers, so if you have a class like Selenium-something that's reserved for testing, so for allowing testers to hook up to that, identify that element and do stuff with it. It's important not to tie any CSS or JavaScript functionality to that because you're just going to go loophole. It's important to keep those test hooks focused only on that. So how does the test look like nowadays? The page is pretty much the same except that we have this stuff like Selenium prefix for everything, so you don't go in and rely on IDs or CSS classes that will put in on developers and maybe make that specific element red or pop out or something. And also, we kept the same page object model. And that's pretty much it. And test-wise, because we switched to PyTest, we started using a couple of interesting plugins from the Mazele foundation. They open-sourced a few plugins that I'm going to showcase a bit later in the presentation, but they really helped us to ramp up on a new test suite. So one of the first plugins that I tried out was PyTest variables. This is something that you can put in to your project and keep your fixtures or your credentials or something like that in a JSON file and then pass that into the test. It kind of separates your fixture data, your credentials from the code itself. And it has a pretty simple interface. Pretty cool for the Mazele guys to open-source this. Otherwise, there's PyTest HTML. It's a really cool plugin that is not necessarily tied to the Selenium integration, but it provides you with a test report saying, hey, this test failed here, and these were the values that it failed at. And if you're using it in conjunction with the PyTest HTML, the PyTest Selenium plugin from the Mazele guys, it also hooks in and puts your screenshot from when the test failed. That's pretty cool. PyTest Selenium, this is the PyTest plugin that the Mazele Foundation open-sourced. It's pretty cool. It has a lot of Chrome drivers, a lot of web drivers, sorry, Chrome, Firefox. A fun fact, we initially ran a lot of tests on Firefox, and after switching to Chrome, we saw a significant performance improvement. And it was pretty cool that this plugin allowed us just to change the parameter somewhere, and then all our tests were, like, running on Chrome instead of that. And also, they have support for connecting with cloud-based testing services like Sauce Labs or BrowserStack. Sorry? Yeah. Yeah. But I think for Chrome, you need to install the Chrome driver, other than that, you need to install the Selenium package in your environment. So what I was thinking for you guys to take away from this talk is, and what we're planning to do on the project going forward is leveraging APIs to create data fixtures. So if you have, like, a REST API to create stuff and remove stuff from your app, try to leverage that, you know, like creating a blog post and then deleting it for test purposes, adding additional metadata into your page. So these are that convention between the development of the QA team that everything prefixed with something Selenium. It could be whatever, banana or whatever you want. And last but not least, defining, like, these test classes. So I have large tests, I have small tests, and put time boxes around them. And when that specific class exceeds that time box, then you really should look at failing that whole class and looking at why all of a sudden my whole large test cases, like maybe that's a smoke test, so it's running slower than before. That was about it. Thank you for your time. Do you have any questions? Do you know of an easy way to get video recording of these test runs? Not really. I haven't toyed around with that yet. That's an interesting point. I'll try to check up with the Mozilla program. Maybe they offer that. If not, maybe it's something worth contributing to open source. I know in the past, we were at a previous company, we were using BNC to FLB to record test runs and see why it failed. To answer the previous question, I know Mozilla has something because they use videos in PyTest HTML. So maybe just check the documentation for that. I hope it's documented if not open issue. Then maybe it will be. Any other questions? Have you looked into Splinter at all and PyTest Splinter plugin? Are you? I haven't. It's a convenience wrapper library in Python for Selenium which helps you write better asserts without going into the markup. So it's like instead of putting.sl markup everywhere, it helps you to write better tests that are less brittle. So maybe just a hint, it makes it easier to write nicer tests. Cool. I'm curious to talk to you about that afterwards. Thanks. How did you version your page objects if you were making changes in the application? How did you deal with having different versions of and like regression testing with Selenium? Good one. So what we're trying now is trying to get an element and if you can't get that hook element, you skip the tests and market the skip and say that this test was skipped until that code reaches the environment you're trying to test. Any other questions? Okay. If that's all, let's say thank you to Andre. Thank you.
 Okay, so let's welcome Andrej Komann and he'll talk to us about testing the web application  with Selenium.  Okay.  That's working?  Yep.  Cool.  So effectively testing, effectively test your web, Python, and Selenium.  That's a mouthful, right?  So what's this talk actually about?  I've been working on a project for a pretty long time now, and I got to see, like, from  its beginnings how the project itself evolved, like the code base, and how the tests around  it evolved, because we reached some conclusions at some points in time, and we tried to improve  what we had.  And that's what I want to share with you guys, like, what we learned from that, and hoping  that you don't make the same mistakes that we did.  That's cool.  Okay.  So my name is Andrej Komann.  I'm from Timishwara.  That's a city in Romania, in the west of Romania.  I work at 3peller for PBS.  That's the public broadcast system in the US.  You can find me at friends in my code on GitHub or follow me on Twitter.  That's cool.  How about you guys?  How many of you have been working with Selenium?  Well, quite a lot.  How many of you are, like, QA, software engineering tests, something?  Okay, a lot less.  That's not a bad thing.  So, well, let's get cracking then.  Right?  We kind of have, like, I like to think of it like the way the tests evolved in three phases,  and we're going to take a look at each phase and see what we did right in each phase, what  we did not so good, and how we try to iterate over that in the next phase.  Okay?  Cool.  So, this is how a test looked like in, like, the first phase.  Bear in mind that this has been, like, in the beginnings of the project where the QA team  didn't have that much support for development because we were, like, crunching out to get  features out.  So this is more like what the tests look like.  This created resource, thinkable resource, like a blog post or something.  It gave it a title, right?  Pretty innocent.  And then we had a test that kind of used that particular resource.  Nothing out of the ordinary till now.  So let's take a look at how the page object model for Selenium looked like.  So we were good kids, and we separated the Selenium interaction from the test itself.  So we have, like, this page object model classes that model the page.  And at the time we were using XPath.  So to get the title, we were doing, like, getting the div and going by a specific ID,  then, you know, go a level up and starts to get a little bit confusing, right?  Cool.  So what this test was hiding, actually, that at the time we were using NOS tests as our  test runner, and NOS tests when it starts collecting your tests by default, it runs them  in alphabetical order.  So if C goes before V, you can use your test kind of like means to create the fixture and  the state for the test on the server, and then test that in a different test.  You can see how that kind of doesn't scale and contradicts some good practices of keeping  tests separated and independent.  And yep, this is the kind of the problem with the way we use Selenium.  So we used the page object model.  That was cool.  But we had this really long XPath that makes my head hurt when I try to read it.  And whenever a developer came in unknowingly and changed the way the HTML was structured  for this specific page, well, that XPath, that test got broken inadvertently.  So all these tests were running, like, on environment.  We were giving it, hey, run this on production, run this on QA, run this on staging.  But yeah, you can imagine that if the tests were failing like midway, it's not that cool  that you left leftovers on production.  Okay.  So yeah, as I said, we wanted something better.  We wanted to stop using tests as fixture generators.  We want to move to something more robust in terms of identifying a Selenium element and  interacting with it, getting its value or interacting with that element like it's a  button click on it or something.  So I think at the time we upgraded to Django 2014.  And then we had live server test case.  So we said, yeah, let's give that a shot.  Why not?  And then all the problems with setting up fixtures kind of magically went away because  we were using Django ORN to create, like, the state we were going to test.  And using tests to just do a targeted test to see, yeah, this is the title of the page  where I can click on a specific element and the video plays or stuff like that.  And we weren't stuck in that end-to-end test where you need to create something, maybe  using the admin and then test it using the published state.  So this was like a better world for us.  We can run tests independently.  We can parallelize, which we couldn't do without many headaches before.  So this is good, right?  So we also took note of how we were handling Selenium integration.  And we started moving to something more specific.  So we started hooking into IDs or CSS selectors.  And that gave us also more readability into the code.  Like, you know, it's going to be an ID called title.  And it was also less brittle because once you moved, like, sections of HTML in your  templates, it's not as likely to break the tests with that.  Cool.  So what's the problem with this kind of approach?  It's not testing the real environment.  Now, when you get that email in the middle of the day saying that, hey, we are a team  that you're a client of, so like we have a team that manages RR image, ingestion, and  resizing, right?  And if they make a deploy, they kind of send us an email, hey, run some tests to make sure  your production environment still works.  Well, you can't really use these tests for that kind of thing because these tests spin  up like a really bare bones environment.  You just create just enough data to do your test in them.  And afterwards, you throw them away.  And they're not really designed to test the real environment.  Whereas in the previous approach, we were actually testing like QA production and so  on.  Cool.  This is a good thing that in the page object model, we started using IDs.  But then again, we could do a bit better here, too.  Okay.  So what we had, what we wanted to do better after all these two phases were like separating  long running tests from short tests.  You still like to have end-to-end tests that go maybe to multiple parts of your application  and test the workflow.  But maybe something like if you have a blog and you just check that the specific page works,  that's good enough for like a smoke test.  Google does this.  So in Google test software, they have like a section where they kind of chunk tests in  medium or small, large, extra large.  And small tests are the unit test.  The developer runs on his machine.  The large tests are more like these integration tests.  And the next or large would be like an end-to-end test.  So it's important to put a distinction between these and put time boxes in which a test with  like all my large tests must run under five minutes.  Running them independently, that's kind of a conflict between the first phase and second  phase.  We couldn't run them independently, but we could use an environment.  In the second phase, we ran them independently, but we weren't hitting any real environment.  So we started looking around.  I mentioned we were using those tests, but PyTest was like the new kid on a block and  made sense to try to look at what that has to offer.  We put a limit on our Selenium suite.  It has to run in under five minutes, five or 10 minutes.  Previously our whole Selenium was running in, I think, 45 minutes.  In the first iteration, so in the first phase, it was really hard to debug a test.  So if you wanted to, if it failed in the view part, you had to go through all the creation  and it was a very cumbersome and hard effort to debug.  So also what we wanted to make an emphasis on is decoupling the test from the HTML structure  of the page.  And for this, we went a step further from hooking, so identifying elements from existing  CSS selectors or IDs in the page to setting up a convention between the development team  and the QA team, saying, hey, this is a prefix which all the identifiers, so if you have a  class like Selenium-something that's reserved for testing, so for allowing testers to hook  up to that, identify that element and do stuff with it.  It's important not to tie any CSS or JavaScript functionality to that because you're just going  to go loophole.  It's important to keep those test hooks focused only on that.  So how does the test look like nowadays?  The page is pretty much the same except that we have this stuff like Selenium prefix for  everything, so you don't go in and rely on IDs or CSS classes that will put in on developers  and maybe make that specific element red or pop out or something.  And also, we kept the same page object model.  And that's pretty much it.  And test-wise, because we switched to PyTest, we started using a couple of interesting plugins  from the Mazele foundation.  They open-sourced a few plugins that I'm going to showcase a bit later in the presentation,  but they really helped us to ramp up on a new test suite.  So one of the first plugins that I tried out was PyTest variables.  This is something that you can put in to your project and keep your fixtures or your credentials  or something like that in a JSON file and then pass that into the test.  It kind of separates your fixture data, your credentials from the code itself.  And it has a pretty simple interface.  Pretty cool for the Mazele guys to open-source this.  Otherwise, there's PyTest HTML.  It's a really cool plugin that is not necessarily tied to the Selenium integration, but it  provides you with a test report saying, hey, this test failed here, and these were the values  that it failed at.  And if you're using it in conjunction with the PyTest HTML, the PyTest Selenium plugin  from the Mazele guys, it also hooks in and puts your screenshot from when the test failed.  That's pretty cool.  PyTest Selenium, this is the PyTest plugin that the Mazele Foundation open-sourced.  It's pretty cool.  It has a lot of Chrome drivers, a lot of web drivers, sorry, Chrome, Firefox.  A fun fact, we initially ran a lot of tests on Firefox, and after switching to Chrome,  we saw a significant performance improvement.  And it was pretty cool that this plugin allowed us just to change the parameter somewhere,  and then all our tests were, like, running on Chrome instead of that.  And also, they have support for connecting with cloud-based testing services like Sauce  Labs or BrowserStack.  Sorry?  Yeah.  Yeah.  But I think for Chrome, you need to install the Chrome driver, other than that, you need  to install the Selenium package in your environment.  So what I was thinking for you guys to take away from this talk is, and what we're planning  to do on the project going forward is leveraging APIs to create data fixtures.  So if you have, like, a REST API to create stuff and remove stuff from your app, try  to leverage that, you know, like creating a blog post and then deleting it for test purposes,  adding additional metadata into your page.  So these are that convention between the development of the QA team that everything prefixed with  something Selenium.  It could be whatever, banana or whatever you want.  And last but not least, defining, like, these test classes.  So I have large tests, I have small tests, and put time boxes around them.  And when that specific class exceeds that time box, then you really should look at failing  that whole class and looking at why all of a sudden my whole large test cases, like maybe  that's a smoke test, so it's running slower than before.  That was about it.  Thank you for your time.  Do you have any questions?  Do you know of an easy way to get video recording of these test runs?  Not really.  I haven't toyed around with that yet.  That's an interesting point.  I'll try to check up with the Mozilla program.  Maybe they offer that.  If not, maybe it's something worth contributing to open source.  I know in the past, we were at a previous company, we were using BNC to FLB to record  test runs and see why it failed.  To answer the previous question, I know Mozilla has something because they use videos in PyTest  HTML.  So maybe just check the documentation for that.  I hope it's documented if not open issue.  Then maybe it will be.  Any other questions?  Have you looked into Splinter at all and PyTest Splinter plugin?  Are you?  I haven't.  It's a convenience wrapper library in Python for Selenium which helps you write better  asserts without going into the markup.  So it's like instead of putting.sl markup everywhere, it helps you to write better tests  that are less brittle.  So maybe just a hint, it makes it easier to write nicer tests.  Cool.  I'm curious to talk to you about that afterwards.  Thanks.  How did you version your page objects if you were making changes in the application?  How did you deal with having different versions of and like regression testing with Selenium?  Good one.  So what we're trying now is trying to get an element and if you can't get that hook  element, you skip the tests and market the skip and say that this test was skipped until  that code reaches the environment you're trying to test.  Any other questions?  Okay.  If that's all, let's say thank you to Andre.  Thank you.
Andrei_Coman_-_Effectively_test_your_webapp_with_Python_and_Selenium_0_359.png Andrei_Coman_-_Effectively_test_your_webapp_with_Python_and_Selenium_360_407.png Andrei_Coman_-_Effectively_test_your_webapp_with_Python_and_Selenium_408_455.png Andrei_Coman_-_Effectively_test_your_webapp_with_Python_and_Selenium_456_635.png Andrei_Coman_-_Effectively_test_your_webapp_with_Python_and_Selenium_636_779.png Andrei_Coman_-_Effectively_test_your_webapp_with_Python_and_Selenium_780_1655.png Andrei_Coman_-_Effectively_test_your_webapp_with_Python_and_Selenium_1656_4451.png Andrei_Coman_-_Effectively_test_your_webapp_with_Python_and_Selenium_4452_5003.png Andrei_Coman_-_Effectively_test_your_webapp_with_Python_and_Selenium_5004_5363.png Andrei_Coman_-_Effectively_test_your_webapp_with_Python_and_Selenium_5364_8171.png Andrei_Coman_-_Effectively_test_your_webapp_with_Python_and_Selenium_8172_13463.png Andrei_Coman_-_Effectively_test_your_webapp_with_Python_and_Selenium_13464_13619.png Andrei_Coman_-_Effectively_test_your_webapp_with_Python_and_Selenium_13620_14243.png Andrei_Coman_-_Effectively_test_your_webapp_with_Python_and_Selenium_14244_14339.png Andrei_Coman_-_Effectively_test_your_webapp_with_Python_and_Selenium_14340_15359.png Andrei_Coman_-_Effectively_test_your_webapp_with_Python_and_Selenium_15360_17027.png Andrei_Coman_-_Effectively_test_your_webapp_with_Python_and_Selenium_17028_17279.png Andrei_Coman_-_Effectively_test_your_webapp_with_Python_and_Selenium_17280_17843.png Andrei_Coman_-_Effectively_test_your_webapp_with_Python_and_Selenium_17844_18263.png Andrei_Coman_-_Effectively_test_your_webapp_with_Python_and_Selenium_18264_19895.png Andrei_Coman_-_Effectively_test_your_webapp_with_Python_and_Selenium_19896_21611.png Andrei_Coman_-_Effectively_test_your_webapp_with_Python_and_Selenium_21612_23735.png Andrei_Coman_-_Effectively_test_your_webapp_with_Python_and_Selenium_23736_29819.png Andrei_Coman_-_Effectively_test_your_webapp_with_Python_and_Selenium_29820_30050.png