Frontera: open source large-scale web crawling framework
CC Attribution - NonCommercial - ShareAlike 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this
Sibiryakov, Alexander
Alexander Sibiryakov - Frontera: open source large-scale web crawling framework In this talk I'm going to introduce Scrapinghub's new open source framework [Frontera]. Frontera allows to build real-time distributed web crawlers and website focused ones. Offering: - customizable URL metadata storage (RDBMS or Key-Value based), - crawling strategies management, - transport layer abstraction. - fetcher abstraction. Along with framework description I'll demonstrate how to build a distributed crawler using [Scrapy], Kafka and HBase, and hopefully present some statistics of Spanish internet collected with newly built crawler. Happy EuroPythoning!
 I would just hand it over to him. He will present us Frontera large scale open source webcrawling frameworks. Welcome Alex. Thank you. Thanks. Hololos participantes. So a few words about myself. I was born in Yereinburg in Russia. It's like in the middle about 1,500 kilometers from Moscow to the east. I was working five years at Yandex. Yandex is a so-called Russian Google number one search giant in Russia. I was working in search quality department and was responsible for development of social search QA search. At the moment, like So we had access to the whole Twitter data. So we built our search based on Twitter data. Later I moved to Czech Republic and worked two years at the Vastantivirus. This is like Now later the most popular one in the world. It has about 200 million users and I was responsible for automatic false positive solving and a large-scale prediction of malicious download attempts. So let's go with frontier. I put this quote here because crawl frontier became such a common term in the webcrawling society. So basically Crawling works that way you pull the seeds in crawler starts to go there gets some links from there and then continues to get these links out of there. So Where these links are stored before we will be fetched is called frontier. So here this term comes from shipping. Obviously all the Spanish guys knows what is frontier. But I just realized that frontier is not so or frontier is not so used word. Especially in the countries when where is no sea. So this is a place where all the stuff is like people and goods are Before they go to the land or to the sea. So a few words about motivation. Why we decided to build frontier. We had a client. We came to us and said one billion of pages per week. So and then we want you to process these pages and Tell us what are the biggest hubs frequently changing. So we just have a look at it. One billion what does it mean. It means 150 millions per day and About about one thousand and a half per second. So that was quite a lot. Later I will show you that current scrappy abilities is kind of One thousand and five hundred pages per minute not per second. So here you see the very important picture. It's an illustration of hyperlink induced topic search from joint Kleinberg. Hubs are the nodes with lots of outgoing links. But bigger nodes are a 40 sites which has lots of incoming links. It's similar to the scientific publications. If you are excited much it means society gives you support. How to calculate hub and afford to score on a link graph. And that became history. So now every major search system is using this to rank the pages and To research the link graph. And Another thing is like It's not like scrappy wasn't suitable for broad crawls. That's not true. But Like broad crawls with scrappy was hard really hard. And nobody does that. So Like favored Apache Nudge instead of And we did not like that. So we wanted to make them able to crawl with scrappy whatever they want. So there are two modes of execution of frontera. Single threaded and distributed. Frontera is mostly all about what to crawl next and when it's basically Guiding the crawler what to do next. Single threaded mode for up to 100 websites. This is a proper value because It heavily depends on intensity of your parsing task. Your documents can be can connect can have like a lot of links which is like Gives more documents or your websites can be like Less like Not so responsive as others. So and also Sometimes Sometimes Additional post processing which is also CPU intensive It's basically all about spew for performance broad crawls. There is a distributed node Here's the main features of single threaded version The main feature from my point of view is like it's real time to nudge. What does that mean? It's like When you work with nudge first thing you do is like you pull the seeds and then you run crawling of one second Then the whole thing stops. You need to run The command to process what was crawled to generate a new links to crawl like new batch and then continue with crawling So it's like a batched with and always has steps Frontera is opposite. Everything is online. So it never stops and It matches every batch like at the end of every batch Patch is requested and then it's continued. Therefore, we avoid waiting for last URLs which are like taking too long to download and this is like For those of you who has experience with your crawling, you probably know about it. Actually, can you raise the hands like who was doing Broad crawls before? Okay, one person. Okay, who knows about Scrappy? Okay, it's much better Well, uh, another thing is we have a storage abstraction. So You have out of the box you have SQLhami and HBase SQLhami means you can plug in popular database, you know MySQL Osgress Oracle and so on Or you can implement your own. There is like pretty straightforward interface Third thing we have canonical URLs resolution abstraction this is like usually underestimated problem and You have each page if you Just take it as a as a Unique content each page from each website can have many URLs Yeah, so it's always a question which one to use if you Find the same content by using two URLs And will not pay attention to this you will end up with duplicates in your database And here we provide an interface to implement your own canonical URLs So It could be different depending on your application And the last thing is like Scrappy ecosystem. We have a big community Good documentation. I believe And it's like really easy to customize mostly because of python So Benefit from from terra when you have a need of A few real metadata storage or content storage, so you have a website and you want to show the content Or you have internet and you want to show the content from database or from metadata so basically Right right right thing to do Even second thing is when you want to isolate your url ordering or queuing from the spider And the third thing when they have like pretty advanced url ordering logic Big websites or you want Big websites means like If your website is so big and there is no way to crawl it full You can Adjust crawling logic. So it will like select the best pages to crawl Here's the architecture single-tradit version Let's go probably from right to left You see the database and you see the back end back end is responsible for communication with database Mostly Also in back end it is coded the model For url ordering and queuing. So you just It's tightly connected with type of storage you use therefore it's in the back end Frontera middlewares allows you to modify the contents of requests or responses As like as you want. So you can put your fingerprinting of url or you can Change the meta fields add scoring fields or another thing you need Fintera API is basically the API looking outside of Frontera framework, which is like Which is like possible to use by any other process management code or crawler So crawler is basically the stuff which makes dns resolution and fetching the content from the web So You can Put anything you want here Obviously we have everything for scrappy And also we have a Example for February Just to demonstrate that Frontera is working well outside of scrappy and sites is the internet Fill the shots during about I put that image here because it's like How we are friending with scrappy So basically Frontera is implemented as a set of custom scheduler and spider middleware for scrappy so All that stuff is pluggable and Frontera doesn't require scrappy can be used separately Mostly scrappy is used for process management and fetching. Yeah, so And of course we are friends forever So guys from scrappy always like attacking me like to Well, not from scrappy from scraping up always attacking me like Let's integrate it even more So my task is like to stand against that Because it's like I have to think about the community and So here's like a short quick start To try Frontera in single thread mode First you have to install it then you have to write a simple spider Maybe like 20 lines of code including imports and Well, you can take example one Edit spider settings by and put scheduler and frontera spider middleware there. So scrappy What scheduler to get and scheduler later will load all the frontera stuff And the API crawl And then finish with that That's it. Check the database if you use database Here is a list of use cases for distributed version It's like completely different story Single version is meant for like maybe 50 or 100 websites and you know all these websites But when you like have a broad crawl, you don't know what you will face So If you have set of urals and you need to revisit them like set of urals, I mean like hundreds of thousands And if you are building your search engine and you need to get content somewhere If you are doing some research on web graph frontera also could be useful Therefore like you don't need to save content which is like making work a bit easier You have a topic and you want to crawl the documents about the topic imagine like you have like You want to crawl about sport cars So you run the frontera After sometime you have a lot of documents much better than google because google will show you like only first few pages and Still it's like how to get this page out of google More general focus crawling tasks as I mentioned previously like if you have If you want to search at some peak for a big cup You probably will get benefit from frontera So here's the architecture distributed version. Let's go from scrappy You pull I will just describe the data flow and Operation how all this stuff works. So you pull your seats in spiders Above then these seats are passed to spider lock by means of catheter transport This is our kevka topics And then from kevka we will get to strategy worker and db worker strategy worker is responsible for all the scoring stuff and for Making decision when do we have to stop the crawling? It's like when crawling goal is achieved db worker is responsible for building new Urals or old ones doesn't matter and producing new batches Scoring block is a place where all the scores about urals are passed to the db worker so Seats are going to strategy worker and db worker strategy worker saves Sites that Are new urals and we have to crawl them Calculate score for them score is propagated to db worker and db worker Is making a new batch for them New To spiders and spiders are now loading this batch of urals after that we get a We get a content and we send this content. Well, we actually do also parsing and then we send this content by means on spider lock Again to strategy worker and db worker Strategy worker extracts links Look at them if they are new they needs to be scheduled It calculates score again puts this quarter scoring clock and db worker is saving the information about what was downloaded and so on so basically we have a closed circle So actually Now i'm running out of time so You can put any strategy you want in strategy worker It's implemented in python strategy worker and db workers and Well, let's go Here's the main features of distributed frontier. Well, we use Kafka as a communication layer and we use a rolling strategy abstraction As I mentioned in strategy workers, so you can implement your cloning goal your ordering scoring model In separate model polite by design. It means you will not get blocked because Your That site will be downloaded by at most one spider this is achieved by means of partitioning in Kafka I'm the Python everything is in Python The requirements so you need to have hbase and Kafka scrappy 024 at least First two is easier to get by installing cloud era cdh DNS service Because we are making DNS intensive stuff. So it's better than your DNS service will be pointing top stream service servers like some big ones from big providers Maybe American Bayzone or open DNS Hard requirements Quite interesting slide So here's like how to calculate From your need what hardware you need for her for frontier Typically each spider gives you one page per minute. It's including parsing and Spiders Is about four to one. So here's an example if you have 12 spiders That will give you 14,000 pages per minute 14,000 pages per minute. It means three strategy workers and three db workers total 18 cores Because each worker will consume one core Well memory would be would be nice also for strategy workers So some gotchas Well, I would better skip this If you want Because we are running out of time So here's like A short Quick start, but it's not quick at all So prepare hbase and Kafka and install distributed frontera I think if you have hbase and Kafka you will need like two three hours to get it running from scratch So it's like all the instructions are mostly at this website Of course, we will be like Working more on this at the moment documentation is like he's not the best It's its best state So we made a quick spanish crawl. I just told like before the presentation so To test from terror, we just wanted to find out what are you guys doing here in spain besides playing football so We decided to check out what are your biggest websites? And I just took from demos all the spain spanish content all the spanish urals and put pool them as a seeds Having like 12 spiders And running in this for one and a half months So probably You are now at least one of these websites at the top and At the after all we crawled about 47 millions of pages We know that you have at least 22 websites with more than a hundred pages it's like a lot but considering this count of them domains found it's like We should found much more I think Here's some future plans We won't definitely revisit in strategy out of the box So Yeah It means if you performed a crawl You then probably you need to recrawl it for some to get What what was the changes in your content? so And also you want to recrawl it by some ordering Which is based on how content is changing Yeah page rank and hits based I already told about peds page rank is just another link algorithm link graph algorithm We want our own neural parsing It's like scrappy So we will I guess we will get it soon because of that And yeah, we will test it on larger scales So I have a small question. How do you guys work out canonical? I Don't know if you want to ask questions anyone Okay, so I have a small question. How do you guys work out canonical URLs? Because I think that's that might get really tricky in in some pages so very like few approaches Actually some website webmasters they provide canonical URL in the content so you can get if it is there That's the best if it is not there like What you can do you can you can probably Analyze the structures for example if you have a chain of redirects you can get the last one in the chain and Yeah, basically with some set of heuristics. It's like there is no clear decision The target of front error is to provide interface for this and so That's it actually If you look in You will find out that we are just picking last one from the direct chain This like gives us ability like to avoid duplicate to do the thing We By seban, I have a question sno scrappy have has a web base it's dashboard and Do you spider work with it too? Actually, actually they should work. Yeah, because You can put your own scheduler and spider middleware in these spiders and that potentially should work as I know this the boards should Create some rules and then scrappy uses early so I'm sorry. What rules? Like a rules like like a spider rules, you know Sorry, I don't remember The thing is like, you know, I'm kind of more dedicated to scrolling so I'm Honestly, I'm not well aware of what scrappy is all about yeah, so Let's talk later. I will can't I will just point you to the right guys second question, uh, do you use some asynchronous library? Or uh, sno, uh, uh, if you run your Our application as a single traded Do you do you use some asynchronous? Yes, we use twist it mostly because it helps to Calls to some functions And let just makes code more readable So, yeah, thank you Okay, thank you. Maybe one quick last question before we change rooms. Is there anyone I think otherwise, um, Alex will be outside I can show you something interesting if you don't have questions Okay, really quick. We have 45 I think the next talk we begin and so that That was done 15 years ago This is was done by And I'm not sure if you can see it But it's done by andrey brother and others. This is a They are from yahoo research. This is a structure of the internet They think of So in the middle, we have a strong strongly connected continent They think it's like there are a lot of links Website highly connected inside and here Like a butterfly, you know Here's like incoming links to this strongly connected component Right where like outgoing links and a lot of them and uh, you have This butterfly has a tendrils So it's like a bit like So, um These tendrils they have outgoing links and some tendrils have only Ingoing links like to the Out And they have tubes You can bypass strongly connected component from in links right to the out links so And actually you also have a disconnected stuff That means There is a pages we will never find If we will just go and try to grow the internet So I wish Someday these days Refine we speak For it is true I'll see you perfect. Thank you everyone
 I would just hand it over to him. He will present us Frontera large scale open source webcrawling frameworks.  Welcome Alex. Thank you.  Thanks.  Hololos participantes.  So a few words about myself.  I was born in Yereinburg in Russia. It's like in the middle about 1,500 kilometers from Moscow  to the east.  I was working five years at Yandex. Yandex is a so-called Russian Google number one search giant in Russia.  I was working in search quality department and was responsible for development of social search QA search.  At the moment, like  So we had access to the whole Twitter data. So we built our search based on Twitter data.  Later I moved to Czech Republic and worked two years at the Vastantivirus. This is like  Now later the most popular one in the world.  It has about  200 million users  and I was responsible for automatic false positive solving  and a large-scale prediction of malicious download attempts.  So let's go with frontier.  I put this quote here because  crawl frontier became such a common term in the webcrawling society. So basically  Crawling works that way you pull the seeds in  crawler starts to go there gets some  links from there and then continues to get these links out of there.  So  Where these links are stored before we will be fetched is called frontier.  So here this term comes from shipping.  Obviously all the Spanish guys knows what is frontier.  But I just realized that frontier is not so or frontier is not so used word.  Especially in the countries when  where is no sea.  So this is a place where all the stuff is like people and goods are  Before they go to the land or to the sea.  So a few words about motivation. Why we decided  to build frontier.  We had a client.  We came to us and said  one billion of pages per week.  So and then we want you to process these pages and  Tell us what are the biggest hubs  frequently changing.  So we just have a look at it.  One billion what does it mean. It means 150 millions per day and  About about one thousand and a half per second.  So that was quite a lot.  Later I will show you that current scrappy abilities is kind of  One thousand and five hundred pages per minute not per second.  So here you see the very important picture.  It's an illustration of hyperlink induced topic search from joint Kleinberg.  Hubs are the nodes with lots of outgoing links.  But bigger nodes are a 40 sites which has lots of incoming links.  It's similar to the scientific publications.  If you are excited much it means society gives you support.  How to calculate hub and afford to score on a link graph.  And that became history.  So now every major search system is using this to rank the pages and  To research the link graph.  And  Another thing is like  It's not like scrappy wasn't suitable for broad crawls. That's not true.  But  Like broad crawls with scrappy was hard really hard.  And nobody does that.  So  Like favored Apache Nudge instead of  And we did not like that. So we wanted to make them able to crawl with scrappy whatever they want.  So there are two modes of execution of frontera. Single threaded and distributed.  Frontera is mostly all about what to crawl next and when it's basically  Guiding the crawler what to do next.  Single threaded mode for up to 100 websites. This is a proper value because  It heavily depends on intensity of your parsing task.  Your documents can be can connect can have like a lot of links which is like  Gives more documents or your websites can be like  Less like  Not so responsive as others. So and also  Sometimes  Sometimes  Additional post processing which is also CPU intensive  It's basically all about spew for performance broad crawls. There is a distributed node  Here's the main features of single threaded version  The main feature from my point of view is like it's real time to nudge. What does that mean? It's like  When you work with nudge first thing you do is like you pull the seeds and then you run crawling of one second  Then the whole thing stops. You need to run  The command to process what was crawled to generate a new links to crawl like new batch and then continue with crawling  So it's like  a batched with and always has steps  Frontera is opposite. Everything is online. So it never stops  and  It matches every batch like at the end of every batch  Patch is requested and then it's continued. Therefore, we avoid waiting for  last URLs which are like taking too long to download and this is like  For those of you who has experience with your crawling, you probably know about it. Actually, can you raise the hands like who was doing  Broad crawls before?  Okay, one person. Okay, who knows about Scrappy? Okay, it's much better  Well, uh, another thing is we have a storage abstraction. So  You have out of the box you have SQLhami and HBase  SQLhami means you can plug in popular database, you know  MySQL  Osgress Oracle and so on  Or you can implement your own. There is like pretty straightforward interface  Third thing we have canonical URLs resolution abstraction  this is like  usually underestimated problem and  You have  each page if you  Just take it as a as a  Unique content each page from each website can have many URLs  Yeah, so it's always a question which one to use  if you  Find the same content by using two URLs  And will not pay attention to this you will end up with duplicates in your database  And here we provide an interface to implement your own canonical URLs  So  It could be different depending on your application  And the last thing is like Scrappy ecosystem. We have a big community  Good documentation. I believe  And it's like really easy to customize mostly because of python  So  Benefit from  from terra  when you have a need of  A few real metadata storage or content storage, so you have a website and you want to show the content  Or you have internet and you want to show the content from database or from metadata  so  basically  Right right right thing to do  Even second thing is when you want to isolate your url ordering or queuing from the spider  And the third thing when they have like pretty advanced url ordering logic  Big websites or you want  Big websites means like  If your website is so big and there is no way to crawl it full  You can  Adjust crawling logic. So it will like select the best pages to crawl  Here's the architecture single-tradit version  Let's go probably from right to left  You see the database and you see the back end back end is responsible for communication with database  Mostly  Also in back end it is coded the model  For url ordering and queuing. So you just  It's tightly connected with type of storage you use therefore it's in the back end  Frontera middlewares  allows you to  modify the  contents of requests or responses  As like as you want. So you can put your fingerprinting  of url or  you can  Change the meta fields add scoring fields or another thing you need  Fintera API is basically the  API looking outside of Frontera framework, which is like  Which is like  possible to use by any other  process management code or  crawler  So crawler is basically the stuff which makes  dns resolution and fetching the content from the web  So  You can  Put anything you want here  Obviously we have everything for scrappy  And also we have a  Example for February  Just to demonstrate that Frontera is working well outside of scrappy and sites is the internet  Fill the shots during about  I put that image here because it's like  How we are friending with scrappy  So basically  Frontera is implemented as a set of custom scheduler and spider middleware  for scrappy  so  All that stuff is pluggable and  Frontera doesn't require scrappy can be used separately  Mostly scrappy is used for process management and fetching. Yeah, so  And of course we are friends forever  So guys from scrappy always like attacking me like to  Well, not from scrappy from scraping up always attacking me like  Let's integrate it even more  So my task is like to stand against that  Because it's like I have to think about the community and  So here's like a short quick start  To try Frontera in single thread mode  First you have to install it then you have to write a simple spider  Maybe like 20 lines of code including imports and  Well, you can take example one  Edit spider settings by and put scheduler and frontera spider middleware there. So scrappy  What scheduler to get and scheduler later will load all the frontera stuff  And the API crawl  And then finish with that  That's it. Check the database if you use database  Here is a list of use cases for distributed version  It's like completely different story  Single version is meant for like maybe 50 or 100 websites and you know all these websites  But when you like have a broad crawl, you don't know what you will face  So  If you have set of urals and you need to revisit them like set of urals, I mean like hundreds of thousands  And if you are building your search engine and you need to get content somewhere  If you are doing some research on web graph frontera also could be useful  Therefore like you don't need to save content which is like making work a bit easier  You have a topic and you want to crawl the documents about the topic imagine like you have like  You want to crawl about sport cars  So you run the frontera  After sometime you have a lot of documents much better than google because google will show you like only first few pages and  Still it's like how to get this page out of google  More general focus crawling tasks as I mentioned previously like if you have  If you want to search at some peak for a big cup  You probably will get benefit from frontera  So here's the architecture distributed version. Let's go from scrappy  You pull I will just describe the data flow and  Operation how all this stuff works. So you pull your seats in spiders  Above then these seats are passed to spider lock by means of catheter transport  This is our kevka topics  And then from kevka we will get to strategy worker and db worker  strategy worker is responsible for all the scoring stuff and for  Making decision when do we have to stop the crawling? It's like when crawling goal is achieved  db worker is responsible for  building new  Urals or old ones doesn't matter and producing new batches  Scoring block is a place where all the scores about urals are passed to the db worker  so  Seats are going to strategy worker and db worker  strategy worker saves  Sites that  Are new urals and we have to crawl them  Calculate score for them  score is propagated to db worker and db worker  Is making a new batch  for them  New  To spiders and spiders are now loading this batch of urals  after that  we get a  We get a content and we send this content. Well, we actually do also parsing and then we send this content by means on spider lock  Again to strategy worker and db worker  Strategy worker extracts links  Look at them if they are new they needs to be scheduled  It calculates score again puts this quarter scoring clock and  db worker is saving the information about what was downloaded and so on so basically we have a closed circle  So  actually  Now i'm running out of time  so  You can put any strategy you want in strategy worker  It's implemented in python strategy worker and db workers and  Well, let's go  Here's the main features of distributed frontier. Well, we use Kafka as a communication layer and we use a rolling strategy abstraction  As I mentioned in strategy workers, so you can implement your cloning goal your ordering scoring model  In separate model polite by design. It means you will not get blocked because  Your  That site will be downloaded by at most  one spider this is achieved by  means of partitioning in Kafka  I'm the Python everything is in Python  The requirements so you need to have hbase and Kafka scrappy 024 at least  First two is easier to get by installing cloud era cdh  DNS service  Because we are making DNS intensive stuff. So it's better than your  DNS service will be pointing top stream service servers like some big ones from big providers  Maybe American Bayzone or open DNS  Hard requirements  Quite interesting slide  So here's like how to calculate  From your need what hardware you need for her for frontier  Typically each spider gives you one page per minute. It's including parsing and  Spiders  Is about four to one. So here's an example if you have 12 spiders  That will give you  14,000 pages per minute  14,000 pages per minute. It means three strategy workers and three db workers total 18 cores  Because each worker will consume one core  Well memory would be would be nice also  for strategy workers  So some gotchas  Well, I would better skip this  If you want  Because we are running out of time  So here's like  A short  Quick start, but it's not quick at all  So prepare hbase and Kafka and install distributed frontera  I think if you have hbase and Kafka you will need like two three hours to get it running from scratch  So it's like all the instructions are mostly at this website  Of course, we will be like  Working more on this at the moment documentation is like he's not the best  It's its best state  So we made a quick spanish crawl. I just told like before the presentation  so  To test from terror, we just wanted to find out what are you guys doing here in spain besides playing football  so  We decided to check out what are your biggest websites?  And I just took from demos all the spain spanish content all the spanish urals  and put pool them as a seeds  Having like 12 spiders  And running in this for one and a half months  So probably  You are now at least one of these websites  at the top  and  At the after all we crawled about 47 millions of pages  We know that you have at least 22 websites with more than a hundred pages  it's like  a lot but  considering this count of them domains found it's like  We should found much more I think  Here's some future plans  We won't definitely revisit in strategy out of the box  So  Yeah  It means if you performed a crawl  You then probably you need to recrawl it for some to get  What what was the changes in your content?  so  And also you want to recrawl it by some ordering  Which is based on how content is changing  Yeah page rank and hits based I already told about peds page rank is just another link algorithm link graph algorithm  We want our own neural parsing  It's like scrappy  So we will I guess we will get it soon because of that  And yeah, we will test it on larger scales  So I have a small question. How do you guys work out canonical?  I  Don't know if you want to ask questions anyone  Okay, so I have a small question. How do you guys work out canonical URLs?  Because I think that's that might get really tricky in in some pages  so  very like  few approaches  Actually some website webmasters they provide canonical URL in the content so you can get if it is there  That's the best  if it is not there like  What you can do you can  you can probably  Analyze the structures for example if you have a chain of redirects you can get the last one in the chain and  Yeah, basically with some set of heuristics. It's like there is no clear decision  The target of front error is to provide interface for this  and so  That's it actually  If you look in  You will find out that we are just picking last one from the direct chain  This like gives us ability like to avoid duplicate to do the thing  We  By seban, I have a question sno  scrappy have  has a web base it's dashboard  and  Do you spider work with it too?  Actually, actually they should work. Yeah, because  You can put your own scheduler and spider middleware in these spiders and that potentially should work  as I know this  the boards should  Create some rules and then scrappy uses early so  I'm sorry. What rules?  Like a rules like like a spider rules, you know  Sorry, I don't remember  The thing is like, you know, I'm kind of more dedicated to scrolling so  I'm  Honestly, I'm not well aware of what scrappy is all about  yeah, so  Let's talk later. I will can't I will just point you to the right guys  second question, uh, do you use some asynchronous library?  Or  uh, sno, uh, uh, if you run your  Our application as a single traded  Do you do you use some asynchronous?  Yes, we use twist it mostly because it helps to  Calls to some functions  And let just makes code more readable  So, yeah, thank you  Okay, thank you. Maybe one quick last question before we change rooms. Is there anyone I think otherwise, um, Alex will be outside  I can show you something interesting if you don't have questions  Okay, really quick. We have  45  I think the next talk we begin and  so  that  That was done 15 years ago  This is was done by  And I'm not sure if you can see it  But it's done by andrey brother and others. This is a  They are from yahoo research. This is a structure of the internet  They think of  So in the middle, we have a strong  strongly connected continent  They think it's like there are a lot of links  Website highly connected inside and here  Like a butterfly, you know  Here's like incoming links to this strongly connected component  Right where like outgoing links and a lot of them  and uh, you have  This butterfly has a tendrils  So it's like a bit like  So, um  These tendrils they have outgoing links  and some tendrils have only  Ingoing links like to the  Out  And they have tubes  You can bypass strongly connected component from in links right to the out links  so  And actually you also have a disconnected stuff  That means  There is a pages we will never find  If we will just go and try to grow the internet  So I wish  Someday these days  Refine we speak  For it is true  I'll see you perfect. Thank you everyone
Alexander_Sibiryakov_-_Frontera__open_source_large-scale_web_crawling_framework_0_263.png Alexander_Sibiryakov_-_Frontera__open_source_large-scale_web_crawling_framework_264_347.png Alexander_Sibiryakov_-_Frontera__open_source_large-scale_web_crawling_framework_348_407.png Alexander_Sibiryakov_-_Frontera__open_source_large-scale_web_crawling_framework_408_1979.png Alexander_Sibiryakov_-_Frontera__open_source_large-scale_web_crawling_framework_1980_2039.png Alexander_Sibiryakov_-_Frontera__open_source_large-scale_web_crawling_framework_2040_2951.png Alexander_Sibiryakov_-_Frontera__open_source_large-scale_web_crawling_framework_2952_3731.png Alexander_Sibiryakov_-_Frontera__open_source_large-scale_web_crawling_framework_3732_3839.png Alexander_Sibiryakov_-_Frontera__open_source_large-scale_web_crawling_framework_3840_7223.png Alexander_Sibiryakov_-_Frontera__open_source_large-scale_web_crawling_framework_7224_7643.png Alexander_Sibiryakov_-_Frontera__open_source_large-scale_web_crawling_framework_7644_7691.png Alexander_Sibiryakov_-_Frontera__open_source_large-scale_web_crawling_framework_7692_8963.png Alexander_Sibiryakov_-_Frontera__open_source_large-scale_web_crawling_framework_8964_9155.png Alexander_Sibiryakov_-_Frontera__open_source_large-scale_web_crawling_framework_9156_12419.png Alexander_Sibiryakov_-_Frontera__open_source_large-scale_web_crawling_framework_12420_13427.png Alexander_Sibiryakov_-_Frontera__open_source_large-scale_web_crawling_framework_13428_13727.png Alexander_Sibiryakov_-_Frontera__open_source_large-scale_web_crawling_framework_13728_15047.png Alexander_Sibiryakov_-_Frontera__open_source_large-scale_web_crawling_framework_15048_17495.png Alexander_Sibiryakov_-_Frontera__open_source_large-scale_web_crawling_framework_17496_19079.png Alexander_Sibiryakov_-_Frontera__open_source_large-scale_web_crawling_framework_19080_20111.png Alexander_Sibiryakov_-_Frontera__open_source_large-scale_web_crawling_framework_20112_22547.png Alexander_Sibiryakov_-_Frontera__open_source_large-scale_web_crawling_framework_22548_23279.png Alexander_Sibiryakov_-_Frontera__open_source_large-scale_web_crawling_framework_23280_26711.png Alexander_Sibiryakov_-_Frontera__open_source_large-scale_web_crawling_framework_26712_27515.png Alexander_Sibiryakov_-_Frontera__open_source_large-scale_web_crawling_framework_27516_27683.png Alexander_Sibiryakov_-_Frontera__open_source_large-scale_web_crawling_framework_27684_27827.png Alexander_Sibiryakov_-_Frontera__open_source_large-scale_web_crawling_framework_27828_29579.png Alexander_Sibiryakov_-_Frontera__open_source_large-scale_web_crawling_framework_29580_29687.png Alexander_Sibiryakov_-_Frontera__open_source_large-scale_web_crawling_framework_29688_29807.png Alexander_Sibiryakov_-_Frontera__open_source_large-scale_web_crawling_framework_29808_30503.png Alexander_Sibiryakov_-_Frontera__open_source_large-scale_web_crawling_framework_30504_30731.png Alexander_Sibiryakov_-_Frontera__open_source_large-scale_web_crawling_framework_30732_30863.png Alexander_Sibiryakov_-_Frontera__open_source_large-scale_web_crawling_framework_30864_32435.png Alexander_Sibiryakov_-_Frontera__open_source_large-scale_web_crawling_framework_32436_33731.png Alexander_Sibiryakov_-_Frontera__open_source_large-scale_web_crawling_framework_33732_39023.png Alexander_Sibiryakov_-_Frontera__open_source_large-scale_web_crawling_framework_39024_39071.png Alexander_Sibiryakov_-_Frontera__open_source_large-scale_web_crawling_framework_39072_39143.png Alexander_Sibiryakov_-_Frontera__open_source_large-scale_web_crawling_framework_39144_40103.png Alexander_Sibiryakov_-_Frontera__open_source_large-scale_web_crawling_framework_40104_40151.png Alexander_Sibiryakov_-_Frontera__open_source_large-scale_web_crawling_framework_40152_41003.png Alexander_Sibiryakov_-_Frontera__open_source_large-scale_web_crawling_framework_41004_41315.png Alexander_Sibiryakov_-_Frontera__open_source_large-scale_web_crawling_framework_41316_41776.png