FIELD OF THE INVENTION

The disclosure belongs to the area of data scraping. More specifically, the disclosures present an efficient way to analyze the history of the exit nodes, organized in groups, or pools, and predict their performance and behavior as a group to receive advanced web scraping results.

BACKGROUND OF THE INVENTION

A proxy server acts as an intermediary for requests from the user seeking resources from target servers. A user connects to the proxy server, requesting data. The proxy server evaluates the request and forwards the request to the actual target system or systems containing the corresponding data. After obtaining the data, the proxy server normally forwards the data to the original requestor, but other actions can also be performed by the proxy (e.g., return error message, etc.). Depending on the type of request, a proxy server may or may not have full visibility into the actual content fetched to the original requestor, e.g., in case of an encrypted Hypertext Transfer Protocol Secure (HTTPS) session, the proxy may serve as an intermediary, blindly forwarding the data without being aware of what is being forwarded.

The proxies can be divided into different types depending on what functions are provided or what servers are used. The proxies can also be divided into Residential Internet Protocol (IP) proxies, Datacenter IP proxies, and Mobile IP proxies. A Residential IP address is an address from the range specifically designated by the owning party as assigned to private customers. Usually a Residential proxy is an IP address linked to a physical device, for example, mobile phone or desktop computer, however businesswise the blocks of Residential IP addresses may be bought from the owning Proxy Service Provider by another company directly, in bulk. The real owners of the Residential IP address ranges, namely Internet service providers (ISPs), register residential IP addresses in public databases, which allows websites to determine a device's internet provider, network, and location. Datacenter IP proxy is the proxy server assigned with a datacenter IP. Datacenter IPs are IPs owned by companies, not by individuals. The datacenter proxies are actually IP addresses that are not located in a natural person's home. Instead, the datacenter proxies are associated with a secondary corporation. Mobile IP proxies may be considered a subset of the Residential proxy category. A mobile IP proxy is essentially one IP address that is obtained from mobile operators. Mobile IP proxies use mobile data, as opposed to a residential proxy that uses broadband ISPs or home Wi-Fi. When a user sends a request, the request goes to the proxy server first. The proxy server then makes the request on the user's behalf, collects the response from the web server, and forwards the web page data so that the user can access the page. When the proxy server forwards the requests, it can make changes to the data but yet still provide the data requested. A proxy server changes the user's IP address, so the web server is not provided with the geographical location of the user. A proxy can encrypt the user's data so that the user's data becomes unreadable in transit. Also, a proxy server can block access to certain web pages, based on IP address, domain name, or the communication protocol parameters, such as the port requested.

Exit node proxies, or simply exit nodes, are gateways where the traffic hits the Internet. There can be several proxies used to perform a user's request, but the exit node proxy is the final proxy that contacts the target and forwards the information from the target to the user. There can be several proxies serving the user's request, forming a proxy chain, passing the request through each proxy, with the exit node being the last link in the chain that ultimately passes the request to the target.

In the current embodiments, proxies and exit nodes can be used as synonyms. The current embodiments are not limited only to the exit nodes and same technologies can be used for proxies of different types. However, the term exit node is employed in the current description to clarify the functional differences between exit nodes and proxies.

A proxy provider can control the quality of proxies and decide which IP addresses are going to be used for users in a set of proxies. If the same proxy is used for too many requests, it will ultimately be banned by the ISP or the web page and it will not be possible to use such a proxy to make subsequent requests. If too many requests come in from one IP address in a short period of time, then the site will block the requests from that proxy for a pre-set period of time.

This problem is most often encountered in web scraping. Web scraping (also known as screen scraping, data mining, web harvesting) in its most general sense is the automated gathering of data from the Internet. More technically, it is the practice of gathering data from the internet through any means other than a human using a web browser or a program interacting with an application programming interface (API). Web scraping is usually accomplished by writing a program that queries a web server and requests data automatically, then parses the data to extract the requested information.

Web scrapers—programs written for web scraping—can have a significant advantage over other means of accessing information, like web browsers. The latter are designed to present the information in a readable way for humans, whereas web scrapers are excellent at collecting and processing large amounts of data quickly. Rather than opening one page at a time through a monitor (as web browsers do), web scrapers are able to view large databases consisting of thousands or even millions of pages at once.

Sometimes the website allows another automated way to transfer its structured data from one program to another via an API. Typically, a program will make a request to an API via Hypertext Transfer Protocol (HTTP) for some type of data, and the API will return this data from the website in the structured form. It serves as a medium to transfer the data. However, using APIs is not considered web scraping since the API is offered by the website (or a third party) and it removes the need for web scrapers.

An API can transfer well-formatted data from one program to another and the process of using it is easier than building a web scraper (a bot) to get the same data. However, APIs are not always available for the needed data. Also, APIs often use volume and rate restrictions and limit the types and the format of the data. Thus, a user would use web scraping for the data for which an API does not exist or which is restricted in any way by the API.

Usually, web scraping includes the following steps: retrieving Hypertext Markup Language (HTML) data from a website; parsing the data for target information; saving target information; repeating the process if needed on another page. A program that is designed to do all of these steps is called a web scraper. A related program—a web crawler (also known as a web spider)—is a program or an automated script which performs the first task, i.e. it navigates the web in an automated manner to retrieve raw HTML data of the accessed web sites (the process also known as indexing).

There are techniques that websites use to stop or slow down a hot since scraping may overload the website. For example, they may try to identify the IP address of the bot and block it to prevent further access by the bot. To do that, the website needs to identify the hot-like behavior of the web scraper and to identify its IP address.

Recognizing the hot-like behavior can be done in multiple ways. One involves a limit on the rate of actions (or actions over time) since humans normally perform less actions than a bot would. To circumvent this, web scrapers often choose to employ proxies which mask the real IP address of the web scraper and perform web scraping through multiple proxy IP addresses at the same time to both keep up the gathering speed and avoid being blocked.

Another instance, in which the quality of the proxy is important is media streaming. Media can be distributed on the internet in one of three ways: full download, progressive download, and streaming. Full download has the disadvantage of having to wait until the download is complete to view the media. It is only usable for non-real-time media. Progressive download is applied to the parts of the media progressively, so the user can begin viewing the media before it is fully downloaded. Progressive download is another method for non-real-time media distribution because a significant pre-load is usually required to begin viewing the media. Streaming, conversely, is real-time media distribution. It works by filling out a playout buffer at the client (usually a few seconds) and playing from the buffer as soon as it fills up. The advantage of streaming is almost instantaneous distribution of the media content.

However, streaming is extremely susceptible to network impairments, like packet loss and jitter. More specifically, if the rate of transmission of the media is lower than the playback rate, the playback buffer runs out and the media freezes until the buffer is filled out again.

Media streaming companies strive to offer an uninterrupted experience to its users. They want playback to start instantly and to not stop unexpectedly in any network environment. One way to do that is to reduce the buffer size by compressing the media in more efficient ways. The other way is to ensure that the connection is stable and reliable and the servers can handle the load.

A key issue in media streaming is connection reliability. Reliability, formally speaking, is the ability of the connection to function under stated conditions for a period of time. Put simply, reliability means a connection should work and continue working in a way that supports continuous streaming of the media.

The same criteria apply to proxy connections. When a proxy user requests media streaming, the proxy can become the bottleneck for the user's connection. This means that the proxy user's media streaming connection will be as fast and as reliable, as the exit node's connection is. Thus, a proxy provider has an interest in finding the fastest and most reliable exit nodes for the user that requests media streaming. If the exit node's speed is slow or unreliable, the user will experience jitters, stoppage, or a drop in media quality.

Proxy providers face a set of problems related to both scraping and streaming activities. For scraping, it is important to provide the user with the exit nodes that have a lower probability of being blocked. For streaming, it is important that the exit nodes provided to the user are reliable and fast. Otherwise, the media streaming experience will be hindered because of the quality of the proxies used.

Moreover, if the exit node disconnects during a session, thus interrupting the established path toward the content provider, the user has to reconnect to the streaming service. If upon reconnection the user is being assigned a different exit node, the website may demand to repeat authentication or to confirm human interaction. The repeated authentication also interrupts the quality of service.

To solve at least these problems, in one aspect, the present embodiments detailed herein store and employ an exit node (a proxy) performance and attribute history at a service provider's side. By analyzing an exit node's history, a service provider is able to both heuristically predict the performance and reliability of the exit nodes that are being assigned as well as ensure that the same exit nodes are reserved for a user over time, thus maximizing the efficiency of the exit node pool.

SUMMARY OF THE INVENTION

The present embodiments analyze the history of the exit nodes, organized in groups, or pools, and predicting their performance and behavior as a group.

In one instance, the history allows proxy service providers to assign an exclusive proxy pool to a user, where the pool is not shared with other users. In another instance, it allows to heuristically predict the performance and behavior of the exit nodes, or groups of exit nodes, so that new users are assigned the potentially best fitting exit nodes. The heuristic prediction also allows to predict a risk factor associated with connection reliability and to address these risks before they cross a certain threshold assigned to each user.

The success of web scraper operations highly depends on the scraping functionality presenting their activities as organic traffic to the website. The proxy service provider can help the scraping request appear organic and human-like by relaying said requests through exit nodes that exhibit attributes typical of organic users. This is enabled by collecting historical data of the exit node, such as its geographical location, IP type, and usage history with a target website. Using collected and aggregated historical data, a proxy provider is able to predict which exit nodes will be more successful at scraping a particular website at a given time or in the future.

Similarly, in at least one embodiment, historical data is used to predict which exit nodes will have the best speed and reliability required by streaming activities through a proxy server. Using the collected historical data about an exit node's average speed, average uptime, connection time and duration, average amount of traffic per day, variations in which median and percentile groups are used instead of average values, and other similar attributes the service provider is able to heuristically predict which exit nodes will provide the fastest and the most reliable connection for a streaming session.

Once the best fitting exit nodes are identified, the service provider ensures that these exit nodes are exclusive to the user that made the initial request. Otherwise, the best rated proxies would be assigned a disproportionately heavy load and their performance would decrease. Once an exit node disconnects from the current pool, another one is taken from the exclusive group. If no proxies are available in the exclusive group, a new one is found in the non-exclusive groups and assigned to exclusive. This ensures that a user is always working with the best quality exclusive exit nodes and the efficient utilization of the exit nodes is maximized.

In at least one embodiment, exit nodes report their utilization data to the proxy server management platform which aggregates the data and uses it to predict future transformations of the current pool. It predicts whether the pool will increase or decrease in quality over a period of time. This enables a proxy server provider to refill the exclusive group candidate queue before the current exit nodes disconnect, thus addressing connection reliability risks before they cause issues for the user.

