<commit_before>---
layout: post
title: Downloading files from a webserver, and failing.
author: roel_hogervorst
modified: 
categories: blog
excerpt: 
tags: [ hack, curl, purrr, scraping, intermediate]
image:
date: 2017-12-08
---

Recently I wanted to download all the transcripts of a podcast (600+ episodes). 
The transcripts are simple txt files so in a way
I am  not even 'web'-scraping but just reading in 600 or so text files which is 
not really a big deal. I thought. 

This post shows you where I went wrong

Also here is a [picture](https://www.flickr.com/photos/32123311@N00/502155430 "source: flickr, cc-by 2.0 jbloom") I found of scraping. 

![Scraping a plate ]({{ site.github.url }}/images/scraping_a_plate.jpg)


### Webscraping general 
For every download you ask the server for a file and it returns the file (this is also how you normally browse the web btw, your browser requests the pages).

In general it is nice if you ask permission (I did, on twitter and the author was really nice! I recommend it!) and don't 
push the website to its limit. The  servers where these files are hosted are quite beefy and I will 
probably not even make a dent in them, when I'm downloading these files. But still, be gentle. 

**No really, be a responsible scraper and tell the website owners you are scraping (in person or by identifying in the header) and check if it is allowed**

I recently witnessed a demo where someone explained a lot of dirty tricks on how to get over those pesky servers denying them access and generally ignoring good practices and it made me sick...  

Here are some general guides:

- [<NAME> (@hrbrmstr) about scraping and robots.txt](https://rud.is/b/2017/07/28/analyzing-wait-delay-settings-in-common-crawl-robots-txt-data-with-r/)
- [excellent vignette of the httr package, read the part about user-agent](https://cran.r-project.org/web/packages/httr/vignettes/api-packages.html)

### Downloading non-html files
There are multiple ways I could do this downloading:
if I had used rvest to scrape a website I would have set a user-agent
header^[a piece of information we snd with every request that describes who we are]
and I would have used incremental backoff: when the server refuses a connection
we would wait and retry again, if it still refuses we would wait twice as long
and retry again etc.


However, since these are txt files I can just use read_lines^[This is the readr variant of readLines from base-R, it is much faster then the original]
to read the txt file of a transcript and apply further work downstream. 


#### A first, failing approach, tidy but wrong
This was my first approach:

- all episodes are numbered and the transcript files are sequental too, so just a paste0 of "https://first-part-of-link" number".txt" would work.
- put all links as row into dataframe
- apply some purrr magic by mapping every link to a read_lines function (*what? use the read_lines() function on every link* ).


```r
latest_episode <- 636
 

system.time(
 
    df_sn <- data_frame(link = paste0("https:linktowebsite.com/firstpart-",
 

                                      formatC(1:latest_episode, width = 3,flag = 0),".txt")) %>%
 
        mutate(transcript = map(link, read_lines2))
 
)
```

This failed.

Some episodes don't exists or have no transcript (I didn't know). Sometimes the internet connection didn't want to work and just threw me out. Sometimes the server stopped my requests. 

On every of those occasions the process would stop, give an informative error^[really, it did]. But the R-process would stop and I had no endresult.

#### Getting more information to my eyeballs and pausing in between requests

Also I didn't know where it failed. So I created a new function that also sometimes waited (to not overwhelm the server)

```r
## to see where we are this function wraps read_lines and prints the episodenumber
 

read_lines2 <- function(file){
 

    print(file)
 
    if(runif(1,0,1) >0.008)Sys.sleep(5)

    read_lines(file)

}
```

This one also failed, but more informatively, I now knew if it failed on a certain episode.

But ultimately, downloading files from the internet is a somewhat unpredictable process. And it is much easier to just first download all the files and read them in afterwards.

#### A two step approach, download first, parse later.

Also I wanted to let the logs show that I was the one doing the scraping and how to reach me if I was overwhelming the service. 

Enter curl. 
Curl is a library that helps you download stuff, it is used by the httr package and is a wrapper around the c++ package with the same name, *wrapped by Jeroen 'c-plus-plus' Ooms*. 

Since I ran this function a few times I downloaded some of the files, and didn't really want to download every file again, so I also added a check to see if the file wasn't already downloaded^[I thought that was really clever, didn't you?]. And I wanted it to print to the screen, because I like moving text over the screen when I'm debugging.

```r<commit_msg>made the footnote work, I think<commit_after>---
layout: post
title: Downloading files from a webserver, and failing.
author: roel_hogervorst
modified: 
categories: blog
excerpt: 
tags: [ hack, curl, purrr, scraping, intermediate]
image:
date: 2017-12-08
---

Recently I wanted to download all the transcripts of a podcast (600+ episodes). 
The transcripts are simple txt files so in a way
I am  not even 'web'-scraping but just reading in 600 or so text files which is 
not really a big deal. I thought. 

This post shows you where I went wrong

Also here is a [picture](https://www.flickr.com/photos/32123311@N00/502155430 "source: flickr, cc-by 2.0 jbloom") I found of scraping. 

![Scraping a plate ]({{ site.github.url }}/images/scraping_a_plate.jpg)


### Webscraping general 
For every download you ask the server for a file and it returns the file (this is also how you normally browse the web btw, your browser requests the pages).

In general it is nice if you ask permission (I did, on twitter and the author was really nice! I recommend it!) and don't 
push the website to its limit. The  servers where these files are hosted are quite beefy and I will 
probably not even make a dent in them, when I'm downloading these files. But still, be gentle. 

**No really, be a responsible scraper and tell the website owners you are scraping (in person or by identifying in the header) and check if it is allowed**

I recently witnessed a demo where someone explained a lot of dirty tricks on how to get over those pesky servers denying them access and generally ignoring good practices and it made me sick...  

Here are some general guides:

- [<NAME> (@hrbrmstr) about scraping and robots.txt](https://rud.is/b/2017/07/28/analyzing-wait-delay-settings-in-common-crawl-robots-txt-data-with-r/)
- [excellent vignette of the httr package, read the part about user-agent](https://cran.r-project.org/web/packages/httr/vignettes/api-packages.html)

### Downloading non-html files
There are multiple ways I could do this downloading:
if I had used rvest to scrape a website I would have set a user-agent
header^[a piece of information we snd with every request that describes who we are]
and I would have used incremental backoff: when the server refuses a connection
we would wait and retry again, if it still refuses we would wait twice as long
and retry again etc.


However, since these are txt files I can just use read_lines^[This is the readr variant of readLines from base-R, it is much faster then the original]
to read the txt file of a transcript and apply further work downstream. 


#### A first, failing approach, tidy but wrong
This was my first approach:

- all episodes are numbered and the transcript files are sequental too, so just a paste0 of "https://first-part-of-link" number".txt" would work.
- put all links as row into dataframe
- apply some purrr magic by mapping every link to a read_lines function (*what? use the read_lines() function on every link* ).


```r
latest_episode <- 636
 

system.time(
 
    df_sn <- data_frame(link = paste0("https:linktowebsite.com/firstpart-",
 

                                      formatC(1:latest_episode, width = 3,flag = 0),".txt")) %>%
 
        mutate(transcript = map(link, read_lines2))
 
)
```

This failed.

Some episodes don't exists or have no transcript (I didn't know). Sometimes the internet connection didn't want to work and just threw me out. Sometimes the server stopped my requests. 

On every of those occasions the process would stop, give an informative error^[really, it did]. But the R-process would stop and I had no endresult.

#### Getting more information to my eyeballs and pausing in between requests

Also I didn't know where it failed. So I created a new function that also sometimes waited (to not overwhelm the server)

```r
## to see where we are this function wraps read_lines and prints the episodenumber
 

read_lines2 <- function(file){
 

    print(file)
 
    if(runif(1,0,1) >0.008)Sys.sleep(5)

    read_lines(file)

}
```

This one also failed, but more informatively, I now knew if it failed on a certain episode.

But ultimately, downloading files from the internet is a somewhat unpredictable process. And it is much easier to just first download all the files and read them in afterwards.

#### A two step approach, download first, parse later.

Also I wanted to let the logs show that I was the one doing the scraping and how to reach me if I was overwhelming the service. 

Enter curl. 
Curl is a library that helps you download stuff, it is used by the httr package and is a wrapper around the c++ package with the same name, *wrapped by Jeroen 'c-plus-plus' Ooms*. 

Since I ran this function a few times I downloaded some of the files, and didn't really want to download every file again, so I also added a check to see if the file wasn't already downloaded^[I thought that was really clever, didn't you?] . And I wanted it to print to the screen, because I like moving text over the screen when I'm debugging.

```r
{'fraction_non_alphanumeric': 0.06873570078583507, 'fraction_numerical': 0.012135680891276236, 'mean_word_length': 3.868765133171913, 'pattern_counts': {'":': 0, '<': 11, '<?xml version=': 0, '>': 9, 'https://': 8, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 28, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '3435456', 'n_tokens_mistral': 2967, 'n_tokens_neox': 2689, 'n_words': 1508}