<commit_before># Step 3

## Setup and run instructions
* ```$ yarn install```
* ```$ docker run -v <absolute path>/ecs_s3_scraper/step3/:/workspace ecs_s3_scraper:latest index.js "code and coffee vancouver"```

*Note: We are using the yarn package manager from now onwards because our docker image uses it, and it is also better than npm*

## Development log

### Search for a nightmarejs docker image on [hub.docker.com](https://hub.docker.com)
![Searching for nightmare on hub.doker.com](images/searching_for_nightmare_on_docker_hub.png)


### This one looks good.
![ivanvanderbyl/docker-nightmare docker image](images/ivanvanderbyl_docker-nightmare.png)
1. The setup instructions make sense.
2. The github repo is linked.
3. The Dockerfile seems reasonable, although versions don't seem to be well defined.


### Install docker
[Docker CE and EE for various platforms](https://www.docker.com/get-docker)

[I installed Docker CE for Mac](https://store.docker.com/search?type=edition&offering=community)


### Pull the nightmare docker image
```$ docker pull ivanvanderbyl/docker-nightmare ```


### Create Dockerfile and copy/paste the usage instructions into it
```
FROM quay.io/ivanvanderbyl/docker-nightmare:latest

ADD . /workspace
RUN yarn install
```
[*Above code has been referenced from here*](https://hub.docker.com/r/ivanvanderbyl/docker-nightmare/)

*Note: The Dockerfile uses yarn instead of npm to install the dependencies/libs. We can also use yarn instead of npm on our local machines. To know more about the differences between yarn and npm, [click here](https://www.sitepoint.com/yarn-vs-npm/)*


### Build the new docker image
```$ docker build -t ecs_s3_scraper . ```


### Run the scraper in a docker container
```$ docker run ecs_s3_scraper index.js "code and coffee vancouver" ```


### Wait, where is my HTML file ??!!
> It has been written to the filesystem of the docker container.
> More specifically, the /workspace directory of the container.
> The docker container turned off after running the command that it was supposed to.

We can prove this my doing to following
```
$ docker container ls -a
docker export <container_id> -o "scraper.tar.gz"
tar -xvf scraper.tar.gz -C scraper
ls scraper
```


### Mapping the local directory to docker container is the easiest way to output the HTML file on the local machine
[Explanation of how to do this from Docker docs](https://docs.docker.com/engine/tutorials/dockervolumes/#mount-a-host-directory-as-a-data-volume)

Basically,

```$ docker run -v /Users/<path>:/<container path> ... ```

Let's give this a try:

```$ docker run -v <absolute path>/ecs_s3_scraper/step3/:/workspace ecs_s3_scraper:latest index.js "code and coffee vancouver" ```
*Note: The directories have to be absolute path :(*<commit_msg>fixing formatting issue with readme<commit_after># Step 3

## Setup and run instructions
* ```$ yarn install```
* ```$ docker run -v <absolute path>/ecs_s3_scraper/step3/:/workspace ecs_s3_scraper:latest index.js "code and coffee vancouver"```

*Note: We are using the yarn package manager from now onwards because our docker image uses it, and it is also better than npm*

## Development log

### Search for a nightmarejs docker image on [hub.docker.com](https://hub.docker.com)
![Searching for nightmare on hub.doker.com](images/searching_for_nightmare_on_docker_hub.png)


### This one looks good.
![ivanvanderbyl/docker-nightmare docker image](images/ivanvanderbyl_docker-nightmare.png)
1. The setup instructions make sense.
2. The github repo is linked.
3. The Dockerfile seems reasonable, although versions don't seem to be well defined.


### Install docker
[Docker CE and EE for various platforms](https://www.docker.com/get-docker)

[I installed Docker CE for Mac](https://store.docker.com/search?type=edition&offering=community)


### Pull the nightmare docker image
```$ docker pull ivanvanderbyl/docker-nightmare ```


### Create Dockerfile and copy/paste the usage instructions into it
```
FROM quay.io/ivanvanderbyl/docker-nightmare:latest

ADD . /workspace
RUN yarn install
```
[*Above code has been referenced from here*](https://hub.docker.com/r/ivanvanderbyl/docker-nightmare/)

*Note: The Dockerfile uses yarn instead of npm to install the dependencies/libs. We can also use yarn instead of npm on our local machines. To know more about the differences between yarn and npm, [click here](https://www.sitepoint.com/yarn-vs-npm/)*


### Build the new docker image
```$ docker build -t ecs_s3_scraper . ```


### Run the scraper in a docker container
```$ docker run ecs_s3_scraper index.js "code and coffee vancouver" ```


### Wait, where is my HTML file ??!!
> It has been written to the filesystem of the docker container.
> More specifically, the /workspace directory of the container.
> The docker container turned off after running the command that it was supposed to.

We can prove this my doing to following
```
$ docker container ls -a
docker export <container_id> -o "scraper.tar.gz"
tar -xvf scraper.tar.gz -C scraper
ls scraper
```


### Mapping the local directory to docker container is the easiest way to output the HTML file on the local machine
[Explanation of how to do this from Docker docs](https://docs.docker.com/engine/tutorials/dockervolumes/#mount-a-host-directory-as-a-data-volume)

Basically,

```$ docker run -v /Users/<path>:/<container path> ... ```

Let's give this a try:

```$ docker run -v <absolute path>/ecs_s3_scraper/step3/:/workspace ecs_s3_scraper:latest index.js "code and coffee vancouver" ```

*Note: The directories have to be absolute path :(*
{'fraction_non_alphanumeric': 0.11025595131555396, 'fraction_numerical': 0.004295686414891713, 'mean_word_length': 4.610441767068273, 'pattern_counts': {'":': 0, '<': 13, '<?xml version=': 0, '>': 19, 'https://': 12, 'lorem ipsum': 0, 'www.': 4, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1496369', 'n_tokens_mistral': 1817, 'n_tokens_neox': 1661, 'n_words': 642}