<commit_before># web.dev data server

Server that handles the web.dev profile page bits:

- Runs Lighthouse against a URL and stores reports over time.
- Cron job that schedules tasks to update scores for each URL.
- A demo page and set of example web components that interact with the backend in `public/`.

<img width="1081" alt="screen shot 2018-10-11 at 9 09 35 am" src="https://user-images.githubusercontent.com/238208/46818166-679b6c80-cd35-11e8-8989-d5b1f50abd99.png">

## Development

1. Install it:

```
npm i
```

2. Decrypt the service account and memcache json files:

```
npm run decrypt
```

You will be prompted to for a passphrase to decrypt `serviceAccount.json.enc`
and `memcacheCredentials.json.enc`. These files are required for the backend
to communicate with Firestore and the Redis memcache service. The password can
be found in Google's shared password tool.

3. Run the web server:

```
npm run start
````

4. Build it

To build the CSS/JS bundles, run of `npm run build:js`, `npm run build:css`,
or the single command:

```
npm run build
```

There are also watch tasks if you want to iterate on the files while the server
is running:

```
npm run watch:css
npm run watch:js
```

## API

### Metadata handlers

Lists the Lighthouse categories:

```
GET /lh/categories
```

Lists the Lighthouse audits:

```
GET /lh/audits
```

### Lighthouse data handlers:

Lists the latest `MAX_REPORTS` reports for the URL.

```
GET /lh/reports
```

Lists the URLs that have been saved in the system.

```
GET /lh/urls
```

Displays the latest Lighthouse HTML report for `url`. If the `download` param
is included, the file is downloaded.

```
GET /lh/html?url="https://example.com"
GET /lh/html?url=https://example.com&download
```

Starts a new Lighthouse audit for `url`.

```
POST /lh/newaudit
Content-Type: application/json

{"url": "https://example.com"}
```

### Private handlers

Seeds the db with the list of URLs in `seeds_urls.json` by scheduling a task
to run a Lighthouse audit on each URL. **Note**: this handler can only run
if there are no URLs already stored. It should be run if the db is ever cleared.

```
GET /lh/seed_urls
```

Used by the cron job (see `cron.yaml`) to update Lighthouse scores for each
URL saved in the system. **Note**: this handler is only accessible to the App
Engine backend.

```
GET /cron/update_lighthouse_scores
```<commit_msg>readme: update screen, copy and handlers<commit_after># Lighthouse keeper

> Lighthouse keeper is a backend for providing historical [Lighthouse](https://developers.google.com/web/tools/lighthouse/) results for an URL. It's used to power the Lighthouse integration on web.dev.

There are several bits:

- Runs the Lighthouse API against a URL and stores reports over time (Firestore). Provides querying capabilities.
- The latest (full) report for the URL is stored (Google Cloud Storage).
- The server itself is Google App Engine (NodeJS). Cron jobs schedule tasks to update scores for each URL in the system.

There's page demo page (`/public/index.html`) that shows how this package can be used to render results.
Note: if you're testing, `CI_HOST` needs to be switched over to `http:localhost:8080` in the `app.js`.

<img width="1081" alt="web.dev frontend rendering data from this server" src="https://user-images.githubusercontent.com/238208/47517054-e2877b80-d83b-11e8-97d1-b2becc282604.png">

## Development

1. Install it:

```
npm i
```

2. Decrypt the service account and memcache json files:

```
npm run decrypt
```

You will be prompted to for a passphrase to decrypt `serviceAccount.json.enc`
and `memcacheCredentials.json.enc`. These files are required for the backend
to communicate with Firestore and the Redis memcache service. The password can
be found in Google's shared password tool.

3. Run the web server:

```
npm run start
````

4. Build it

To build the CSS/JS bundles, run of `npm run build:js`, `npm run build:css`,
or the single command:

```
npm run build
```

There are also watch tasks if you want to iterate on the files while the server
is running:

```
npm run watch:css
npm run watch:js
```

## API

### Metadata handlers

Lists the Lighthouse categories:

```
GET /lh/categories
```

Lists the Lighthouse audits:

```
GET /lh/audits
```

Lists the median scores (for each Lighthouse category) of all the URLs in the system,
or just for a particular URL.

```
GET /lh/medians[?url=https://example.com/]
```

### Lighthouse data handlers:

Lists the latest `MAX_REPORTS` reports for the URL. If the `?since=YYYY-MM-DD` query
parameter is ued, the results will be filtered to on or after that date.

```
GET /lh/reports?url=https://example.com/
GET /lh/reports?url=https://example.com/&since=2018-10-25
```

Lists the URLs that have been saved in the system.

```
GET /lh/urls
```

Displays the latest Lighthouse HTML report for `url`. If the `download` param
is included, the file is downloaded.

```
GET /lh/html?url=https://example.com
GET /lh/html?url=https://example.com&download
```

Starts a new Lighthouse audit for `url`.

```
POST /lh/newaudit
Content-Type: application/json

{"url": "https://example.com"}
```

### Private handlers

Seeds the db with the list of URLs in `seeds_urls.json` by scheduling a task
to run a Lighthouse audit on each URL. **Note**: this handler can only run
if there are no URLs already stored. It should be run if the db is ever cleared.

```
GET /lh/seed_urls
```

Used by the cron job (see `cron.yaml`) to update Lighthouse scores for each
URL saved in the system. **Note**: this handler is only accessible to the App
Engine backend.

```
GET /cron/update_lighthouse_scores
```

Used by the cron job (see `cron.yaml`) to delete existing results if the URL hasn't been 
viewed/audited or otherwise "touched" in the last 60 days. **Note**: this handler is only accessible to the App
Engine backend.

```
GET /cron/delete_stale_lighthouse_reports
```
{'fraction_non_alphanumeric': 0.10609137055837563, 'fraction_numerical': 0.019120135363790185, 'mean_word_length': 4.144473455178416, 'pattern_counts': {'":': 2, '<': 5, '<?xml version=': 0, '>': 6, 'https://': 12, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 8, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6140214', 'n_tokens_mistral': 2016, 'n_tokens_neox': 1808, 'n_words': 799}