<commit_before># gumtree-scraper

This tool was written during the last time (March 2016) I looked for an
apartment to rent. The most popular website for that in Poland (and probably
in other parts of Europe) was [Gumtree](http://gumtree.pl). Kind of annoyed
with their interface (though it didn't matter to my girlfriend, who was
searching with me), and also disappointed by fact that Gumtree removed built-in
RSS feeds, I decided to write simple scraper.

![animation](docs/sample.gif)

## Features

- prettier list of offers (when compared with Gumtree's)
- ability to hide or star particular offer
	- starred offer can't be hidden until it's unstarred
- global seen mark to indicate which offers were already clicked
- offers are only *added*; when something gets removed from Gumtree, it's still
  here
- it's *your* database, and you can do whatever you want with it
- autorefresh for both backend and frontend
- fancy "Hide all" button to get rid of all visible entries when you're done
  (except for starred - those stay no matter what)

## Tech stuff

### Requirements

- Python 3.5 (for running everything)
- npm (for compiling frontend assets)

### Notable libraries used

- Flask
- React
- Redux
- [TinyDB](http://tinydb.readthedocs.org/) (yup, it's not threadsafe, so you
  may consider switching to other DB provider before going live - but for us
  it was sufficient)

### Installation

Pretty straightforward. virtualenv or pyenv (or both) are recommended.

```
pip install -r requirements
npm install
./node_modules/webpack/bin/webpack.js -d
```

### Running

No daemonization yet - meaning you need 2 shells to run these:

```
python scrap.py
python server.py
```

### Architecture

#### Scraper

Resides in `scrap.py`. Crawls over 1st page of Gumtree results and stores them
in DB. Rinses and repeats every 5 minutes. Note: you need to provide URL for
Gumtree results page after running it (as each category has different URL;
that means you may also use filters, because they get appended to URL).

#### Server (Flask)

You can find it in `server.py`. Static page used to bootstrap React and
a couple of endpoints for manipulating entries.

#### Client (React)

A couple of components residing in `js/index.jsx`. I didn't find enough reasons
to go overboard with JS modules, thus everything is in single file.

## OK, looks great... but why?

Clicking through all offers with their pagination was tiring. Moreover, I think
their interface is not the most readable one I've ever seen; besides, I like to
see *all* opportunities, and most of the offers are added to their site either
1) when I'm at work, or 2) late in the evening. So I could have had a late
start when applying for an apartment - and believe me, the best offers are
grabbed in matter of hours, if not *minutes*.

Moreover, there were 2 people looking for a flat, and Gumtree doesn't offer any
sharing features - we used Trello to manage our results, but we had to make<commit_msg>A couple of fixes in README<commit_after># gumtree-scraper

This tool was written during the last time (March 2016) I looked for an
apartment to rent. The most popular website for that in Poland (and probably
in other parts of Europe) was [Gumtree](http://gumtree.pl). Kind of annoyed
with their interface (though it didn't matter to my girlfriend, who was
searching with me), and also disappointed by fact that Gumtree removed built-in
RSS feeds, I decided to write simple scraper.

![animation](docs/sample.gif)

## Features

- prettier list of offers (when compared with Gumtree's)
- ability to hide or star particular offer
	- starred offer can't be hidden until it's unstarred
- global seen mark to indicate which offers were already clicked
- offers are only *added*; when something gets removed from Gumtree, it's still
  here
- it's *your* database, and you can do whatever you want with it
- autorefresh for both backend and frontend
- fancy "Hide all" button to get rid of all visible entries when you're done
  (except for starred - those stay no matter what)

## Tech stuff

### Requirements

- Python 3.5 (for running everything)
- npm (for compiling frontend assets)

### Notable libraries used

- Flask
- React
- Redux
- [TinyDB](http://tinydb.readthedocs.org/) (yup, it's not threadsafe, so you
  may consider switching to other DB provider before going live - but for us
  it was sufficient)

### Installation

Pretty straightforward. virtualenv or pyenv (or both) are recommended.

```
pip install -r requirements.txt
npm install
./node_modules/webpack/bin/webpack.js -d
```

### Running

No daemonization yet - meaning you need 2 shells to run these:

```
python scrap.py
python server.py
```

### Architecture

#### Scraper

Resides in `scrap.py`. Crawls over 1st page of Gumtree results and stores them
in DB. Rinses and repeats every 5 minutes. Note: you need to provide URL for
Gumtree results page after running it (as each category has different URL;
that means you may also use filters, because they get appended to URL).

#### Server (Flask)

You can find it in `server.py`. Static page used to bootstrap React and
a couple of endpoints for manipulating entries.

#### Client (React)

Main entry point is in `js/index.jsx` - that's also where application's store
is defined. I followed split between presentation and container components,
which are (respectively) in `js/components.jsx` and `js/containers.jsx`.

## OK, looks great... but why?

Clicking through all offers with their pagination was tiring. Moreover, I think
their interface is not the most readable one I've ever seen; besides, I like to
see *all* opportunities, and most of the offers are added to their site either
1) when I'm at work, or 2) late in the evening. So I could have had a late
start when applying for an apartment - and believe me, the best offers are
grabbed in matter of hours, if not *minutes*.

Moreover, there were 2 people looking for a flat, and Gumtree doesn't offer any
sharing features - we used Trello to manage our results, but we had to make
{'fraction_non_alphanumeric': 0.07115607895173329, 'fraction_numerical': 0.003980759661635429, 'mean_word_length': 4.097210481825867, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '545153', 'n_tokens_mistral': 1736, 'n_tokens_neox': 1625, 'n_words': 910}