<commit_before>Scraper pour le rôle d'évaluation foncière de la Ville de Montréal: http://evalweb.ville.montreal.qc.ca/

For 2012, this represents $214,989,968,755.00 (215 *billion* dollars).

# Installation

    git clone ...
    
On Ubuntu, you may have to install some libraries first:

    sudo apt-get install rubygems libxslt1-dev
    gem install mechanize
    gem install minitar

# Usage

Start scraping:

    ./scrape-evalweb.rb

The scraper produces dir and pag files, where it stores the web pages it fetched.

Once it is done, extract the data. This will create `evaluations.csv` as well as the SQL commands to load it into MySQL:

    ./export-evalweb.rb
    ./export-streets.rb

Let it finish (it can take hours for the tar archive generation, and an hour for the CSV export), then import the resulting files into MySQL:

    mysql -u root < evaluations.sql
    mysql -u root < streets.sql

You can now query the database:

    mysql -u root registre_foncier_montreal

# Downloads

You may spare yourself the scraping by downloading a cache from november 2012 with BitTorrent:

    magnet:?xt=urn:btih:cf07bef9dc9264abd8a0ab9c4412bea634065b1a&dn=evalweb-cache.tgz&tr=udp%3A%2F%2Ftracker.istole.it%3A80

Or download the CSV file directly:

    magnet:?xt=urn:btih:5fdd555ff92acb6fb300a50c71d90516770aeda3&dn=evaluations.csv.gz&tr=udp%3A%2F%2Ftracker.istole.it%3A80

# Example Queries

## Total Value

    SELECT count(*),
        format(sum(valeur_terrain),0) as terrains,
        format(sum(valeur_batiment),0) as batiments,
        format(sum(valeur_immeuble),0) as immeubles
      FROM evaluations;

## Value by Borough

    SELECT ville, arrondissement, count(*),
        format(sum(valeur_terrain),0) as terrains,
        format(sum(valeur_batiment),0) as batiments,
        format(sum(valeur_immeuble),0) as immeubles<commit_msg>MySQL now wants --local-infile<commit_after>Scraper pour le rôle d'évaluation foncière de la Ville de Montréal: http://evalweb.ville.montreal.qc.ca/

For 2012, this represents $214,989,968,755.00 (215 *billion* dollars).

# Installation

    git clone ...
    
On Ubuntu, you may have to install some libraries first:

    sudo apt-get install rubygems libxslt1-dev
    gem install mechanize
    gem install minitar

# Usage

Start scraping:

    ./scrape-evalweb.rb

The scraper produces dir and pag files, where it stores the web pages it fetched.

Once it is done, extract the data. This will create `evaluations.csv` as well as the SQL commands to load it into MySQL:

    ./export-evalweb.rb
    ./export-streets.rb

Let it finish (it can take hours for the tar archive generation, and an hour for the CSV export), then import the resulting files into MySQL:

    mysql --local-infile -u root < evaluations.sql
    mysql --local-infile -u root < streets.sql

You can now query the database:

    mysql -u root registre_foncier_montreal

# Downloads

You may spare yourself the scraping by downloading a cache from november 2012 with BitTorrent:

    magnet:?xt=urn:btih:cf07bef9dc9264abd8a0ab9c4412bea634065b1a&dn=evalweb-cache.tgz&tr=udp%3A%2F%2Ftracker.istole.it%3A80

Or download the CSV file directly:

    magnet:?xt=urn:btih:5fdd555ff92acb6fb300a50c71d90516770aeda3&dn=evaluations.csv.gz&tr=udp%3A%2F%2Ftracker.istole.it%3A80

# Example Queries

## Total Value

    SELECT count(*),
        format(sum(valeur_terrain),0) as terrains,
        format(sum(valeur_batiment),0) as batiments,
        format(sum(valeur_immeuble),0) as immeubles
      FROM evaluations;

## Value by Borough

    SELECT ville, arrondissement, count(*),
        format(sum(valeur_terrain),0) as terrains,
        format(sum(valeur_batiment),0) as batiments,
        format(sum(valeur_immeuble),0) as immeubles
{'fraction_non_alphanumeric': 0.0846914239914507, 'fraction_numerical': 0.04702110606465402, 'mean_word_length': 3.8686605981794537, 'pattern_counts': {'":': 0, '<': 7, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1127390', 'n_tokens_mistral': 1415, 'n_tokens_neox': 1268, 'n_words': 380}