<commit_before>.. _intro:

==============
 Introduction
==============

Data and its Names
==================

`Midas` is about associating two buckets of data and intending to
learn something from this new set of data. 

The first bucket of data are `Alexa Top1M files`: a set of ZIP-files
containing one file each. The ZIP-files naming scheme is like
``top-1m-2012-10-27.csv.zip`` with ``2012-10-27`` encoding a
time-stamp. As said above, each ZIP-file is supposed to contain
exactly one file named ``top-1m.csv``. This file is pure ASCII with
each line matching the scheme ``site,rank``, where ``site`` is a
domain (e.g. ``youtube.com``) optionally with an additional path
(e.g. ``youtube.com/user/123``) and ``rank`` is an :func:`int` stating
the ranking of this particular site on the date as encoded in the
file-name.

The second bucket of data is information about funding rounds provided
by `CrunchBase <http://www.crunchbase.com/>`_. This data is supposed
to be in a database as generated by the `crawlcrunch
<https://github.com/fuzzy-id/crawlcrunch>`_ library. For example, if
you want the data to reside in a SQLite database simply do a::

    cc_update --sql sqlite:///crunchbase_db.sql

`crawlcrunch` uses `SQLAlchemy <http://www.sqlalchemy.org/>`_ to
access databases and so does `midas`. Hence, if you want to use
another back-end than SQLite read up SQLAlchemy's description on
`engine configuration
<http://docs.sqlalchemy.org/en/latest/core/engines.html>`_. Note that
`crawlcrunch` is installed during the installation process of `midas`.

``cc_update`` fetches *all* the companies from CrunchBase and saves
them together with their associated funding rounds in the database.

Preliminary Steps
=================
  
Set up Hadoop
-------------

Most of the scripts provided by `midas` are made to run in `Hadoop
<http://hadoop.apache.org/>`_. As `midas` is Python Hadoops streaming
interface is used. Hence, most of the scripts are capable to run on
the command-line as well but will probably last too long.

Set up a Virtual Python Environment
-----------------------------------

`Midas` has some dependencies you probably do not want to install in
the systems Python path. A virtual environment is an easy and painless
solution here. Make sure to install the virtual environment in a
directory that is shared among all nodes in the Hadoop cluster.

Install `Midas`
---------------

If you have your virtual environment set up. Grab the source code from
`github <https://github.com/fuzzy-id/midas>`_ and run the installation
via::

    cd path/to/midas
    python setup.py install

This should take care of all further dependencies `crawlcrunch`
included.<commit_msg>unifying white spaces<commit_after>.. _intro:

==============
 Introduction
==============

Data and its Names
==================

`Midas` is about associating two buckets of data and intending to
learn something from this new set of data. 

The first bucket of data are `Alexa Top1M files`: a set of ZIP-files
containing one file each. The ZIP-files naming scheme is like
``top-1m-2012-10-27.csv.zip`` with ``2012-10-27`` encoding a
time-stamp. As said above, each ZIP-file is supposed to contain
exactly one file named ``top-1m.csv``. This file is pure ASCII with
each line matching the scheme ``site,rank``, where ``site`` is a
domain (e.g. ``youtube.com``) optionally with an additional path
(e.g. ``youtube.com/user/123``) and ``rank`` is an :func:`int` stating
the ranking of this particular site on the date as encoded in the
file-name.

The second bucket of data is information about funding rounds provided
by `CrunchBase <http://www.crunchbase.com/>`_. This data is supposed
to be in a database as generated by the `crawlcrunch
<https://github.com/fuzzy-id/crawlcrunch>`_ library. For example, if
you want the data to reside in a SQLite database simply do a::

   cc_update --sql sqlite:///crunchbase_db.sql

`crawlcrunch` uses `SQLAlchemy <http://www.sqlalchemy.org/>`_ to
access databases and so does `midas`. Hence, if you want to use
another back-end than SQLite read up SQLAlchemy's description on
`engine configuration
<http://docs.sqlalchemy.org/en/latest/core/engines.html>`_. Note that
`crawlcrunch` is installed during the installation process of `midas`.

``cc_update`` fetches *all* the companies from CrunchBase and saves
them together with their associated funding rounds in the database.

Preliminary Steps
=================
  
Set up Hadoop
-------------

Most of the scripts provided by `midas` are made to run in `Hadoop
<http://hadoop.apache.org/>`_. As `midas` is Python Hadoops streaming
interface is used. Hence, most of the scripts are capable to run on
the command-line as well but will probably last too long.

Set up a Virtual Python Environment
-----------------------------------

`Midas` has some dependencies you probably do not want to install in
the systems Python path. A virtual environment is an easy and painless
solution here. Make sure to install the virtual environment in a
directory that is shared among all nodes in the Hadoop cluster.

Install `Midas`
---------------

If you have your virtual environment set up. Grab the source code from
`github <https://github.com/fuzzy-id/midas>`_ and run the installation
via::

   cd path/to/midas
   python setup.py install

This should take care of all further dependencies `crawlcrunch`
included.
{'fraction_non_alphanumeric': 0.12344301914854063, 'fraction_numerical': 0.0081799591002045, 'mean_word_length': 3.864376130198915, 'pattern_counts': {'":': 0, '<': 15, '<?xml version=': 0, '>': 15, 'https://': 4, 'lorem ipsum': 0, 'www.': 4, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2840333', 'n_tokens_mistral': 1581, 'n_tokens_neox': 1472, 'n_words': 721}