<commit_before>Importing Files
===============

The LIME Pro API supports importing simple text files. 

File Recommendations
--------------------

The first line of the file should contain names for the columns in the rest of
the file.

The file should preferably be encoded using UTF-8.

Concepts
--------

In order to start an import, we need to define three major components, an
import file, an import configuration, and an import job.

**The import file** contains not only the contents of the file itself, but also
some additional information, such as what delimiter is used to separate the
columns.

**The import configuration** contains information about what file to import,
what type of LIME objects we want to import the file to, and how to map what's
in the file to fields on the fields and relation of that LIME type.

Finally, when **the import job** is created it will start the actual import as
soon as it can. The import job will then contain the status of the import,
and any errors that might have occured during the process.

Importing a file
----------------
Let's say we have the following file containing information about people we want to add to LIME:

::

    name;e-mail;ship;rank
    Ripley;<EMAIL>;Nostromo;Warrant Officer
    Dallas;<EMAIL>;Nostromo;Captain
    Ash;<EMAIL>;Nostromo;Science Officer
    ...

We have a LIME database that contains information about employees and ships, and we've been tasked with importing this file in to LIME.

We install the latest version of limeclient, create a module called weyland.py.

Authenticating
--------------
First of all, we need to authenticate ourselves with the LIME Pro server.
If we're using the API-client this done by creating an instance of LimeClient
and logging in:

.. code-block:: python

    from limeclient

     client = LimeClient('http://localhost:2134', 'weyland_db')
     with client.login('user', 'pass') as c:
         # Do stuff here...

This will start a new session in LIME that will automatically be closed when
program is finished.

Uploading a file
----------------
Once we've authenticated ourselves, it's time for us to tell LIME about the
file we want to import.


.. code-block:: python

    with client.login('user', 'pass') as c:
        with open('nostromo_crew.txt') as content:
            f = ImportFile(c).create(filename='nostromo_crew.txt',
                                     content=content)
            f.delimiter(';')
            f.save()

Here we uploaded a file to LIME, which returns with a file object populated
with default values. We then told LIME a little bit more about how it should
interpret the file, and saved the new information on the server.

Retrieving the LIME data type
-----------------------------
The second piece of information we need before we can start configuring how to
interpret the content of the file is an object that represents the data type
in LIME. In the LIME API this is called an 'entity'.

If the information in the file contains information about people we might want
to load the 'crew' entity type:

.. code-block:: python

    with client.login('user', 'pass') as c:
        with open('nostromo_crew.txt') as content:
            f = ImportFile(c).create(filename='nostromo_crew.txt',
                                     content=content)
            f.delimiter(';')
            f.save()

        crew = EntityTypes(c).get_by_name('crew')

Creating an import configuration
--------------------------------
With that we have enough information to start configuring our import:

.. code-block:: python

    with client.login('user', 'pass') as c:
        with open('nostromo_crew.txt') as content:
            f = ImportFile(c).create(filename='nostromo_crew.txt',
                                     content=content)
            f.delimiter(';')
            f.save()

        crew = EntityTypes(c).get_by_name('crew')

        config = ImportConfigs(c).create(entity=crew, importfile=f)

Behaviour
---------
We can tell LIME what it should do for each row it finds in our import file.

**config.behaviour = ImportConfig.CreateAndUpdate**
Update existing objects if they match  what's in the file, and create new
objects if nothing matches

**config.behaviour = ImportConfig.OnlyUpdate**
Only update objects that match what's in the import file. Don't create any new
objects.

**config.behaviour = ImportConfig.OnlyCreate**
Create a new object for each row in the import file. Don't try to match against
existing objects in LIME.

MAPPING
-------
For each row in the file to import, LIME needs to know what to do with the
data. We tell LIME how to accomplish this by telling it how to map each column
to something a field or relation of the data type we're import to.

The LIME API supports three types of mappings: mappings of simple types, such
as strings and numbers, mappings option fields where the value can be one of
several predefined values, and finally relations to other types of entities in
LIME, such as persons being related to companies.

Adding a simple field mapping
-----------------------------
In our example, the name and e-mail of the crew members are simple types, so we
add simple field mappings for those. We also mark the name field as a key
field, meaning that we will use this when determining if this row matches an
existing object in LIME.

.. code-block:: python

    with client.login('user', 'pass') as c:
        # ...

        crew = EntityTypes(c).get_by_name('crew')

        config = ImportConfigs(c).create(entity=crew, importfile=f)

        name = SimpleFieldMapping(field=crew.fields['name'],
                                   column='name',
                                   key=False)
        config.add_field_mapping(name)

        email = SimpleFieldMapping(field=crew.fields['email'],
                                   column='e-mail',
                                   key=True)
        config.add_field_mapping(email)


We can specify 'key=True' for multiple mappings. In that case all values must
match for the import to consider updating a person in LIME instead of adding a
new.

Adding a mapping to an option field
-----------------------------------

You can map a column in the import file to an option field in LIME by adding
an OptionFieldMapping to your import configuration.

Within the OptionFieldMapping, you specify how a value in a column translates
to one of the possible values of an option field in LIME.

.. todo::
    How do we want to specify mapping to option values? ID? Key?

.. code:: python

    with client.login('user', 'pass') as c:
        # ...

        field = crew.fields['rank']
        position = OptionFieldMapping(field=field, column='rank')
        position.default = field.option_id_for('Engineer')
        position.map_value(column_val='Captain',
                           field_val=field.option_id_for('Captain'))
        position.map_value(column_val='Warrant Officer',
                           field_val=field.option_id_for('Warrant Officer'))
        config.add_field_mapping(position)

In the example above we first say that any values for the 'rank' column that
haven't been explicitly mapped, we should assume that the crew member is
engineer.

We then proceed to explicitly map the values for captain and warrant officer.

[None should mean use default value of field]

Mapping relations
-----------------

Finally, we need to import the ship of each crew member in the file. 'Ship'
is a separate table in the LIME database and we need to tell the import about
this:

.. code:: python

    with client.login('user', 'pass') as c:
        # ...

        crew = EntityTypes(c).get_by_name('crew')

        # ...

        relation = crew['ship']
        ship = relation.related
        relation_mapping = RelationMapping(column='ship', relation=relation,
                                           key_field=ship.fields['name'])
        config.add_mapping(relation_mapping)

        config.save()

We ask the entity type for the relation to the ship type, we use that to get a
hold of the actual ship type. We then tell the importer that the 'ship' column
contains names of ships.

Now, we can save the import configuration and are ready to start the import.

Starting an import job
----------------------

We can now start the import job:

.. code-block:: python

   with client.login(user=args.user, password=args.password) as c:
       # ...

       job = ImportJobs(c).create(config)

       for i in range(10):
          time.sleep(1)
          job = job.refresh()
          print('Current job status: {}'.format(job.status))
          if job.has_errors:
              print('Oh noes! Errors!')
              print(job.errors.errors[:10])
          if job.status != 'pending' and job.status != 'running':
              break

This tells LIME to put the import job on a queue. We the proceed to poll the
status of the job. If something goes wrong, the ten first errors will be
printed to the console.

Working with non-normative import documents
===========================================

LIME is at the time of writing very strict about the structure of the file to
import. If we want to import documents from other sources that might not
adhere to these rules, we need to transform the original data in to something
that LIME accepts.

Let's say we have the following file to import:

::

    name;company;email;phone;title
    <NAME>;Jacobi Inc;<EMAIL>;1-978-451-7502x7744;IT
    <NAME>;Beer, Green and Grant;<EMAIL>;317.432.6066;Support
    ...

We have a single name column that contains both the first and last name.

Let's say that the object type we're importing to has a firstname and lastname
propery respectively.

A (naively simple) solution to this is to add a transformation step before we
upload our import file where we split the name column into a 'first name' and
a 'last name' column:

.. code-block:: python

    NORMALIZED_HEADERS = 'first name;last name;company;email;phone;title'

    def normalize(infile, outfile):
        # Write headers to our normalized file
        outfile.write(NORMALIZED_HEADERS + '\r\n')
        for line, content in enumerate(infile):
            # Ignore headers in original file
            if line > 0:
                row = content.split(';')  # Get original columns
                name = row.pop(0)  # Extract 'name' column
                first, last = name.split()  # Extract first and last names
                row.insert(0, last)  # Reinsert last name in row
                row.insert(0, first)  # Reinsert first name in row
                outfile.write(';'.join(row) + '\r\n')
        outfile.seek(0)

    with open('non-normative.txt', 'r', encoding='utf-8') as raw:
        with tempfile.TemporaryFile('w+', encoding='utf-8') as fixed:
            normalize(raw, fixed)
            f = ImportFiles(c).create(filename='non-normative.txt',
                                      content=fixed)
            f.delimiter = ';'
            f.save()

    # ...

Instead of just passing the original file directly to LIME, we create a
temporary file that we fill with a normalized version with two columns for
storing a name. We then pass this file to LIME instead.
<commit_msg>MAPPING -> Mapping<commit_after>Importing Files
===============

The LIME Pro API supports importing simple text files. 

File Recommendations
--------------------

The first line of the file should contain names for the columns in the rest of
the file.

The file should preferably be encoded using UTF-8.

Concepts
--------

In order to start an import, we need to define three major components, an
import file, an import configuration, and an import job.

**The import file** contains not only the contents of the file itself, but also
some additional information, such as what delimiter is used to separate the
columns.

**The import configuration** contains information about what file to import,
what type of LIME objects we want to import the file to, and how to map what's
in the file to fields on the fields and relation of that LIME type.

Finally, when **the import job** is created it will start the actual import as
soon as it can. The import job will then contain the status of the import,
and any errors that might have occured during the process.

Importing a file
----------------
Let's say we have the following file containing information about people we want to add to LIME:

::

    name;e-mail;ship;rank
    Ripley;<EMAIL>;Nostromo;Warrant Officer
    Dallas;<EMAIL>;Nostromo;Captain
    Ash;<EMAIL>;Nostromo;Science Officer
    ...

We have a LIME database that contains information about employees and ships, and we've been tasked with importing this file in to LIME.

We install the latest version of limeclient, create a module called weyland.py.

Authenticating
--------------
First of all, we need to authenticate ourselves with the LIME Pro server.
If we're using the API-client this done by creating an instance of LimeClient
and logging in:

.. code-block:: python

    from limeclient

     client = LimeClient('http://localhost:2134', 'weyland_db')
     with client.login('user', 'pass') as c:
         # Do stuff here...

This will start a new session in LIME that will automatically be closed when
program is finished.

Uploading a file
----------------
Once we've authenticated ourselves, it's time for us to tell LIME about the
file we want to import.


.. code-block:: python

    with client.login('user', 'pass') as c:
        with open('nostromo_crew.txt') as content:
            f = ImportFile(c).create(filename='nostromo_crew.txt',
                                     content=content)
            f.delimiter(';')
            f.save()

Here we uploaded a file to LIME, which returns with a file object populated
with default values. We then told LIME a little bit more about how it should
interpret the file, and saved the new information on the server.

Retrieving the LIME data type
-----------------------------
The second piece of information we need before we can start configuring how to
interpret the content of the file is an object that represents the data type
in LIME. In the LIME API this is called an 'entity'.

If the information in the file contains information about people we might want
to load the 'crew' entity type:

.. code-block:: python

    with client.login('user', 'pass') as c:
        with open('nostromo_crew.txt') as content:
            f = ImportFile(c).create(filename='nostromo_crew.txt',
                                     content=content)
            f.delimiter(';')
            f.save()

        crew = EntityTypes(c).get_by_name('crew')

Creating an import configuration
--------------------------------
With that we have enough information to start configuring our import:

.. code-block:: python

    with client.login('user', 'pass') as c:
        with open('nostromo_crew.txt') as content:
            f = ImportFile(c).create(filename='nostromo_crew.txt',
                                     content=content)
            f.delimiter(';')
            f.save()

        crew = EntityTypes(c).get_by_name('crew')

        config = ImportConfigs(c).create(entity=crew, importfile=f)

Behaviour
---------
We can tell LIME what it should do for each row it finds in our import file.

**config.behaviour = ImportConfig.CreateAndUpdate**
Update existing objects if they match  what's in the file, and create new
objects if nothing matches

**config.behaviour = ImportConfig.OnlyUpdate**
Only update objects that match what's in the import file. Don't create any new
objects.

**config.behaviour = ImportConfig.OnlyCreate**
Create a new object for each row in the import file. Don't try to match against
existing objects in LIME.

Mapping
-------
For each row in the file to import, LIME needs to know what to do with the
data. We tell LIME how to accomplish this by telling it how to map each column
to something a field or relation of the data type we're import to.

The LIME API supports three types of mappings: mappings of simple types, such
as strings and numbers, mappings option fields where the value can be one of
several predefined values, and finally relations to other types of entities in
LIME, such as persons being related to companies.

Adding a simple field mapping
-----------------------------
In our example, the name and e-mail of the crew members are simple types, so we
add simple field mappings for those. We also mark the name field as a key
field, meaning that we will use this when determining if this row matches an
existing object in LIME.

.. code-block:: python

    with client.login('user', 'pass') as c:
        # ...

        crew = EntityTypes(c).get_by_name('crew')

        config = ImportConfigs(c).create(entity=crew, importfile=f)

        name = SimpleFieldMapping(field=crew.fields['name'],
                                   column='name',
                                   key=False)
        config.add_field_mapping(name)

        email = SimpleFieldMapping(field=crew.fields['email'],
                                   column='e-mail',
                                   key=True)
        config.add_field_mapping(email)


We can specify 'key=True' for multiple mappings. In that case all values must
match for the import to consider updating a person in LIME instead of adding a
new.

Adding a mapping to an option field
-----------------------------------

You can map a column in the import file to an option field in LIME by adding
an OptionFieldMapping to your import configuration.

Within the OptionFieldMapping, you specify how a value in a column translates
to one of the possible values of an option field in LIME.

.. todo::
    How do we want to specify mapping to option values? ID? Key?

.. code:: python

    with client.login('user', 'pass') as c:
        # ...

        field = crew.fields['rank']
        position = OptionFieldMapping(field=field, column='rank')
        position.default = field.option_id_for('Engineer')
        position.map_value(column_val='Captain',
                           field_val=field.option_id_for('Captain'))
        position.map_value(column_val='Warrant Officer',
                           field_val=field.option_id_for('Warrant Officer'))
        config.add_field_mapping(position)

In the example above we first say that any values for the 'rank' column that
haven't been explicitly mapped, we should assume that the crew member is
engineer.

We then proceed to explicitly map the values for captain and warrant officer.

[None should mean use default value of field]

Mapping relations
-----------------

Finally, we need to import the ship of each crew member in the file. 'Ship'
is a separate table in the LIME database and we need to tell the import about
this:

.. code:: python

    with client.login('user', 'pass') as c:
        # ...

        crew = EntityTypes(c).get_by_name('crew')

        # ...

        relation = crew['ship']
        ship = relation.related
        relation_mapping = RelationMapping(column='ship', relation=relation,
                                           key_field=ship.fields['name'])
        config.add_mapping(relation_mapping)

        config.save()

We ask the entity type for the relation to the ship type, we use that to get a
hold of the actual ship type. We then tell the importer that the 'ship' column
contains names of ships.

Now, we can save the import configuration and are ready to start the import.

Starting an import job
----------------------

We can now start the import job:

.. code-block:: python

   with client.login(user=args.user, password=<PASSWORD>) as c:
       # ...

       job = ImportJobs(c).create(config)

       for i in range(10):
          time.sleep(1)
          job = job.refresh()
          print('Current job status: {}'.format(job.status))
          if job.has_errors:
              print('Oh noes! Errors!')
              print(job.errors.errors[:10])
          if job.status != 'pending' and job.status != 'running':
              break

This tells LIME to put the import job on a queue. We the proceed to poll the
status of the job. If something goes wrong, the ten first errors will be
printed to the console.

Working with non-normative import documents
===========================================

LIME is at the time of writing very strict about the structure of the file to
import. If we want to import documents from other sources that might not
adhere to these rules, we need to transform the original data in to something
that LIME accepts.

Let's say we have the following file to import:

::

    name;company;email;phone;title
    <NAME>;Jacobi Inc;<EMAIL>;1-978-451-7502x7744;IT
    <NAME>;<NAME> and Grant;<EMAIL>;317.432.6066;Support
    ...

We have a single name column that contains both the first and last name.

Let's say that the object type we're importing to has a firstname and lastname
propery respectively.

A (naively simple) solution to this is to add a transformation step before we
upload our import file where we split the name column into a 'first name' and
a 'last name' column:

.. code-block:: python

    NORMALIZED_HEADERS = 'first name;last name;company;email;phone;title'

    def normalize(infile, outfile):
        # Write headers to our normalized file
        outfile.write(NORMALIZED_HEADERS + '\r\n')
        for line, content in enumerate(infile):
            # Ignore headers in original file
            if line > 0:
                row = content.split(';')  # Get original columns
                name = row.pop(0)  # Extract 'name' column
                first, last = name.split()  # Extract first and last names
                row.insert(0, last)  # Reinsert last name in row
                row.insert(0, first)  # Reinsert first name in row
                outfile.write(';'.join(row) + '\r\n')
        outfile.seek(0)

    with open('non-normative.txt', 'r', encoding='utf-8') as raw:
        with tempfile.TemporaryFile('w+', encoding='utf-8') as fixed:
            normalize(raw, fixed)
            f = ImportFiles(c).create(filename='non-normative.txt',
                                      content=fixed)
            f.delimiter = ';'
            f.save()

    # ...

Instead of just passing the original file directly to LIME, we create a
temporary file that we fill with a normalized version with two columns for
storing a name. We then pass this file to LIME instead.

{'fraction_non_alphanumeric': 0.10012046580109758, 'fraction_numerical': 0.003747824923035738, 'mean_word_length': 2.600064246707356, 'pattern_counts': {'":': 0, '<': 19, '<?xml version=': 0, '>': 22, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 20, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '325208', 'n_tokens_mistral': 6035, 'n_tokens_neox': 5760, 'n_words': 2664}