<jupyter_start><jupyter_text>Import Agribalyse and link to ecoinvent 3.2 cutoff[Agribalyse](http://www.ademe.fr/en/expertise/alternative-approaches-to-production/agribalyse-program) is a French LCI database of agricultural products. It builds on top of ecoinvent 3.2. It was exported from SimaPro, so the names of ecoinvent processes are mangled, and need to be fixed back to standard ecoinvent.This notebook uses Agribalyse 1.3.<jupyter_code>from brightway2 import *<jupyter_output><empty_output><jupyter_text>Create a new project for this notebook<jupyter_code>projects.set_current("Agribalyse 1.3")<jupyter_output><empty_output><jupyter_text>Add biosphere flowsBiosphere flow names follow the standard in ecoinvent 3.3. We will need to match these names to those in Agribalyse.<jupyter_code>bw2setup()<jupyter_output>Biosphere database already present!!! No setup is needed<jupyter_text>Import ecoinvent 3.2 cutoff as background database<jupyter_code>path = "/Users/cmutel/Documents/LCA Documents/Ecoinvent/3.2/cutoff/datasets"
importer = SingleOutputEcospold2Importer(path, "ecoinvent 3.2 cutoff")
importer.apply_strategies()
importer.statistics()
importer.write_database()<jupyter_output><empty_output><jupyter_text>Load Agribalyse dataThis notebook uses the SimaPro CSV version of Agribalyse.<jupyter_code>path = "/Users/cmutel/Documents/LCA Documents/Agribalyse/1.3/Agribalyse CSV FINAL_no links_Nov2016v3.CSV"
ag = SimaProCSVImporter(path, "Agribalyse 1.3")
ag.apply_strategies()
ag.statistics()<jupyter_output>Extracted 1189 unallocated datasets in 5.21 seconds
Applying strategy: normalize_units
Applying strategy: assign_only_product_as_production
Applying strategy: drop_unspecified_subcategories
Applying strategy: sp_allocate_products
Applying strategy: fix_zero_allocation_products
Applying strategy: split_simapro_name_geo
Applying strategy: strip_biosphere_exc_locations
Applying strategy: migrate_datasets
Applying strategy: migrate_exchanges
Applying strategy: set_code_by_activity_hash
Applying strategy: link_technosphere_based_on_name_unit_location
Applying strategy: change_electricity_unit_mj_to_kwh
Applying strategy: normalize_biosphere_categories
Applying strategy: normalize_simapro_biosphere_categories
Applying strategy: normalize_biosphere_names
Applying strategy: normalize_simapro_biosphere_names
Applying strategy: migrate_exchanges
Applying strategy: fix_localized_water_flows
Applying strategy: link_iterable_by_fields
Applied 19 strategies in 4.46 seconds
1261 datasets
115259 excha[...]<jupyter_text>Link to `ecoinvent 3.2`We start by changing the SimaPro names to what ecoinvent provides, as we are using the ecoinvent-provided version of 3.2. We can then match the unlinked inputs in `Agribalyse` against the outputs provided by `ecoinvent`.<jupyter_code>ag.migrate('simapro-ecoinvent-3.2')
ag.match_database("ecoinvent 3.2 cutoff", fields=('reference product', 'location', 'unit', 'name'))
ag.statistics()<jupyter_output>1261 datasets
115259 exchanges
2904 unlinked exchanges
  Type biosphere: 460 unique unlinked exchanges
  Type technosphere: 33 unique unlinked exchanges<jupyter_text>Let's look at what we can't link. The easiest way is to export the unlinked exchanges to an Excel file.<jupyter_code>ag.write_excel(True)<jupyter_output>Wrote matching file to:
/Users/cmutel/Library/Application Support/Brightway3/Agribalyse-13.a68c51d5c40982bcd21e1aa4ec805302/output/db-matching-Agribalyse-13-unlinked.xlsx<jupyter_text>You can also [download this Excel file](https://docs.brightwaylca.org/_images/db-matching-Agribalyse-13-unlinked.xlsx).  Add unlinked biosphere flowsBrightway2-IO 0.5.7 includes a lot of fixes to import Agribalyse more cleanly: [2a5406a](https://bitbucket.org/cmutel/brightway2-io/commits/2a5406ab9e12f72745a2efc90b7b4c696ade8717), [0154841](https://bitbucket.org/cmutel/brightway2-io/commits/01548417d69438c010e60431a0ca0cf44408a034), [b2d5545](https://bitbucket.org/cmutel/brightway2-io/commits/b2d55453b98f377cb8b1356c5326e9480fac24fa), [99b07b9](https://bitbucket.org/cmutel/brightway2-io/commits/99b07b9daaf8899c636c9aa51466bf833c51f106), [2129804](https://bitbucket.org/cmutel/brightway2-io/commits/2129804044f0503cf67a70d0841e0e1582530b99), [5d6b936](https://bitbucket.org/cmutel/brightway2-io/commits/5d6b9361f32c58cfaed19a6f3352d129eca1f806), and [5d0542a](https://bitbucket.org/cmutel/brightway2-io/commits/5d0542aee7f7ebe3aa418cbd0fac9ac696c0eb26).However, there are still a large number of biosphere flows not used in ecoinvent that we don't have in our `biosphere3` database (and, consequently, in our LCIA methods). The best we can do is to import these flows into another biosphere database. If these flows are important to you, you should do further matching or modify the relevant LCIA methods to include them.<jupyter_code>Database("Agribalyse 1.3 new biosphere").register()
ag.add_unlinked_flows_to_biosphere_database("Agribalyse 1.3 new biosphere")<jupyter_output>Writing activities to SQLite3 database:
0%                          100%
[##############################] | ETA[sec]: 0.751  | ETA[sec]: 0.411  | ETA[sec]: 0.288  | ETA[sec]: 0.224  | ETA[sec]: 0.188  | ETA[sec]: 0.166  | ETA[sec]: 0.145  | ETA[sec]: 0.130  | ETA[sec]: 0.176  | ETA[sec]: 0.155  | ETA[sec]: 0.140  | ETA[sec]: 0.126  | ETA[sec]: 0.114  | ETA[sec]: 0.103  | ETA[sec]: 0.093  | ETA[sec]: 0.083  | ETA[sec]: 0.086  | ETA[sec]: 0.078  | ETA[sec]: 0.069  | ETA[sec]: 0.061  | ETA[sec]: 0.054  | ETA[sec]: 0.047  | ETA[sec]: 0.040  | ETA[sec]: 0.033  | ETA[sec]: 0.029  | ETA[sec]: 0.023  | ETA[sec]: 0.017  | ETA[sec]: 0.011  | ETA[sec]: 0.005  | ETA[sec]: 0.000  | ETA[sec]: 0.000 
Total time elapsed: 0.165 sec<jupyter_text>We should now have no unlinked biosphere flows:  Fixing units for passenger carsFor some reason there are some links to `market for transport, passenger car, medium size, diesel, EURO 4` and `market for transport, passenger car, medium size, petrol, EURO 4` with units of meters (instead of person kilometers, as in ecoinvent). We can create a custom migration to handle these cases:<jupyter_code>agribalyse_transport_data = {
    'fields': ['name', 'unit'],
    'data': [
        (
            ('market for transport, passenger car, medium size, diesel, EURO 4', 'meter'), 
            {
                'unit': 'kilometer', 
                'multiplier': 1e-3
            }
        ),
        (
            ('market for transport, passenger car, medium size, petrol, EURO 4', 'meter'), 
            {
                'unit': 'kilometer', 
                'multiplier': 1e-3
            }
        )
    ]
}

Migration("agribalyse-transport").write(
    agribalyse_transport_data,
    description="Fix some transport unit conversions"
)
ag.migrate("agribalyse-transport")<jupyter_output>Applying strategy: migrate_datasets
Applying strategy: migrate_exchanges<jupyter_text>Link to ecoinvent again, to fix our now linkable transport inputs.<jupyter_code>ag.match_database("ecoinvent 3.2 cutoff", fields=('reference product', 'location', 'unit', 'name'))
ag.statistics()<jupyter_output>1261 datasets
115259 exchanges
113 unlinked exchanges
  Type technosphere: 31 unique unlinked exchanges<jupyter_text>Add waste treatment activitiesThere are a number of unlinked waste flows. I am not sure on the best way to handle these, so we will try a very simple approach. We will just add these as new activities, without any other inputs or environmental effects.<jupyter_code>ag.add_unlinked_activities()
ag.statistics()<jupyter_output>Applying strategy: link_iterable_by_fields
1292 datasets
115259 exchanges
0 unlinked exchanges<jupyter_text>Write the modified, fully linked databaseWe are finished with the importing process.<jupyter_code>ag.write_database()<jupyter_output>Writing activities to SQLite3 database:
0%                          100%
[##############################] | ETA[sec]: 2.333  | ETA[sec]: 4.123  | ETA[sec]: 7.568  | ETA[sec]: 6.914  | ETA[sec]: 7.082  | ETA[sec]: 7.498  | ETA[sec]: 8.129  | ETA[sec]: 7.346  | ETA[sec]: 6.489  | ETA[sec]: 6.087  | ETA[sec]: 5.413  | ETA[sec]: 5.082  | ETA[sec]: 5.265  | ETA[sec]: 4.997  | ETA[sec]: 4.681  | ETA[sec]: 4.413  | ETA[sec]: 4.098  | ETA[sec]: 4.154  | ETA[sec]: 3.783  | ETA[sec]: 3.362  | ETA[sec]: 3.115  | ETA[sec]: 2.671  | ETA[sec]: 2.331  | ETA[sec]: 1.975  | ETA[sec]: 1.623  | ETA[sec]: 1.311  | ETA[sec]: 0.996  | ETA[sec]: 0.656  | ETA[sec]: 0.328  | ETA[sec]: 0.000  | ETA[sec]: 0.000 
Total time elapsed: 9.796 sec<jupyter_text>Checking the imported datasetsWe need to do some basic validation to make sure we have meaningful results. Here I just do some basic testing, but you should validate against known scores if you are frequently using this database. The following code is rahter simple and is not a real validation check.<jupyter_code>gwp = [x for x in methods if "IPCC 2013" in str(x)][0]
gwp
db = Database("Agribalyse 1.3")

lca = LCA({db.random(): 1}, gwp)
lca.lci(factorize=True)
lca.lcia()
lca.score<jupyter_output><empty_output><jupyter_text>Let's calculate the LCIA scores of all activities in Agribalyse<jupyter_code>import pyprind

scores = []

for act in pyprind.prog_bar(db):
    lca.redo_lcia({act: 1})
    scores.append(lca.score)
import numpy as np

scores = np.array(scores)

mask = scores == 0
print("Number of activities with zero GWP score: {} of {}".format(mask.sum(), len(db)))
%matplotlib notebook
import seaborn as sns
sns.distplot(np.log(scores[scores > 0]))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11176718684099536, 'fraction_numerical': 0.0690636862083509, 'mean_word_length': 5.199346405228758, 'pattern_counts': {'":': 0, '<': 48, '<?xml version=': 0, '>': 49, 'https://': 8, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '365088', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 3772, 'n_tokens_neox': 3272, 'n_words': 848, 'path': 'notebooks/IO - Importing Agribalyse 1.3 with Ecoinvent 3.2 cutoff.ipynb', 'repo_name': 'elinlucy/brightway2', 'size': '9484'}