<jupyter_start><jupyter_text>Importing the FORWAST database[FORWAST](http://forwast.brgm.fr/Overview.asp) is a physical MRIO table for Europe.<jupyter_code>from brightway2 import *
projects.set_current("forwast")
bw2setup()<jupyter_output>Biosphere database already present!!! No setup is needed<jupyter_text>We start with an export of the database from SimaPro, provided by the FORWAST team.<jupyter_code>ei = SingleOutputEcospold1Importer("/Users/cmutel/Downloads/FORWAST-ecospold1", "forwast")
ei.apply_strategies()
ei.statistics()<jupyter_output>Extracting ecospold1 files:
0%                          100%
[##############################] | ETA[sec]: 0.000 | Item ID: /Users/cmutel/D
Total time elapsed: 3.399 sec<jupyter_text>Major linking problemsThe first problem is that each process has a category, but none of the exchanges do. The IO library can't link them using its default algorithm, which includes categories when matching. Luckily, we can easily define a new strategy that doesn't include categories.<jupyter_code>from bw2io.strategies import link_iterable_by_fields
import functools

without_subcategories = functools.partial(link_iterable_by_fields, fields=['name', 'unit'], internal=True)
ei.apply_strategy(without_subcategories)

ei.statistics()<jupyter_output>Applying strategy: link_iterable_by_fields
277 datasets
82546 exchanges
4432 unlinked exchanges
  Type biosphere: 16 unique unlinked exchanges<jupyter_text>Biosphere flow conflictsThe next problem is that there are several biosphere flows that have non-matching names or categories, when compared to our base biosphere flow list. There isn't an automatic way to fix this; rather, I used code like:    db = Database("biosphere3")    db.search("natural gas")    to find out what flows were in our reference list, and what fields needed to change so that matching could occur. We use the migrations functionality to fix only the fields we need to change.<jupyter_code>migration_data = {
    'fields': ['name', 'categories', 'type', 'unit'],
    'data': [
        (
            ('Aluminium', ('natural resource',), 'biosphere', 'kilogram'),
            {
                'categories': ('natural resource', 'in ground'),
                'name': 'Aluminium, in ground',
            }
        ),
        (
            ('Carbon dioxide, in air', ('natural resource',), 'biosphere', 'kilogram'),
            {
                'categories': ('natural resource', 'in air'),
            }
        ),
        (
            ('Carbon monoxide', ('air',), 'biosphere', 'kilogram'),
            {
                'name': 'Carbon monoxide, fossil',
            }
        ),
        (
            ('Coal, hard', ('natural resource',), 'biosphere', 'kilogram'),
            {
                'categories': ('natural resource', 'in ground'),
                'name': 'Coal, hard, unspecified, in ground',
            }
        ),
        (
            ('Gas, natural, 46.8 MJ per kg', ('natural resource',), 'biosphere', 'kilogram'),
            {
                'categories': ('natural resource', 'in ground'),
                'name': 'Gas, natural, in ground',
                'unit': 'cubic meter',
                'multiplier': 1 / 0.8  # Approximation, density varies from 0.7/0.9 km/m3
            }
        ),
        (
            ('Sand and clay', ('natural resource',), 'biosphere', 'kilogram'),
            {
                'categories': ('natural resource', 'in ground'),
                'name': 'Sand, unspecified, in ground',
            }
        ),
        (
            ('Methane', ('air',), 'biosphere', 'kilogram'),
            {
                'name': 'Methane, fossil',
            }
        ),
        (
            ('Nitrogen dioxide', ('air',), 'biosphere', 'kilogram'),
            {
                'name': 'Nitrogen oxides',
            }
        ),
        (
            ('Oil, crude', ('natural resource',), 'biosphere', 'kilogram'),
            {
                'name': 'Oil, crude, in ground',
                'categories': ('natural resource', 'in ground'),
            }
        ),
        (
            ('Zinc', ('natural resource',), 'biosphere', 'kilogram'),
            {
                'name': 'Zinc, Zn 0.63%, in mixed ore, in ground',
                'categories': ('natural resource', 'in ground'),
            }
        ),
        (
            ('Lead', ('natural resource',), 'biosphere', 'kilogram'),
            {
                'name': 'Lead, Pb 0.014%, in mixed ore, in ground',
                'categories': ('natural resource', 'in ground'),
            }
        ),
        (
            ('Iron', ('natural resource',), 'biosphere', 'kilogram'),
            {
                'name': "Iron, 46% in ore, 25% in crude ore, in ground",
                'categories': ('natural resource', 'in ground'),
            }
        ),
        (
            ('Copper', ('natural resource',), 'biosphere', 'kilogram'),
            {
                'name': "Copper, 1.13% in sulfide, Cu 0.76% and Ni 0.76% in crude ore, in ground",
                'categories': ('natural resource', 'in ground'),
            }
        ),
        (
            ('Nickel', ('natural resource',), 'biosphere', 'kilogram'),
            {
                'name': 'Nickel, 1.98% in silicates, 1.04% in crude ore, in ground',
                'categories': ('natural resource', 'in ground'),
            }
        )
    ]
}

migration = Migration("simapro-forwast")
migration.write(
    migration_data, 
    description="Migration for biosphere flows from FORWAST database, as imported into SimaPro"
)<jupyter_output><empty_output><jupyter_text>We then apply our migration.<jupyter_code>ei.migrate("simapro-forwast")
ei.apply_strategy(ei.strategies[-2])

ei.statistics()<jupyter_output>Applying strategy: migrate_datasets
Applied 1 strategies in 0.00 seconds
Applying strategy: migrate_exchanges
Applied 1 strategies in 0.72 seconds
Applying strategy: link_iterable_by_fields
277 datasets
82546 exchanges
554 unlinked exchanges
  Type biosphere: 2 unique unlinked exchanges<jupyter_text>This is very good. Let's look at the remaining two flows:<jupyter_code>for x in ei.unlinked:
    print(x)<jupyter_output>{'unit': 'kilogram', 'categories': ['natural resource'], 'loc': 0.0, 'type': 'biosphere', 'name': 'Other minerals, extracted for use', 'amount': 0.0, 'uncertainty type': 0}
{'unit': 'kilogram', 'categories': ['natural resource'], 'loc': 0.0, 'type': 'biosphere', 'name': 'Other minerals, related unused extraction', 'amount': 0.0, 'uncertainty type': 0}<jupyter_text>We don't actually care about these `other minerals`, so just drop them.<jupyter_code>ei.drop_unlinked()<jupyter_output>Warning: This is the nuclear weapon of linking, and should only be used in extreme cases. Must be called with the keyword argument ``i_am_reckless=True``!<jupyter_text>Our IO library doesn't like dropping unlinked exchanges, but we are comfortable with this, so we proceed anyway.<jupyter_code>ei.drop_unlinked(i_am_reckless=True)<jupyter_output>Applying strategy: drop_unlinked
Applied 1 strategies in 0.02 seconds<jupyter_text>One last manipulation before writing to disk: Let's delete all exchanges with zero amounts, as this is just noise.<jupyter_code>for ds in ei.data:
    ds['exchanges'] = [exc for exc in ds['exchanges'] if (exc['amount'] or exc['uncertainty type'] != 0)]
    
ei.statistics()<jupyter_output>277 datasets
39194 exchanges
0 unlinked exchanges<jupyter_text>Removing exchanges with zero amounts and no uncertainty reduced the database size by over half. As zero is the default value for any value in our matrix, unless specified otherwise, we haven't lost any information. To prevent others from having to do the same manipulation, we write the database, and then export a Brightway2 package for easy installation in the future.<jupyter_code>ei.write_database()
BW2Package.export_obj(Database("forwast"))<jupyter_output><empty_output><jupyter_text>Finally, let's visualize the database technosphere matrix:<jupyter_code>%matplotlib inline
from bw2analyzer.matrix_grapher import SparseMatrixGrapher
lca = LCA({Database('forwast').random(): 1})
lca.lci()
# Skip a bunch of very large negative numbers... ?
lca.technosphere_matrix[lca.technosphere_matrix < 0] = 0
SparseMatrixGrapher(lca.technosphere_matrix).magnitude_graph(width=5, height=5)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12671846981470414, 'fraction_numerical': 0.01470412432755529, 'mean_word_length': 2.929544387036167, 'pattern_counts': {'":': 0, '<': 38, '<?xml version=': 0, '>': 37, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '661904', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 2560, 'n_tokens_neox': 2352, 'n_words': 742, 'path': 'notebooks/IO - importing FORWAST.ipynb', 'repo_name': 'elinlucy/brightway2', 'size': '8365'}