<jupyter_start><jupyter_text>This notebook gives a fairly complicated example of building a Sankey diagram from the sample "fruit" database. Other examples (TODO) break this process down into simpler stages.<jupyter_code>from sankeyview import *
from sankeyview.jupyter import show_sankey, show_view_graph
from IPython.display import SVG<jupyter_output><empty_output><jupyter_text>Load the dataset:<jupyter_code>dataset = Dataset.from_csv('fruit_flows.csv', 'fruit_processes.csv')<jupyter_output><empty_output><jupyter_text>This made-up dataset describes flows from farms to consumers:<jupyter_code>dataset._flows.head()<jupyter_output><empty_output><jupyter_text>Additional information is available in the process dimension table:<jupyter_code>dataset._processes.head()<jupyter_output><empty_output><jupyter_text>We'll also define some partitions that will be useful:<jupyter_code>farm_ids = ['farm{}'.format(i) for i in range(1, 16)]

farm_partition_5 = Partition.Simple('process', [('Other farms', farm_ids[5:])] + farm_ids[:5])
partition_fruit = Partition.Simple('material', ['bananas', 'apples', 'oranges'])
partition_sector = Partition.Simple('process.sector', ['government', 'industry', 'domestic'])<jupyter_output><empty_output><jupyter_text>Now define the Sankey diagram definition.- Process groups represent sets of processes in the underlying database. The underlying processes can be specified as a list of ids (e.g. `['inputs']`) or as a Pandas query expression (e.g. `'function == "landfill"'`).- Waypoints allow extra control over the partitioning and placement of flows.<jupyter_code>nodes = {
    'inputs':     ProcessGroup(['inputs'], title='Other inputs'),
    'compost':    ProcessGroup('function == "composting stock"', title='Compost'),
    'farms':      ProcessGroup('function in ["allotment", "large farm", "small farm"]', farm_partition_5),
    'eat':        ProcessGroup('function == "consumers" and location != "London"', partition_sector,
                               title='consumers by sector'),
    'landfill':   ProcessGroup('function == "landfill" and location != "London"', title='Landfill'),
    'composting': ProcessGroup('function == "composting process" and location != "London"', title='Composting'),

    'fruit':        Waypoint(partition_fruit, title='fruit type'),
    'w1':           Waypoint(direction='L', title=''),
    'w2':           Waypoint(direction='L', title=''),
    'export fruit': Waypoint(Partition.Simple('material', ['apples', 'bananas', 'oranges'])),
    'exports':      Waypoint(title='Exports'),
}<jupyter_output><empty_output><jupyter_text>The ordering defines how the process groups and waypoints are arranged in the final diagram. It is structured as a list of vertical *layers* (from left to right), each containing a list of horizontal *bands* (from top to bottom), each containing a list of process group and waypoint ids (from top to bottom).<jupyter_code>ordering = [
    [[], ['inputs', 'compost'], []],
    [[], ['farms'], ['w2']],
    [['exports'], ['fruit'], []],
    [[], ['eat'], []],
    [['export fruit'], ['landfill', 'composting'], ['w1']],
]<jupyter_output><empty_output><jupyter_text>Bundles represent flows in the underlying database:<jupyter_code>bundles = [
    Bundle('inputs', 'farms'),
    Bundle('compost', 'farms'),
    Bundle('farms', 'eat', waypoints=['fruit']),
    Bundle('farms', 'compost', waypoints=['w2']),
    Bundle('eat', 'landfill'),
    Bundle('eat', 'composting'),
    Bundle('composting', 'compost', waypoints=['w1', 'w2']),
    Bundle('farms', Elsewhere, waypoints=['exports', 'export fruit', ]),
]<jupyter_output><empty_output><jupyter_text>Finally, the process groups, waypoints, bundles and ordering are combined into a Sankey diagram definition (SDD). When applied to the dataset, the result is a Sankey diagram!<jupyter_code>sdd = SankeyDefinition(nodes, bundles, ordering, flow_partition=dataset.partition('material'))
sankey = show_sankey(sdd, dataset, width=800, height=500)
sankey
# For viewing on nbviewer, save a static version of the diagram
SVG(sankey.svg)<jupyter_output><empty_output><jupyter_text>To help get a better understanding of what's going on, it may be helpful to look at the intermediate "view graph":> This depends on graphviz being available<jupyter_code>show_view_graph(sdd)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.14463380930446798, 'fraction_numerical': 0.004606172270842929, 'mean_word_length': 6.084828711256117, 'pattern_counts': {'":': 1, '<': 41, '<?xml version=': 0, '>': 42, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '427635', 'license': 'MIT', 'n_tokens_mistral': 1349, 'n_tokens_neox': 1301, 'n_words': 367, 'path': 'examples/Fruit - complete example.ipynb', 'repo_name': 'LeoPaoli/sankeyview', 'size': '4342'}