<jupyter_start><jupyter_text>Baseline Supply Chain ScraperThis notebook runs the scraper for our baseline sample. This consists of IDs that we have collected from different sources. Below, we describe the sample we used to begin with. This gave us a first body of IDs. The next phase is randomization (below) which simply tries to guess and record IDs.  Supermarket in Tokyo Sample    Satoshi’s friend took photos of beef sold in a supermarket in Tokyo Sep.13 2021 Recalled Sample        Due to the possibility of nonstandard ID tags, one maker recalls the suspicious ID tags distributed in FY2020. The company announced the ID numbers subject to the recall on its website. Satoshi is confused! Especially for smaller ID numbers on the list, Satoshi found the IDs are already registered in the database. If the IDs are newly issued in 2020, why are they already on the database? One possibility is the tags accidentally came off and the farmers needed to get new tags with the same IDs. Another possibility is that the gov reuses the same IDs several years after death, or slaughter. Considering the average life years of cattle/cow (= 5~6 yrs), the latter is more plausible.     Fukushima Tracking Sample        For accountability to citizens, the gov published the IDs of cattle and their descendants raised within 20 km from the Fukushima nuclear power plant as of March 31 in 2012. The cattle should not be shipped to slaughterhouses because they are considered “contaminated by radiation”. **Import the module:**<jupyter_code>from supply_chain_mapping import supply_chain_data_scraper as scd
from supply_chain_mapping import random_id_generator as rig<jupyter_output><empty_output><jupyter_text>**Original Sample:**<jupyter_code># Import the three samples 
sample_recall, sample_fukushima, sample_tokyo_sm = scd.quickload_samples()

print('# of ids in Tokyo SM sample:', len(sample_tokyo_sm))
print('# of ids in Fukushima sample:', len(sample_fukushima))
print('# of ids in Recall sample:', len(sample_recall))

master_sample = sample_tokyo_sm.append(sample_fukushima)
master_sample = master_sample.append(sample_recall)

print('Total # of ids in our baseline sample:', len(master_sample))<jupyter_output># of ids in Tokyo SM sample: 7
# of ids in Fukushima sample: 340
# of ids in Recall sample: 7000
Total # of ids in our baseline sample: 7347<jupyter_text>*Review how much of the initial sample has been collected:*<jupyter_code>uncollected_sample = scd.check_collected_data(list(master_sample['id'].values))<jupyter_output>10 IDs in the submitted list have not been collected
0 new IDs in the temporary folder have been appended to 98261 IDs in the collected folder
There are a total of 98261  collected IDs
There are a total of 137847 failed IDs<jupyter_text>**Generate completely random and unique IDs that do not overlap with collected ones (including failed ones):**We go through this step first in order to identify rules in how IDs are generated which we can then use to create targetted random IDs.<jupyter_code>#uncollected_random_ids = rig.random_cowid_generator(batch_size=10000)<jupyter_output><empty_output><jupyter_text>**Examine Generate random IDs based on collected ones:**<jupyter_code># Check if there are any patterns in the digits of the IDs that do not fail 
import pandas as pd
collected_ids, failed_ids = scd.get_collected_ids()
collected_ids = pd.DataFrame({'collected':collected_ids})
for i in range(2,4): collected_ids['dig'+str(i)] = collected_ids['collected'].str[:i]
print(collected_ids['dig2'].value_counts().sort_index())

failed_ids = pd.DataFrame({'failed':failed_ids})
for i in range(2,4): failed_ids['dig'+str(i)] = failed_ids['failed'].str[:i]
print(failed_ids['dig2'].value_counts().sort_index())
list_of_random_ids = rig.bounded_random_ids(['08','11','12','13','14','15','16'], 2, batch_size=100000, lower_bound=11000000, upper_bound=14000000)<jupyter_output><empty_output><jupyter_text>Scraping the original supply chain sample!<jupyter_code>import os
from path import Path
project_path = Path(os.getcwd()).parent

print('The number of uncollected IDs from the original sample is',len(list_of_random_ids))
print('The project path is:\n',project_path)
final_data, complete_data, failures  = scd.supply_chain_data_scraper(list_of_random_ids, 
                                                                    japanese=True, 
                                                                    split_save=200,
                                                                    temporary_file_path=project_path+'/data/temporary',
                                                                    final_file_path=project_path+'/data/collected')<jupyter_output>Last error was:
list index out of range
Scraped 2021 IDs from the 100000 originally submitted
Current loop is working on the 98010 remaining IDs
Working on ID # 2021 of the current loop<jupyter_text>Scraper Evaluation To evaluate a reasonable stopping point we can use the marginal entropy of each additional unit scraped. In other words, how much new information about our network do we get with each additional cow. The following functions describe the number of nodes and number of connections for each additional ID scraped. It randomly samples IDs until the entire sample is acccounted for and then repeats this process in a monte-carlo simulation. The resulting rates of change and confidence intervals are graphed.<jupyter_code>from supply_chain_mapping import data_cleaning_and_processing as dc

complete_data = dc._load_complete_data(rename=True)
plot_scraper_entropy(5000, complete_data)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.06583850931677018, 'fraction_numerical': 0.022892635314995562, 'mean_word_length': 4.63036963036963, 'pattern_counts': {'":': 0, '<': 26, '<?xml version=': 0, '>': 26, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 8, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '419646', 'license': 'Unlicense', 'n_tokens_mistral': 1626, 'n_tokens_neox': 1471, 'n_words': 628, 'path': 'notebooks/baseline_supply_chain_scraper.ipynb', 'repo_name': 'losDaniel/supply-chain-vulnerability', 'size': '5635'}