<jupyter_start><jupyter_text>[![Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/AyanSinhaMahapatra/scancode-results-analyzer/blob/master/src/notebooks/load_results_files.ipynb)  The Following 6 Cells are Only to be run in Google Colab [Link to Installing Conda in Colab Docs, which is used below](https://towardsdatascience.com/conda-google-colab-75f7c867a522)<jupyter_code>%env PYTHONPATH=
%%bash

MINICONDA_INSTALLER_SCRIPT=Miniconda3-4.5.4-Linux-x86_64.sh
MINICONDA_PREFIX=/usr/local
wget https://repo.continuum.io/miniconda/$MINICONDA_INSTALLER_SCRIPT
chmod +x $MINICONDA_INSTALLER_SCRIPT
./$MINICONDA_INSTALLER_SCRIPT -b -f -p $MINICONDA_PREFIX
conda install --channel defaults conda python=3.6 --yes
conda update --channel defaults --all --yes
import sys
_ = (sys.path.append("/usr/local/lib/python3.6/site-packages"))
!conda install -c conda-forge pandas numpy matplotlib seaborn -y
!git clone -l -s git://github.com/AyanSinhaMahapatra/scancode-results-analyzer.git scancode-results-analyzer
%cd scancode-results-analyzer
!ls
sys.path.append('/content/scancode-results-analyzer/src')<jupyter_output><empty_output><jupyter_text>`load_results_file.py`<jupyter_code>import sys
import numpy as np
import pandas as pd
import os

# Path To Local Folder
sys.path.append('/home/ayan/Desktop/nexB/gsoc20/scancode-results-analyzer/src')
from results_analyze.load_results_package import ResultsDataFramePackage
results_package = ResultsDataFramePackage()<jupyter_output><empty_output><jupyter_text>Import Data From JSON instead of Database, on Google Colab<jupyter_code>json_filename = "lic_scancode_before.json"
json_filepath = os.path.join(results_package.json_input_dir, json_filename)
mock_metadata_filepath = os.path.join(results_package.json_input_dir, results_package.mock_metadata_filename)
path_json_dataframe = results_package.mock_db_data_from_json(json_filepath, mock_metadata_filepath)
path_json_dataframe<jupyter_output><empty_output><jupyter_text>Importing Data From Postgres DataBase Simulating Data going into `ResultsDataFrameFile.create_file_level_dataframe` function, which is called by `ResultsDataFramePackage.create_package_level_dataframe`.Using code snippets from `ResultsDataFramePackage.create_package_level_dataframe`.<jupyter_code>path_json_dataframe = results_package.convert_records_to_json(20)<jupyter_output><empty_output><jupyter_text>Creates `files_dataframe` and breaks at a good example, `file_list` is passed into `ResultsDataFrameFile.create_file_level_dataframe`.<jupyter_code>files_dataframe, metadata_dataframe = results_package.modify_package_level_dataframe(path_json_dataframe)

for package_scan_result in files_dataframe.itertuples():
    file_list = package_scan_result[2]
    if package_scan_result[0] == 0:
        break
type(file_list)
np.shape(file_list)<jupyter_output><empty_output><jupyter_text>One of the entries inside the list of dicts.<jupyter_code>file_list[3]<jupyter_output><empty_output><jupyter_text>Loads List of Dicts into DataFrame<jupyter_code>from results_analyze.load_results_file import ResultsDataFrameFile
results_file = ResultsDataFrameFile()
file_level_dataframe = pd.DataFrame(file_list)<jupyter_output><empty_output><jupyter_text>Viewing DataFrame Columns and their types by calling `DataFrame.dtypes`<jupyter_code>file_level_dataframe.dtypes
file_level_dataframe.shape
results_file.modify_file_level_dataframe?
results_file.modify_file_level_dataframe(file_level_dataframe)
file_level_dataframe.shape
file_level_dataframe.dtypes<jupyter_output><empty_output><jupyter_text>Some entries inside `file_level_dataframe`, here `licenses` column contains list of dicts, where list length is number of license detections per file.<jupyter_code>file_level_dataframe.head(5)<jupyter_output><empty_output><jupyter_text>These lines takes out all these licenses into `DataFrames`.<jupyter_code>lic_level_dataframe = file_level_dataframe.groupby('sha1').licenses.apply(lambda x: pd.DataFrame(x.values[0])).reset_index()
lic_level_dataframe.rename(columns={'level_1': 'lic_det_num'}, inplace=True)<jupyter_output><empty_output><jupyter_text>These are only license level information in the columns.<jupyter_code>lic_level_dataframe.dtypes
results_file.modify_lic_level_dataframe?
lic_level_dataframe = results_file.modify_lic_level_dataframe(lic_level_dataframe)
lic_level_dataframe.dtypes
lic_level_dataframe.set_index('sha1', inplace=True)<jupyter_output><empty_output><jupyter_text>Joins License level and File level Dataframes using Join operations, by the primary key `sha1`.<jupyter_code>merged_df = file_level_dataframe.join(lic_level_dataframe, lsuffix='_file', rsuffix='_lic')
merged_df.reset_index(inplace=True)
merged_df.set_index(['sha1', 'lic_det_num'], inplace=True)<jupyter_output><empty_output><jupyter_text>Notice how under one file, there can be many license rows, and there are 2 Primary key columns on the left, where there is a one-to-many relationship.<jupyter_code>merged_df.head(5)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09265491981785785, 'fraction_numerical': 0.007721243318154821, 'mean_word_length': 9.352459016393443, 'pattern_counts': {'":': 0, '<': 53, '<?xml version=': 0, '>': 53, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '497431', 'license': 'Apache-2.0', 'n_tokens_mistral': 1765, 'n_tokens_neox': 1712, 'n_words': 289, 'path': 'etc/notebooks/load_results_files.ipynb', 'repo_name': 'nexB/scancode-analyzer', 'size': '5051'}