<jupyter_start><jupyter_text>Exploratory Data Analysis on all fluid inclusion data (not geolocated)- Clean data - DONE- Use [pandas profiling](https://github.com/pandas-profiling/pandas-profiling)  - This did not work due to:      `MemoryError: Unable to allocate 319. TiB for an array with shape (43812872618454,) and data type float64`- Use [missingno](https://github.com/ResidentMario/missingno) as first pass- Use [seaborn](https://seaborn.pydata.org/examples/index.html) for quick plotsData files:<jupyter_code>clean = './data/cleaned_types.csv'<jupyter_output><empty_output><jupyter_text>Imports<jupyter_code>import numpy as np
import pandas as pd

import missingno as msno
from pandas_profiling import ProfileReport

import matplotlib.pyplot as plt
import seaborn as sns

from IPython.display import display

pd.options.display.max_columns = None<jupyter_output><empty_output><jupyter_text>First pass<jupyter_code>df = pd.read_csv(clean)
df.head()
df['Well ID'].unique()
len(df['Well ID'].unique())
df.size
df.shape
df.AMU1.plot()
# Yields MemoryError: Unable to allocate 319. TiB for an array with shape (43812872618454,) and data type float64
#profile = ProfileReport(df, title='COSO fluid inclusions profiling')
#profile<jupyter_output><empty_output><jupyter_text>We can't load the file into pandas profiling as it's too large but the values need to be floats, we could perhaps change them to float32 to save some space but that probably won't be enough.From [this](https://www.zdnet.com/article/how-much-does-1-5tb-of-ram-cost/) webpage, 1TB of RAM costs USD12,000 so for 319 TB the base cost would be USD3,828,000  `missingno`<jupyter_code>msno.matrix(df)<jupyter_output><empty_output><jupyter_text>Map chemical names to AMU<jupyter_code>df.columns[0:20]
amu_map = {'AMU16':'CH4', 'AMU28':'N2', 'AMU39':'propene',
           'AMU40':'Ar', 'AMU43':'propane', 'AMU44':'CO2'}

df = df.rename(mapper=amu_map, axis=1)
df['N2/Ar'] = df['N2']/df['Ar']
df['CO2/CH4'] = df['CO2']/df['CH4']
df['CO2/N2'] = df['CO2']/df['N2']
df['R1'] = df['N2/Ar'] + df['CO2/CH4']
df['R2'] = df['N2/Ar'] + df['CO2/N2']
df['dCO2/N2'] = df['CO2/N2'].diff()
df['shallow_fluids1'] = df['N2/Ar'] < 200
df['shallow_fluids2'] = df['CO2/CH4'] < 4
df['shallow_fluids3'] = df['propane']/df['propene'] > 1
df['shallow_fluids4'] = 1/df['R1'] > 0.5
df = df.replace([np.inf, -np.inf], 0)
df['CO2/N2'].plot()
df['dCO2/N2'].plot()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1642651296829971, 'fraction_numerical': 0.04981473857554549, 'mean_word_length': 5.962750716332378, 'pattern_counts': {'":': 0, '<': 23, '<?xml version=': 0, '>': 23, 'https://': 4, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '346169', 'license': 'CC0-1.0', 'n_tokens_mistral': 1013, 'n_tokens_neox': 932, 'n_words': 212, 'path': 'EDA_wip_jackie.ipynb', 'repo_name': 'BKJackson/geothermal_fluidinclusions_wells', 'size': '2429'}