<jupyter_start><jupyter_text>ODOC Public Inmate DataThis notebook is intended as a start for research of the ODOC data.  To use this notebook, follow the setup instructiion found [here](https://github.com/codefortulsa/odoc-parse). Then download the data published [here](http://doc.publishpath.com/odoc-public-inmate-data).  Unzip the file and place the files in a subdirectory called 'data'.The set of files includes a ReadMe.txt which describes the files and their fixed formats. The sections of this notebook show description of each file and how to import it into pandas dataframes.  NOTE: the widths variables differ slighty from the description to handle some difference in the data.<jupyter_code>import pandas as pd<jupyter_output><empty_output><jupyter_text>Schedule A - Profile Data Layout ``` ======================================================= Name                            Null?    Type ------------------------------- -------- ---- DOC_NUM                         NOT NULL NUMBER(10) LAST_NAME                                VARCHAR2(30) FIRST_NAME                               VARCHAR2(30) MIDDLE_NAME                              VARCHAR2(30) SUFFIX                                   VARCHAR2(5) LAST_MOVE_DATE                           DATE 'DD-MMM-YY' (9) FACILITY                                 VARCHAR2(40) BIRTH_DATE                               DATE 'DD-MMM-YY' (9) SEX                                      VARCHAR2(1) RACE                                     VARCHAR2(40) HAIR                                     VARCHAR2(40) HEIGHT_FT                                VARCHAR2(2) HEIGHT_IN                                VARCHAR2(2) WEIGHT                                   VARCHAR2(4) EYE                                      VARCHAR2(40) STATUS                                   VARCHAR2(10) ```<jupyter_code>file = 'data/Vendor_Profile_Sample_Text.dat'

# uncomment this line to use the full dataset
# file = 'data/Vendor_Profile_Extract_Text.dat'

names = [
    "DOC_NUM"       
    ,"LAST_NAME"     
    ,"FIRST_NAME"    
    ,"MIDDLE_NAME"   
    ,"SUFFIX"        
    ,"LAST_MOVE_DATE"
    ,"FACILITY"      
    ,"BIRTH_DATE"    
    ,"SEX"           
    ,"RACE"          
    ,"HAIR"          
    ,"HEIGHT_FT"     
    ,"HEIGHT_IN"     
    ,"WEIGHT"        
    ,"EYE"           
    ,"STATUS"
]
widths = [
    11,
    30,
    30,
    30,
    5,
    9,
    40,
    9,
    1,
    40,
    40,
    2,
    2,
    4,
    40,
    10
]

profile_df = pd.read_fwf(file, 
    header=None,
    widths=widths,
    names=names)
                 
profile_df.head(20)<jupyter_output><empty_output><jupyter_text>Schedule B - Alias Data Layout```======================================================= Name                            Null?    Type ------------------------------- -------- ---- DOC_NUM                         NOT NULL NUMBER(10) LAST_NAME                                VARCHAR2(30) FIRST_NAME                               VARCHAR2(30) MIDDLE_NAME                              VARCHAR2(30) SUFFIX                                   VARCHAR2(5)```<jupyter_code>file = 'data/Vendor_Alias_Sample_Text.dat'

# uncomment this line to use the full dataset
# file = 'data/Vendor_Alias_Extract_Text.dat'

names = [
         "DOC_NUM",
         "LAST_NAME",
         "FIRST_NAME",
         "MIDDLE_NAME",
         "SUFFIX"
]

widths = [
          11,
          30,
          30,
          30,
          5
]

alias_df = pd.read_fwf(file, 
    header=None,
    widths=widths,
    names=names)
                 
alias_df.head(20)<jupyter_output><empty_output><jupyter_text>Schedule C - Sentence Data Layout```=======================================================Incarcerated_Term_In_Years = 9999 indicates a death sentenceIncarcerated_Term_In_Years = 8888 indicates a life without parole sentenceIncarcerated_Term_In_Years = 7777 indicates a life sentence======================================================= Name                            Null?    Type ------------------------------- -------- ---- DOC_NUM                         NOT NULL NUMBER(10) STATUTE_CODE                    NOT NULL VARCHAR2(40) SENTENCING_COUNTY                        VARCHAR2(40) JS_DATE                                  DATE 'YYYYMMDD' (8) CRF_NUMBER                               VARCHAR2(40) INCARCERATED_TERM_IN_YEARS               NUMBER(10,2) PROBATION_TERM_IN_YEARS                  NUMBER(10,2) ```<jupyter_code>file = 'data/Vendor_sentence_Sample_Text.dat'

# uncomment this line to use the full dataset
# file = 'data/Vendor_sentence_Extract_Text.dat'

names =[
    "DOC_NUM",                   
    "STATUTE_CODE",              
    "SENTENCING_COUNTY",         
    "JS_DATE",                   
    "CRF_NUMBER",                
    "INCARCERATED_TERM_IN_YEARS",
    "PROBATION_TERM_IN_YEARS"
]


# first char in file is blank so offset by one in DOC_NUM field
widths = [
    11,
    40,
    40,
    9,
    40,
    13,
    13
]

sentence_df = pd.read_fwf(file, 
    header=None,
    widths=widths,
    names=names)
                 
sentence_df.head(20)<jupyter_output><empty_output><jupyter_text>Schedule D - Offense Codes Layout======================================================= ``` Name                            Null?    Type ------------------------------- -------- ---- STATUTE_CODE                    NOT NULL VARCHAR2(38) DESCRIPTION                     NOT NULL VARCHAR2(40) VIOLENT                                  VARCHAR2(1)```<jupyter_code># NOTE:  No sample file for this data

file = 'data/Vendor_Offense_Extract_Text.dat'


names = [
    "STATUTE_CODE",
    "DESCRIPTION", 
    "VIOLENT",
]

widths = [
    38,
    40,
    1
]

offense_df = pd.read_fwf(file, 
    header=None,
    widths=widths,
    names=names)
                 
offense_df.head(20)<jupyter_output><empty_output><jupyter_text>Examples- profile_df:  demographics- alias_df: - sentence_df: statute- offense_dfNOTE:  Don't forget, these examples are based on sample data.<jupyter_code>active_inmates = profile_df.query("STATUS == 'Active'")

joined_df = pd.merge(active_inmates, sentence_df, on='DOC_NUM', how='left')
joined_df.head(20)
import numpy as np

import matplotlib.pyplot as plt


graph_df = joined_df.groupby(['RACE'])['DOC_NUM'].count()
graph_df.plot.pie(y='RACE', figsize=(5, 5))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.14898989898989898, 'fraction_numerical': 0.024147727272727272, 'mean_word_length': 1.6931576710582235, 'pattern_counts': {'":': 0, '<': 25, '<?xml version=': 0, '>': 25, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 16, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '117465', 'license': 'MIT', 'n_tokens_mistral': 2003, 'n_tokens_neox': 1834, 'n_words': 407, 'path': 'odoc_download.ipynb', 'repo_name': 'julia-codes/odoc-parse', 'size': '6336'}