<jupyter_start><jupyter_text>ZYu Data Helper 1 Load files to `./train/` and create `/train/labels.json`STEPSFirst unzip `lines.tgz` into folder "data/line_data", also put the file `lines.txt` there.- Read `line.txt` into a data-frame, which will serve as `index` to data-files- Choose some files from the `index` to serve as train data- Copy those files to `data/train`- Create `labels.json` and place it in `data/train`<jupyter_code>import warnings
warnings.filterwarnings('ignore')

import re
import os

import numpy as np
import pandas as pd

import matplotlib.pyplot as plt
%matplotlib inline

from tqdm import tnrange, tqdm_notebook

from sklearn.model_selection import train_test_split

import json<jupyter_output><empty_output><jupyter_text>Preliminary EDA<jupyter_code>!head -23 lines.txt
!head -25 lines_copy.txt<jupyter_output>a01-000u-00 ok 154 19 408 746 1661 89 A|MOVE|to|stop|Mr.|Gaitskell|from
a01-000u-01 ok 156 19 395 932 1850 105 nominating|any|more|Labour|life|Peers
a01-000u-02 ok 157 16 408 1106 1986 105 is|to|be|made|at|a|meeting|of|Labour
a01-000u-03 err 156 23 430 1290 1883 70 M Ps|tomorrow|.|Mr.|Michael|Foot|has
a01-000u-04 ok 157 20 395 1474 1830 94 put|down|a|resolution|on|the|subject
a01-000u-05 err 156 21 379 1643 1854 88 and|he|is|to|be|backed|by|Mr.|Will
a01-000u-06 ok 159 20 363 1825 2051 87 Griffiths|,|M P|for|Manchester|Exchange|.
a01-000x-00 ok 182 30 375 748 1561 148 A|MOVE|to|stop|Mr.|Gaitskell|from|nominating
a01-000x-01 ok 181 23 382 924 1595 148 any|more|Labour|life|Peers|is|to|be|made|at|a
a01-000x-02 ok 181 30 386 1110 1637 140 meeting|of|Labour|0M Ps|tomorrow|.|Mr.|Michael
a01-000x-03 ok 179 22 375 1276 1584 154 Foot|has|put|down|a|resolution|on|the|subject
a01-000x-04 ok 173 25 397 1458 1647 148 and|he|is|to|be|backed|by|Mr.|Will|Griffiths|,
a01-000x-05 ok 173 16 393 1635 1082 [...]<jupyter_text>NOTEThere are lines (4, 13 ...) with label containing spaces that can cause trouble when spliting by " ":`a01-000x-05 ok 173 16 393 1635 1082 155 0M P|for|Manchester|Exchange|.`EDA shows 13317 rows have 9 fields, but there are 23 rows with 10 fields, and 13 rows with 11 fields, these will cause trouble if reading the 'lines_copy.txt' file into CSV.<jupyter_code># SHOW LINES WITH > 9 FIELDS
# count the number of fields if delimited by " "
from collections import defaultdict

file_in = 'lines_copy.txt'

fhand = open(file_in)

num_count = defaultdict(int)

for line in fhand:
    num = len(line.split(" "))
    num_count[num] +=1
    if num > 9:
        print(line)

fhand.close()

num_count<jupyter_output>a01-000u-03 err 156 23 430 1290 1883 70 M Ps|tomorrow|.|Mr.|Michael|Foot|has

a01-000u-06 ok 159 20 363 1825 2051 87 Griffiths|,|M P|for|Manchester|Exchange|.

a01-000x-02 ok 181 30 386 1110 1637 140 meeting|of|Labour|0M Ps|tomorrow|.|Mr.|Michael

a01-000x-05 ok 173 16 393 1635 1082 155 0M P|for|Manchester|Exchange|.

a01-003-01 ok 176 29 319 1078 1787 126 large|majority|of|Labour|M Ps|are|likely|to

a01-003-03 ok 176 28 316 1441 1750 70 Foot's|line|will|be|that|as|Labour|M Ps

a01-003u-02 ok 161 21 356 1162 1908 96 M Ps|are|likely|to|turn|down|the|Foot-

a01-003u-04 ok 161 13 351 1515 1869 103 be|that|as|Labour|M Ps|opposed|the

a01-003x-01 ok 176 28 307 1046 1831 170 large|majority|of|Labour|0M Ps|are|likely|to|turn

a01-003x-03 err 169 26 325 1423 1839 128 will|be|that|as|Labour|0M Ps|opposed|the|Govern-

a01-030-02 ok 176 20 305 1083 1815 100 M Ps|yesterday|:|"|I|have|no|knowledge|of|secret|negotiations|.|"

a01-030u-02 err 170 22 330 1163 1856 93 meeting|.|But|he|told|M Ps|yesterd[...]<jupyter_text>1. Read meta-data into a dataframe<jupyter_code>file_in = 'lines_copy.txt'

fhand = open(file_in)
data_list = [] 

for line in fhand:
    item = line.split(" ")   # split the line
    num = len(item)          # number of fields if line delimited by " "
    
    if num == 9:             # good lines
        label = item[8]      # last field is label if only 9 field
    else:                    # bad lines with spaces in "label"
        label = " ".join(item[8:])    # join the remaining fields - this is label

    new_data  ={'id': item[0], 
                'wseg_status': item[1], 
                'graylevel': int(item[2]), 
                'num_components': int(item[3]), 
                'x': int(item[4]), 
                'y': int(item[5]), 
                'w': int(item[6]), 
                'h': int(item[7]), 
                'label': label.rstrip().replace("|", " ")}   # remove "\n", replace "|" with " "
    
    data_list.append(new_data)
    
fhand.close()

df = pd.DataFrame(data_list, 
                  columns=['id', 'wseg_status', 'graylevel', 'num_components', 'x', 'y', 'w', 'h', 'label'])

df.head()
df.dtypes
plt.figure(1, figsize=(15,4))
plt.subplot(121)
df['w'].hist(bins=50)
plt.subplot(122)
df['h'].hist(bins=50)
plt.show()<jupyter_output><empty_output><jupyter_text>2. Choose files from the dataframe- `wseg_status` has 11344 `ok`, 2009 `err`- perform a random split on our data<jupyter_code>df_train, df_test = train_test_split(df, test_size=0.2)

print(df_train.shape)
print(df_test.shape)

df_test.head()<jupyter_output>(10682, 9)
(2671, 9)<jupyter_text>3. Copy the chosen files to the destination and create jsonFor example, the first line:- source: `./line_data/a03/a03-066/a03-066-01.png`- destination: `./train/a03-066-01.png`- label: `remove|these|doubts|,|and|to|persuade|the`Work flow:- initialize `file_list={}`- loop through each line in the data frame- use `id` field to create `source` and `destination`, `file_name`- use `label` to create `label`- copy `source` to `destination`- add `{file_name: label}` to `file_list`- create `labels.json` and put it in `./train`  Define helper functions<jupyter_code>def clean_folder(destination_dir):
    """
    only delete files and to used the os.path.join() method  
    If you also want to remove subdirectories, uncomment the elif statement.
    """
    
    import os, shutil
    folder = destination_dir
    
    count = 0
    for the_file in os.listdir(folder):
        file_path = os.path.join(folder, the_file)
        try:
            if os.path.isfile(file_path):
                os.unlink(file_path)
                count += 1
            #elif os.path.isdir(file_path): shutil.rmtree(file_path)
        except Exception as e:
            print(e)
    print("Successfully removed", count, "files")        
    

def copy_files(df, source_dir, destination_dir):
    """
    copying files from source to destionation
    INPUT: df is a dataframe containing file information
    OUTPUT: json_list of {file_name:file_label} pairs    
    """

    from shutil import copy2
    json_dict = dict()

    for i in tnrange(df.shape[0], desc='copying files'):    

        # read data from each row
        file_data = df.iloc[i]            # each line in dataframe
        file_id = file_data['id']         # data for file
        file_name = file_id + '.png'      # file_name
        file_label = file_data['label']   # label
        
        # create source and destination 
        id_pc = file_id.split("-")
        source = source_dir + id_pc[0] + "/" + id_pc[0] + "-" + id_pc[1] + "/" + file_name
        destination = destination_dir  + file_name

        # copy source to destination
        copy2(source, destination)

        # append the {file_name: file_label} to json_list
        json_dict[file_name]= file_label

    return json_dict

def copy_deslanted_files(df, source_dir, destination_dir):
    """
    copying files from source to destionation
    INPUT: df is a dataframe containing file information
    OUTPUT: json_list of {file_name:file_label} pairs    
    """

    from shutil import copy2
    json_dict = dict()

    for i in tnrange(df.shape[0], desc='copying files'):    

        # read data from each row
        file_data = df.iloc[i]            # each line in dataframe
        file_id = file_data['id']         # data for file
        file_name = file_id + '.png'      # file_name
        file_label = file_data['label']   # label
        
        # create source and destination 
        id_pc = file_id.split("-")
        source = source_dir + file_name
        destination = destination_dir  + file_name

        # copy source to destination
        copy2(source, destination)

        # append the {file_name: file_label} to json_list
        json_dict[file_name]= file_label

    return json_dict

def write_json_file(destination_dir, json_dict):
    """
    write json to the 'labels.json file in the destination directory
    """

    with open(destination_dir + 'labels.json', 'w') as fout:
        json.dump(json_dict, fout)
        
    print(destination_dir+'labels.json has been created.')<jupyter_output><empty_output><jupyter_text>Copy files listed in the input dataframe to destination<jupyter_code>source_dir = './deslanted_lines/'
destination_dir = './train_deslanted/'
# OPTIONAL: clean the destination folder if needed
clean_folder(destination_dir)
# copying files from source to destination
json_dict = copy_deslanted_files(df_train, source_dir, destination_dir)

print("Copied", len(json_dict), "files.")
# take a look
!ls train | wc<jupyter_output>10683   10683  162519<jupyter_text>Write json to `labels.json` and put it in destination directory<jupyter_code>write_json_file(destination_dir, json_dict)
with open('./train/labels.json') as f:
    train_json = json.load(f)
    
for key, val in list(train_json.items())[0:10]:
    print('{:20s}    {}'.format(key, val))<jupyter_output>g06-045n-09.png         Dr. <NAME> .
g04-007-07.png          Burge became Bishop of Southwark .
p02-017-00.png          Gay , sitting opposite Gavin on the terrace could
k04-093-08.png          of the retainer .
b05-042-03.png          explanatory TV pictures , into becoming a highly
g06-042a-04.png         of death , the new German curative method
b06-045-01.png          to the two major conferences of the movement and
a02-032-04.png          of the people is the supreme law " ) reports : " The
c06-111-02.png          The Marriage-Go-Round is not entirely a waste of
f04-024-04.png          pestle such as chemists use , with some hairs<jupyter_text>copy some files to `./test` as well<jupyter_code>source_dir = './deslanted_lines/'
destination_dir = './test_deslanted/'

# OPTIONAL: clean the destination folder if needed
clean_folder(destination_dir)
# copying files from source to destination
json_dict = copy_deslanted_files(df_test, source_dir, destination_dir)

print("Copied", len(json_dict), "files.")
write_json_file(destination_dir, json_dict)
with open('./test/labels.json') as f:
    train_json = json.load(f)
    
for key, val in list(train_json.items())[0:10]:
    print('{:20s}    {}'.format(key, val))<jupyter_output>a01-049u-05.png         Committee , which is headed by another
l04-147-00.png          Unless we can get a definite line on him it 's
e01-018-04.png          among the effects left by <NAME> is an
n06-182-00.png          The rest of the men there said nothing but
a01-049x-02.png         television audience it was " current Washington
n02-045-05.png          party to the best of my ability - saying the
d06-003-04.png          that children should be molly-coddled - they must be made
m01-115-08.png          them .
d01-019-02.png          present state of the Gospel is the result of an accident-prone history . The
l01-199-09.png          and it would not be of much use if
{'fraction_non_alphanumeric': 0.1112666200139958, 'fraction_numerical': 0.08213785864240727, 'mean_word_length': 3.2596870342771984, 'pattern_counts': {'":': 1, '<': 33, '<?xml version=': 0, '>': 35, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '210470', 'license': 'MIT', 'n_tokens_mistral': 4690, 'n_tokens_neox': 3884, 'n_words': 1146, 'path': 'data/zyu_data_helper_v1.ipynb', 'repo_name': 'danvolk/LineHTR', 'size': '11445'}