<jupyter_start><jupyter_text>Import Libraries & Set `filepath` Variable<jupyter_code>import os
import json
import pandas as pd

filepath = '/home/prp1277/Github-Repos/datasets/json/RedditExcel'<jupyter_output><empty_output><jupyter_text>List The Files in the Target Directory<jupyter_code>os.listdir(filepath)<jupyter_output><empty_output><jupyter_text>Create an Iterable String[Loop Link](https://www.techbeamers.com/python-list-all-files-directory/)`r` = root`d` = dirs`f` = files`item` = full path<jupyter_code>dir_files = []

for r, d, f in os.walk(filepath):
    for item in f:
        if '.json' in item:
            dir_files.append(os.path.join(r, item))

for item in dir_files:
    print(item)<jupyter_output>/home/prp1277/Github-Repos/datasets/json/RedditExcel/2020-01-11T12_45_00.8389322Z.json
/home/prp1277/Github-Repos/datasets/json/RedditExcel/2020-01-21T12_45_01.9506855Z.json
/home/prp1277/Github-Repos/datasets/json/RedditExcel/2019-11-13T12_45_01.3840842Z.json
/home/prp1277/Github-Repos/datasets/json/RedditExcel/2019-11-19T12_45_01.3002403Z.json
/home/prp1277/Github-Repos/datasets/json/RedditExcel/2019-12-19T12_45_01.3500134Z.json
/home/prp1277/Github-Repos/datasets/json/RedditExcel/2019-12-14T12_45_00.3978264Z.json
/home/prp1277/Github-Repos/datasets/json/RedditExcel/2019-12-13T12_45_01.3899260Z.json
/home/prp1277/Github-Repos/datasets/json/RedditExcel/2020-02-10T12_45_00.6581482Z.json
/home/prp1277/Github-Repos/datasets/json/RedditExcel/2020-01-27T12_45_01.3321668Z.json
/home/prp1277/Github-Repos/datasets/json/RedditExcel/2019-11-15T12_45_01.0134167Z.json
/home/prp1277/Github-Repos/datasets/json/RedditExcel/2020-02-28T12_45_00.9294110Z.json
/home/prp1277/Github-Repos/datasets/json/Re[...]<jupyter_text>Cleaning the FilesMost of the files have an extra `}` at the end of them that needs to be removed. Find the length of the file, check to see if it ends with `}}` and if it does, remove it and save the file. Use a Sample File to Test the Script[Script Link](https://www.geeksforgeeks.org/python-check-whether-a-string-is-valid-json-or-not/)<jupyter_code>test_file = '/home/prp1277/Github-Repos/datasets/json/RedditExcel/2020-03-08T17_13_50.6856489Z.json'
test_file2 = '/home/prp1277/Github-Repos/datasets/json/json-test.json'

try:
    json_obj = json.loads(test_file2)
    print('Valid JSON')
except ValueError as e:
    print('Not Valid JSON')
try:
    json_obj2 = json.dumps(test_file)
    print('Valid JSON')
except ValueError as e:
    print('Not Valid JSON')
f = open(test_file2, "r")
json.load(f)
with os.scandir(filepath) as dir_entries:
    for entry in dir_entries:
        if not entry.name.startswith('.') and entry.is_file():
            print(filepath + '/' + entry.name)
posts = json.load(open('/home/prp1277/Github-Repos/datasets/json/json-test.json'))['data']['children']
df = pd.DataFrame(
    ([post['data']['author'], post['data']['title'], post['data']['selftext'], post['data']['url']] for post in posts), 
    columns=['author', 'title', 'selftext','url'])
df
posts = json.load(open(filepath + '/2019-11-09T19_46_03.6912336Z.json'))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.14738865748157642, 'fraction_numerical': 0.10925985261134252, 'mean_word_length': 5.038684719535784, 'pattern_counts': {'":': 0, '<': 16, '<?xml version=': 0, '>': 16, 'https://': 2, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '649624', 'license': 'MIT', 'n_tokens_mistral': 1452, 'n_tokens_neox': 1246, 'n_words': 176, 'path': 'venv/src/Notebooks/RedditData.ipynb', 'repo_name': 'prp1277/finance', 'size': '3121'}