<jupyter_start><jupyter_text>'message.json' file structureFiles are composed of 5 or 6 categories (mainly 6) :* **participants** : List of participants as `{'name': Name}`* **messages** : List of messages. See later.* **is_still_participant** : Bool. If am still in conversation* **thread_type** : *Regular* (friends), *RegularGroup* (Group Chat) or Pending (*not friends*)* **thread_path** : file in which the `message.json` is found* **title** (Optional) : title of conversation (or name of participants) Messages* **type** Type of message    - Generic (*only one we keep*)    - Call    - Subscribe    - Unsubscribe (leave group)    - Share (send an attachment)    - Plan (evenement?)* **timestamp_ms*** **sender_name** : *important pour savoir qui envoie les messages!!!!** **content** : *partie importante. contient le message** **reactions** (Optional) : reactions to a message. Presented as a list of {'actor': Actor, 'reaction' : Reaction}* **stickers** (Optional) : Contains emoji. Filter stickers... Can discard for now...WE DISCARD MESSAGES IF THEY CONTAIN ANY OF THOSE THINGS.May be possible to parse to see if the messages not having the `Name sent a photo` pattern are useful...* **photos*** **gifs*** **audio_files*** **plan*** **videos*** **files**<jupyter_code>def show_file_structure(json_file):
    
    with open(json_file) as file:
        conversation = json.load(file)
        
#     print(json.dumps(conversation, indent=4, sort_keys=True))

    messages = conversation['messages']
    
    for message in messages:
        
        if message['type'] == 'Generic' and 'content' in message and chosen_categ in message:
            print(json.dumps(message, indent=4, sort_keys=True))
# parsing a message.json file

def parse_conversation(json_file, all_conversations):
    
    with open(json_file) as file:
        conversation = json.load(file)
        
    messages = conversation['messages']
    
    for message in messages:
        if message_is_valid(message):
            
            all_conversations.append({'content': message['content'],
                                      'sender_name': message['sender_name'],
                                      'timestamp_ms': message['timestamp_ms']})
        

def message_is_valid(message):
    
    # if message contains unwanted category we discard
    if UNWANTED_CATEGORIES.intersection(set([categ for categ in message])):
        return False

    # weird case
    elif 'content' not in message or 'sender_name' not in message:
        return False
    
    return True
all_conversations = []


for i, dir in enumerate(os.listdir(DATADIR)):
    
    conv_dir = os.path.join(DATADIR, dir)
    conv_file = os.path.join(conv_dir, FILENAME)

    parse_conversation(conv_file, all_conversations)
    
with open(OUTPUT_FILENAME, 'w') as fout:
    json.dump(all_conversations, fout)
    
### Preprocess the facebook data<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11987704918032786, 'fraction_numerical': 0.0017076502732240437, 'mean_word_length': 3.3781763826606874, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '1', 'dir': 'jupyter-structured-clean-dedup', 'id': '354276', 'license': 'MIT', 'n_tokens_mistral': 868, 'n_tokens_neox': 838, 'n_words': 269, 'path': 'explore_facebook_data.ipynb', 'repo_name': 'patricebechard/chatbot', 'size': '2928'}