<jupyter_start><jupyter_text>So We have 27486 tweets in the train set and 3535 tweets in the test set<jupyter_code>print(train.info())
print("=================")
print(test.info())<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 27481 entries, 0 to 27480
Data columns (total 4 columns):
 #   Column         Non-Null Count  Dtype 
---  ------         --------------  ----- 
 0   textID         27481 non-null  object
 1   text           27480 non-null  object
 2   selected_text  27480 non-null  object
 3   sentiment      27481 non-null  object
dtypes: object(4)
memory usage: 858.9+ KB
None
=================
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 3534 entries, 0 to 3533
Data columns (total 3 columns):
 #   Column     Non-Null Count  Dtype 
---  ------     --------------  ----- 
 0   textID     3534 non-null   object
 1   text       3534 non-null   object
 2   sentiment  3534 non-null   object
dtypes: object(3)
memory usage: 83.0+ KB
None<jupyter_text>We have one null Value in the train , as the test field for value is NAN we will just remove it. And no nulls in test set.<jupyter_code>train.dropna(inplace=True)
train.describe().T<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.14858096828046743, 'fraction_numerical': 0.0659432387312187, 'mean_word_length': 2.689230769230769, 'pattern_counts': {'":': 0, '<': 10, '<?xml version=': 0, '>': 10, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '527315', 'license': 'MIT', 'n_tokens_mistral': 445, 'n_tokens_neox': 384, 'n_words': 127, 'path': 'NLP/tweet-sentiment-extraction/tweet-sentiment-extraction.ipynb', 'repo_name': 'KickItLikeShika/cracking-kaggle-competitions', 'size': '1198'}