<jupyter_start><jupyter_text>MIT LicenseCopyright (c) 2021  <NAME>,                    <NAME>,                    <NAME>,                    <NAME>,                    <NAME>,                    <NAME>,                    <NAME>Permission is hereby granted, free of charge, to any person obtaining a copyof this software and associated documentation files (the "Software"), to dealin the Software without restriction, including without limitation the rightsto use, copy, modify, merge, publish, distribute, sublicense, and/or sellcopies of the Software, and to permit persons to whom the Software isfurnished to do so, subject to the following conditions:The above copyright notice and this permission notice shall be included in allcopies or substantial portions of the Software.THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS ORIMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THEAUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHERLIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THESOFTWARE.  Sesgos de tweets con/sin tildes<jupyter_code>import pandas as pd
import numpy as np
if 'google.colab' in str(get_ipython()):
  from google.colab import drive
  drive.mount('/content/drive')
  tweets = pd.read_csv('/content/drive/Shareddrives/ETICA/TweetMod_csv/tweet_mods_accents.csv')
else:
  tweets = pd.read_csv('../data/tweet_mods_accents.csv')<jupyter_output><empty_output><jupyter_text>Análisis de todos los datos<jupyter_code>tweets.head()
print('Con tildes toxic: ', np.mean(tweets['toxic_original']))
print('Con tildes very toxic: ', np.mean(tweets['very_toxic_original']))
print('Sin tildes toxic: ', np.mean(tweets['toxic_noaccent']))
print('Sin tildes very toxic: ', np.mean(tweets['very_toxic_noaccent']))<jupyter_output>Con tildes toxic:  0.29571437127105915
Con tildes very toxic:  0.08084586595985278
Sin tildes toxic:  0.300242413898577
Sin tildes very toxic:  0.08014870762158596<jupyter_text>> Parece que en cuanto al valor "toxic", son un poco más tóxicos tweets sin tildes, pero vemos en el caso de "very toxic" es muy similar.    > Aun así, parece que las medias están bastante igualadas.<jupyter_code># Aquí calculamos las veces que un tweet es más tóxico con tildes que sin tildes y viceversa
print('Toxic con tildes', np.count_nonzero(tweets.most_toxic_grammar == 'con tildes'))
print('Toxic sin tildes', np.count_nonzero(tweets.most_toxic_grammar == 'sin tildes'))
print('Very toxic con tildes', np.count_nonzero(tweets.most_verytoxic_grammar == 'con tildes'))
print('Very toxic sin tildes', np.count_nonzero(tweets.most_verytoxic_grammar == 'sin tildes'))<jupyter_output>Toxic con tildes 4825
Toxic sin tildes 7223
Very toxic con tildes 7712
Very toxic sin tildes 4336<jupyter_text>> En general, la variable "toxic" aumenta al no haber tildes pero la variable "very_toxic" es mayor con tildes  Análisis de datos acotadosAhora vamos a analizar tweets donde la ausencia de tildes cambia de manera significante la toxicidad. Solo miraremos tweets donde la diferencia sea superior a 10.<jupyter_code>column_names = []
for i in tweets:
  column_names.append(i)
tweets_filtered = pd.DataFrame(columns = column_names)
for idx, row in tweets.iterrows():
  if abs(row.toxic_original - row.toxic_noaccent) > 0.10:
    tweets_filtered = tweets_filtered.append(row)
tweets_filtered
print('Con tildes toxic: ', np.mean(tweets_filtered['toxic_original']))
print('Con tildes very toxic: ', np.mean(tweets_filtered['very_toxic_original']))
print('Sin tildes toxic: ', np.mean(tweets_filtered['toxic_noaccent']))
print('Sin tildes very toxic: ', np.mean(tweets_filtered['very_toxic_noaccent']))<jupyter_output>Con tildes toxic:  0.4433678603518351
Con tildes very toxic:  0.0031834938606548945
Sin tildes toxic:  0.49841069664810655
Sin tildes very toxic:  0.00299137524673699<jupyter_text>> Aquí vemos que en "toxic" la toxicidad es más alta en tweets sin tildes pero en "very_toxic" es más similar, siendo algo más alta en los tweets con tilde<jupyter_code>print('Toxic con tildes', np.count_nonzero(tweets_filtered.most_toxic_grammar == 'con tildes'))
print('Toxic sin tildes', np.count_nonzero(tweets_filtered.most_toxic_grammar == 'sin tildes'))
print('Very toxic con tildes', np.count_nonzero(tweets_filtered.most_verytoxic_grammar == 'con tildes'))
print('Very toxic sin tildes', np.count_nonzero(tweets_filtered.most_verytoxic_grammar == 'sin tildes'))<jupyter_output>Toxic con tildes 325
Toxic sin tildes 498
Very toxic con tildes 368
Very toxic sin tildes 455
{'fraction_non_alphanumeric': 0.08759278897136798, 'fraction_numerical': 0.03817603393425239, 'mean_word_length': 5.46027397260274, 'pattern_counts': {'":': 0, '<': 24, '<?xml version=': 0, '>': 29, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '465320', 'license': 'MIT', 'n_tokens_mistral': 1711, 'n_tokens_neox': 1522, 'n_words': 491, 'path': 'analysis_notebooks/Sesgos_tildes.ipynb', 'repo_name': 'AlbertoGarciaDomenech/SesgosTocicidadTweets', 'size': '4831'}