Spaces:

huggingface
/

text-data-filtering

Running

App Files Files Community

text-data-filtering

6 contributors

History: 51 commits

HugoLaurencon's picture

distributions for the filters on words and discarded words by filter

da13b29 almost 3 years ago

.gitattributes

1.39 kB

chinese visu almost 3 years ago
.gitignore

25 Bytes

new tool to analyse our own doc almost 3 years ago
LICENSE

11.4 kB

Create LICENSE almost 3 years ago
README.md

909 Bytes

initial commit almost 3 years ago
app.py

33.2 kB

distributions for the filters on words and discarded words by filter almost 3 years ago
en.arpa.bin

4.4 GB
LFS

test almost 3 years ago
en.sp.model

1.39 MB
LFS

test almost 3 years ago
en_examples_with_stats.json

241 MB
LFS

distributions for the filters on words and discarded words by filter almost 3 years ago
explanation_filtering_pipeline.pdf

216 kB

rename badwords to flagged words + new flagged words list of 68 words almost 3 years ago
filtering.py

30.6 kB

rename badwords to flagged words + new flagged words list of 68 words almost 3 years ago
flagged_words.py

51.8 kB

rename badwords to flagged words + new flagged words list of 68 words almost 3 years ago
languages_id.py

5.61 kB

rename badwords to flagged words + new flagged words list of 68 words almost 3 years ago
lid.176.bin

131 MB
LFS

test almost 3 years ago
normalization.py

941 Bytes

test almost 3 years ago
parameters_filtering.py

31.7 kB

rename badwords to flagged words + new flagged words list of 68 words almost 3 years ago
requirements.txt

76 Bytes

fix requirements almost 3 years ago
stopwords.py

99.2 kB

test almost 3 years ago