Hugging Face
Models
Datasets
Spaces
Posts
Docs
Solutions
Pricing
Log In
Sign Up
Spaces:
huggingface
/
text-data-filtering
like
33
Running
App
Files
Files
Community
4
4809033
text-data-filtering
6 contributors
History:
55 commits
HugoLaurencon
new filter on word repetition ratio
4809033
over 2 years ago
.gitattributes
1.39 kB
chinese visu
almost 3 years ago
.gitignore
25 Bytes
new tool to analyse our own doc
over 2 years ago
LICENSE
11.4 kB
Create LICENSE
almost 3 years ago
README.md
909 Bytes
initial commit
almost 3 years ago
app.py
40.6 kB
new filter on word repetition ratio
over 2 years ago
en.arpa.bin
4.4 GB
LFS
test
over 2 years ago
en.sp.model
1.39 MB
LFS
test
over 2 years ago
en_examples_with_stats.json
241 MB
LFS
new filter on word repetition ratio
over 2 years ago
explanation_filtering_pipeline.pdf
217 kB
new filter on word repetition ratio
over 2 years ago
filtering.py
30.6 kB
rename badwords to flagged words + new flagged words list of 68 words
over 2 years ago
flagged_words.py
51.8 kB
rename badwords to flagged words + new flagged words list of 68 words
over 2 years ago
languages_id.py
5.61 kB
rename badwords to flagged words + new flagged words list of 68 words
over 2 years ago
lid.176.bin
131 MB
LFS
test
over 2 years ago
normalization.py
941 Bytes
test
over 2 years ago
parameters_filtering.py
31.7 kB
rename badwords to flagged words + new flagged words list of 68 words
over 2 years ago
requirements.txt
76 Bytes
fix requirements
over 2 years ago
stopwords.py
99.2 kB
test
over 2 years ago
zh.arpa.bin
3.39 GB
LFS
visualization: choose between several languages
over 2 years ago
zh.sp.model
1.37 MB
LFS
visualization: choose between several languages
over 2 years ago
zh_examples_with_stats.json
63.6 MB
LFS
new filter on word repetition ratio
over 2 years ago
