## CC_FILTER
this is ja cc filter for reference from ja wiki vs random ja common crawl, and build with following procedure.
1. get ja wiki dump file, and extract the all url inside, get about 4M urls
2. crawl 300K of 4M webpages from the urls
3. get pure text and remove content len less than 1k,
4. use langdetect to tell the lang of the pages, we finally get total 101K ja pages
5. random sample from commoncrawl 202303 and use langdetect to find 101k ja pages
6. tokenize all text with "rinna/japanese-roberta-base"
7. feed tokens to fasttext to get model.bin
{}
region:us
TAGS
#region-us 

## CC_FILTER
this is ja cc filter for reference from ja wiki vs random ja common crawl, and build with following procedure.
1. get ja wiki dump file, and extract the all url inside, get about 4M urls
2. crawl 300K of 4M webpages from the urls
3. get pure text and remove content len less than 1k,
4. use langdetect to tell the lang of the pages, we finally get total 101K ja pages
5. random sample from commoncrawl 202303 and use langdetect to find 101k ja pages
6. tokenize all text with "rinna/japanese-roberta-base"
7. feed tokens to fasttext to get URL
## CC_FILTER
this is ja cc filter for reference from ja wiki vs random ja common crawl, and build with following procedure.
1. get ja wiki dump file, and extract the all url inside, get about 4M urls
2. crawl 300K of 4M webpages from the urls
3. get pure text and remove content len less than 1k,
4. use langdetect to tell the lang of the pages, we finally get total 101K ja pages
5. random sample from commoncrawl 202303 and use langdetect to find 101k ja pages
6. tokenize all text with "rinna/japanese-roberta-base"
7. feed tokens to fasttext to get URL
TAGS
#region-us 
 ## CC_FILTER
this is ja cc filter for reference from ja wiki vs random ja common crawl, and build with following procedure.
1. get ja wiki dump file, and extract the all url inside, get about 4M urls
2. crawl 300K of 4M webpages from the urls
3. get pure text and remove content len less than 1k,
4. use langdetect to tell the lang of the pages, we finally get total 101K ja pages
5. random sample from commoncrawl 202303 and use langdetect to find 101k ja pages
6. tokenize all text with "rinna/japanese-roberta-base"
7. feed tokens to fasttext to get URL