Update web.py

web.py

@@ -310,8 +310,8 @@ def web_data():
             ),
         #DV2("data/sample_wet.json", "data/sample_warc.json", 3),
         
-        H3("1.2 Language Identification"),
-        P("""
+        
+        P(B("Language Identification: "), """
         After text extraction, the non-English texts are then filtered out by fastText language identifier with a threshold of 0.65.
         This step removes over 60% of the whole data.
         """),
@@ -347,13 +347,13 @@ def web_data():
             """, 
         ),
         
-        H3("1.3 URL Filtering"),
-        P("""
+        
+        P(B("URL Filtering: "), """
         The following section details the decisions behind utilizing the UT1 blocklist. We chose to use the UT1 blocklist as a simple method for filtering
         out potentially harmful content such as adult content. We also excluded URLs that contained the digital version of the curated curated data (e.g. wikipedia.org) to avoid duplication.
         """),
-        H3("1.3.1 URL Blocklist"),
-        P("""
+        
+        P(B("URL Blocklist: "), """
         Following RefinedWeb [3], we manually inspected the UT1 blocklist to reduce false positives like news 
         articles, sex education, technical blogs, etc. Specifically, we randomly took 903M URLs and matched them with
         4.6M domain names in the UT1 blocklist. Of note, 24 URLs were detected with more than 4k matches and are shown below.
@@ -407,8 +407,7 @@ def web_data():
             """, 
         ),
         
-        H3("1.3.2 Excluded High Quality Sources"),
-        P("""
+        P(B("Excluded High Quality Sources: "), """
         To avoid duplication with our high-quality curated datasets, we exclude the following domains from our dataset.
         """),
         
@@ -449,8 +448,7 @@ def web_data():
         Before filtering low-quality documents, we perform the line-level removal to remove low-quality lines. 
         This ensured that computing quality signals would align with the final kept texts.
         """),
-        H3("Terminal Punctuation"),
-        P("""
+        P(B("Terminal Punctuation: "), """
         The terminal punctuation has been used in C4 [5] and Dolma [6] to remove lines that do not end with a terminal
         punctuation mark (i.e., “.”, “?”, “!”, or “"”). However, we found it could be too aggressive to remove these
         lines, especially when the text extraction tool “trafilatura”. 
@@ -481,8 +479,7 @@ def web_data():
         ),
         
         
-        H3('2.1 Word "Javascript"'),
-        P("""
+        P(B('"Word "Javascript"'), """
         In C4 [5], the authors remove any line with the word "Javascript" since they found that many of the scraped
         pages contained warnings stating that Javascript should be enabled. However, this filtering strategy is too
         strict, which will filter out many lines that are really talking about “Javascript”. 
@@ -509,8 +506,7 @@ def web_data():
             margin-bottom: 15px
             """, 
         ),
-        H3("2.2 Other Rules from RefinedWeb"),
-        P("""
+        P(B("Other Rules from RefinedWeb: "), """
         We also adopt rules from RefinedWeb [3] to remove lines if they satisfy any of the following criteria:
         """),
         Ul(
@@ -536,8 +532,7 @@ def web_data():
             margin-bottom: 15px
             """, 
         ),
-        H3("2.3 Toxic Lines"),
-        P("""
+        P(B("Toxic Lines: "), """
         When doing manual inspection on the data, we found that there are some adult ads in the beginning or end of the 
         document (with a sample shown below), which are hard to remove via document-level filtering strategies. Inspired 
         by this, we develop line-level detoxification using a bad word list from LDNOOBW (+ rule: word length < 10 + the 
@@ -589,13 +584,11 @@ def web_data():
         In our pipeline, we referenced earlier implementations that were publicly available such as Dolma [6], DataTrove [4], 
         and RedPajama V2 [7], and selected the most suitable method based on manual inspections.
         """),
-        H3("3.1 Repetition-based Heuristics"),
-        P("""
+        P(B("Repetition-based Heuristics: "), """
         Many documents contain repeated sequences, potentially due to crawling errors or low-quality sources. In line with previous 
         work ([2], [3], [6]), we choose to remove any document with excessive line, paragraph, or n-gram repetitions.
         """),
-        H3("3.1.1 Fraction of (Characters in) Repeated Lines"),
-        P("""
+        P(B("Fraction of Characters in Repeated Lines: "), """
         Following Gopher [2], we remove documents containing mupltiple, short duplicate passages, as well as those with few, 
         but longer duplicate passages. To achieve this goal, we calculate over the document both the fraction of passages 
         that are duplicates, and the fraction of characters contained within those duplicated passages.
@@ -675,20 +668,17 @@ def web_data():
         After evaluating the implementations of Dolma and DataTrove (note: RedPajama V2 does not implement these two quality 
         signals), we have made the following decisions:
         """),
-        H3("Passage Separation"),
-        P("""
+        P(B("Passage Separation: "), """
         Our manual review of the data revealed that documents extracted using trafilatura do not feature more than one newline 
         symbol separating passages. Testing the splitting pattern "\\n(2,)" on 10,000 sample documents resulted in no more than 
         one split. Consequently, we decided to disregard the distinction between lines and paragraphs in our implementation, 
         opting instead to use a single newline symbol to segment the text into passages.
         """),
-        H3("First Occurrence"),
-        P("""
+        P(B("First Occurrence: "), """
         In line with DataTrove's implementation, we chose to exclude the first occurrence. This more conservative strategy 
         helps retain a larger number of documents.
         """),
-        H3("Character Count"),
-        P("""
+        P(B("Character Count: "), """
         We adjusted the method in Dolma for counting characters within lines by excluding whitespace. This modification 
         ensures consistency with the overall document character count calculation.
         """),
@@ -738,8 +728,7 @@ def web_data():
             margin-bottom: 15px
             """, 
         ),
-        H3("3.1.2 Fraction of Characters in the Most Common N-grams (n=2,3,4)"),
-        P("""
+        P(B("Fraction of Characters in the Most Common N-grams (n=2,3,4): "), """
         Following Gopher [2], we remove documents with a high portion of n-grams. For each n ∈ (2, 3, 4), we calculate the 
         fraction of characters contained within the most frequently-occurring n-gram.
         """),
@@ -902,8 +891,7 @@ def web_data():
             margin-bottom: 15px
             """,
         ),
-        H3("3.1.3 Fraction of Characters in Duplicated N-grams (n=5,...,10)"),
-        P("""
+        P(B("Fraction of Characters in Duplicated N-grams (n=5,...,10): "), """
         Following Gopher [2], we remove documents with a high portion of n-grams. For each n ∈ (5, ..., 10), we calculate the 
         fraction of characters contained within all duplicate n-grams, taking care not to count characters that occur in 
         overlapping n-grams more than once.
@@ -1135,8 +1123,7 @@ def web_data():
             margin-bottom: 15px
             """, 
         ),
-        H3("3.2 Line-wise Heuristics"),
-        P("""
+        P(B("Line-wise Heuristics: "), """
         Some line-wise information could also be helpful to distinguish low-quality and high-quality documents. Following 
         RefinedWeb [3], we remove the document if the corrected lines represent more than 5% of words. In line with previous 
         works ([2], [3], [6]), we remove the documents if more than 30% of the lines end with an ellipsis or more than 
@@ -1243,8 +1230,9 @@ def web_data():
             """, 
         ),
         
-        H3("3.3 Statistics-based Heuristics"),
-        P("We summarize other statistics-based rules originated from Gopher [7] in this section. The statistics can be used include:"),
+        P(B("Statistics-based Heuristics: "), """
+        We summarize other statistics-based rules originated from Gopher [7] in this section. The statistics can be used include:
+        """),
         Ul(
             Li("the word count in the document", style = "margin-bottom: 5px"),
             Li("the mean word length", style = "margin-bottom: 5px"),
@@ -1338,8 +1326,7 @@ def web_data():
         We decided to use simple `len(text.split())` to compute the word count.
         """),
         
-        H3("Mean Word Length"),
-        P("""
+        P(B("Mean Word Length: "), """
         There is minimal variation among existing pipeline implementations. We simply compute the mean word length as follows:
         """),
         D_code("""
@@ -1355,8 +1342,7 @@ def web_data():
                 from statistics import median
                 median_word_length = median(len(word) for word in words)
             """, block="block", language="python"),
-        H3("Number of Sentences"),
-        P("""
+        P(B("Number of Sentences: "), """
         The only publicly available implementation of this quality signal is from RedPajama V2, which uses regular expressions 
         to split text into sentences.
         """),
@@ -1410,8 +1396,7 @@ def web_data():
             """, 
         ),
         
-        H3("Symbol to Word Ratio"),
-        P("""
+        P(B("Symbol to Word Ratio: "), """
         Following RedPajama-V2 and DataTrove, we use the symbols of ("#", "...", "…").
         We calculate the ratio as the number of symbols divided by the total number of words.
         """),
@@ -1584,8 +1569,7 @@ def web_data():
         RedPajama-V2 employs regular expressions for this purpose. We opt to use regular expressions since `char.isalpha()` 
         can also match words in other languages as long as they are not punctuations.
         """),
-        H5("Number of Stop Words"),
-        P("""
+        P(B("Number of Stop Words: "), """
         The implementations across existing pipelines are largely identical. We adopt them and apply them to our pipeline.
         """),
         D_code("""
@@ -1614,8 +1598,7 @@ def web_data():
             margin-bottom: 15px
             """, 
         ),
-        H3("3.4 Others"),
-        P("""
+        P(B("Additional Filters: "), """
         Following C4, we remove any page where the phrase “lorem ipsum” appeared since some pages had placeholder “lorem ipsum” 
         text.
         """),
@@ -1633,15 +1616,4 @@ def web_data():
             margin-bottom: 15px
             """, 
         ),
-        H2("4. Deduplication"),
-        P("""
-        After careful filtering, although data quality has improved, a large fraction of the content is repeated across documents. This may be due to the crawler indirectly hitting the same page multiple times, to boilerplate content being repeated (e.g., licences), or even to plagiarism. These duplicates can strongly impact models, favoring memorization instead of generalization.
-        """),  # Add detailed content and images as needed
-        P("We perform two-level deduplication: local exact deduplication and global fuzzy deduplication"),
-        P(B("Local Exact Deduplication")),
-        P("To reduce the expensive cost of global deduplication, we apply a local exact deduplication before it. Specifically, each dump is split into 70 splits. A bloom filter is applied within each split."),
-        P(B("Global Fuzzy Deduplication")),
-        P("NEED TO UPDATE"),
-        H2("5. PII Removal"),
-        P("..."),  # Add detailed content and images as needed
     )