omkarenator commited on
Commit
aba6489
1 Parent(s): 2f4db88

add example citation, fix formatting

Browse files
Files changed (2) hide show
  1. common.py +4 -2
  2. curated.py +0 -0
common.py CHANGED
@@ -299,7 +299,7 @@ global_div = Div(
299
  style="margin-bottom: 5px",
300
  ),
301
  Li("Normailzation Form C Discussion", style="margin-bottom: 5px"),
302
- ),
303
  id="section1",
304
  ),
305
  Section(
@@ -376,7 +376,9 @@ global_div = Div(
376
  H3("Finding Connected Components using MapReduce"),
377
  Img(src="images/findcc.svg", style="max-width: 100%;"),
378
  P(
379
- "The purpose of this step is to create a set of clusters of matching pairs. For example, a list of pairs (A, B), (B, C), (D, E) is merged into a list of components (A, B, C) and (D, E). Using a third-party library like NetworkX to find connected components would require all pairs to fit into the memory of a single machine, which is not feasible. Instead, we implemented a distributed connected component finder [4] using the Dask framework, which can scale across multiple machines. The algorithm works by mapping edges by both the source and destination of pairs and reducing only edges where the source is greater than the destination. It performs successive iterations of this MapReduce computation until convergence, meaning the number of new edges produced becomes zero. In the end, every document in a cluster points to the smallest document within the cluster. Later, we compile a list of duplicate documents that need deletion and gather statistics about each component."
 
 
380
  ),
381
  P(
382
  "We needed to partition the duplicate pairs generated in the third stage into three groups to reduce memory pressure on the final stage. We observed that the second stage itself generates partial components which have some overlap. These overlapping clusters cause some documents to appear in the delete set multiple times. However, our deletion code handled this overlap."
 
299
  style="margin-bottom: 5px",
300
  ),
301
  Li("Normailzation Form C Discussion", style="margin-bottom: 5px"),
302
+ ),
303
  id="section1",
304
  ),
305
  Section(
 
376
  H3("Finding Connected Components using MapReduce"),
377
  Img(src="images/findcc.svg", style="max-width: 100%;"),
378
  P(
379
+ "The purpose of this step is to create a set of clusters of matching pairs. For example, a list of pairs (A, B), (B, C), (D, E) is merged into a list of components (A, B, C) and (D, E). Using a third-party library like NetworkX to find connected components would require all pairs to fit into the memory of a single machine, which is not feasible. Instead, we implemented a distributed connected component finder",
380
+ D_cite(bibtex_key="6785473"),
381
+ "using the Dask framework, which can scale across multiple machines. The algorithm works by mapping edges by both the source and destination of pairs and reducing only edges where the source is greater than the destination. It performs successive iterations of this MapReduce computation until convergence, meaning the number of new edges produced becomes zero. In the end, every document in a cluster points to the smallest document within the cluster. Later, we compile a list of duplicate documents that need deletion and gather statistics about each component.",
382
  ),
383
  P(
384
  "We needed to partition the duplicate pairs generated in the third stage into three groups to reduce memory pressure on the final stage. We observed that the second stage itself generates partial components which have some overlap. These overlapping clusters cause some documents to appear in the delete set multiple times. However, our deletion code handled this overlap."
curated.py CHANGED
The diff for this file is too large to render. See raw diff