<commit_before># Visualizing an XML sitemap using Python

Source code repository for the ["How to visualize an XML sitemap using Python"](https://www.ayima.com/guides/how-to-visualize-an-xml-sitemap-using-python.html) Ayima blog post.

![](IPython-notebook/static/sitemap_graph_2_layer.png)

## Extraction, Categorization and Visualization

The code has been split into three scripts that can be [downloaded here](https://github.com/Ayima/sitemap-visualization-tool/archive/master.zip). Running these requires Python and external library dependencies as [detailed in the section below](#dependencies).

### Reproducing our Results

Before plugging in a new sitemap, you may wish to test the script by reproducing a result from our blog post. This can be done by opening a terminal session, navigating to the folder containing the three `.py` scripts and running the following commands:

```
python extract_urls.py   
python categorize_urls.py   
python visualize_urls.py   
```

### Plugging in an XML Sitemap URL

A custom sitemap URL can be ingested by passing arguments to the extraction script. The commands to execute would look something like this (where we are categorizing and visualizing with a granularity depth of 3):

```
python extract_urls.py --url "site.com/sitemap-index.xml"   
python categorize_urls.py --depth 3   
python visualize_urls.py --depth 3 --title "My Sitemap" --size "20"   
```

If your XML sitemap file contains the page URLs (instead of linking to other sitemaps) then make sure to pass the `--not_index` argument:

```
python extract_urls.py --url "site.com/sitemap.xml" --not_index
```

There is also built in support for compressed XML files:

```
python extract_urls.py --url "site.com/sitemap.xml" --not_index --gzip
```

The `visualize_urls.py` script also has a `--limit` argument that can be passed. This can be used to limit the number of edges spawning from a node, and is useful for creating deep sitemap visualizations that don't grow out of control. For example:
<commit_msg>Pick better depth for example<commit_after># Visualizing an XML sitemap using Python

Source code repository for the ["How to visualize an XML sitemap using Python"](https://www.ayima.com/guides/how-to-visualize-an-xml-sitemap-using-python.html) Ayima blog post.

![](IPython-notebook/static/sitemap_graph_2_layer.png)

## Extraction, Categorization and Visualization

The code has been split into three scripts that can be [downloaded here](https://github.com/Ayima/sitemap-visualization-tool/archive/master.zip). Running these requires Python and external library dependencies as [detailed in the section below](#dependencies).

### Reproducing our Results

Before plugging in a new sitemap, you may wish to test the script by reproducing a result from our blog post. This can be done by opening a terminal session, navigating to the folder containing the three `.py` scripts and running the following commands:

```
python extract_urls.py   
python categorize_urls.py   
python visualize_urls.py --depth 1   
```

### Plugging in an XML Sitemap URL

A custom sitemap URL can be ingested by passing arguments to the extraction script. The commands to execute would look something like this (where we are categorizing and visualizing with a granularity depth of 3):

```
python extract_urls.py --url "site.com/sitemap-index.xml"   
python categorize_urls.py --depth 3   
python visualize_urls.py --depth 3 --title "My Sitemap" --size "20"   
```

If your XML sitemap file contains the page URLs (instead of linking to other sitemaps) then make sure to pass the `--not_index` argument:

```
python extract_urls.py --url "site.com/sitemap.xml" --not_index
```

There is also built in support for compressed XML files:

```
python extract_urls.py --url "site.com/sitemap.xml" --not_index --gzip
```

The `visualize_urls.py` script also has a `--limit` argument that can be passed. This can be used to limit the number of edges spawning from a node, and is useful for creating deep sitemap visualizations that don't grow out of control. For example:

{'fraction_non_alphanumeric': 0.08357915437561456, 'fraction_numerical': 0.0031956735496558504, 'mean_word_length': 4.361001317523057, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 4, 'lorem ipsum': 0, 'www.': 2, 'xml': 8}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6657936', 'n_tokens_mistral': 1206, 'n_tokens_neox': 1155, 'n_words': 510}