This function takes as an argument a json file (could contain anything in json format, since I scrap hundreds of random pages) and returns a list of dictionaries where a URL is mapped to its corresponding headers, based on the extraction of headers using beautifulsoup and a regex pattern.I'm looking for suggestions regarding performance readability and clarity.Following  my 1st iteration I improved my code and here is the result: import jsonfrom tqdm import tqdmimport refrom bs4 import BeautifulSoupimport csvimport stringLoad HTML body, and fetch headersdef get_headers_from_json(local_path):        The function takes a json file with html_body and returns a list of headers.    It parses the titles, based on tags starting with 'h' + num.        data = json.loads(open(local_path).read())    pattern = re.compile(r^h[0-9]$)    headers_urls = []    printable = set(string.printable)    for dict in tqdm(data):        headers = []        for val in dict.values():            soup = BeautifulSoup(val, 'html.parser')            url = dict.values()[0]        for element in soup.find_all(pattern):            element = element.get_text().strip().encode('utf-8')            element = filter(lambda word: word in printable, element)              headers.append(element)        cleaned_data = {url: url, headers: headers}        headers_urls.append(cleaned_data)    return headers_urlsExample of json input:[[<body class=\ catalog-category-view categorypath-sale-html category-sale\>\n<script type=\text/javascript\>\n//<![CDATA[\nif (typeof(Varien.searchForm) !== 'undefined') {\n    Varien.searchForm.prototype._selectAutocompleteItem = function(element) {\n        var link = element.down();\n        if (link && link.tagName == 'A') {\n            setLocation(link.href);\n        } else {\n            if (element.title){\n                this.field.value = element.title;\n            }\n            this.form.submit();\n        }\n    };\n    Varien.searchForm.prototype.initAutocomplete = function(url, destinationElement) {\n        new Ajax.Autocompleter(\n            this.field,\n            destinationElement,\n            url,\n            {\n                paramName: this.field.name,\n                method: 'get',\n                minChars: 2,\n                frequency: .3,\n                updateElement: this._selectAutocompleteItem.bind(this),\n                onShow : function(element, update) {\n                    if(!update.style.position || update.style.position=='absolute') {\n                        update.style.position = 'absolute';\n                        Position.clone(element, update, {\n                            setHeight: false,\n                            offsetTop: element.offsetHeight\n                        });\n                    }\n                    Effect.Appear(update,{duration:0});\n                }\n\n            }\n        );\n    };\n    Autocompleter.Base.prototype.markPrevious = function() {\n        if (this.index > 0) {\n            this.index--;\n        } else {\n            this.index = this.entryCount - 1;\n        }\n        var entry = this.getEntry(this.index);\n        if (entry.select('a').length === 0) {\n            this.markPrevious(); // Ignore items that don't have link\n        }\n    };\n    Autocompleter.Base.prototype.markNext = function() {\n        if (this.index < this.entryCount - 1) {\n   
Performance and Readability Improvements for HTML Parser with BeautifulSoup
python;html;parsing;web scraping;beautifulsoup
There are multiple things you can do to speed things up:faster JSON parsing - try out ujson or simplejson; or even simplejson on PyPyuse lxml instead of html.parser (requires lxml to be installed)do not parse the complete HTML, parse only what you need with SoupStrainer:pattern = re.compile(r^h[0-9]$)parse_only = SoupStrainer(pattern)soup = BeautifulSoup(data, lxml, parse_only=parse_only)You can then do simple soup.find_all() (or via a shortcut - soup()) since you'll only have header elements in your HTML soup.Here is a demonstration of what SoupStrainer would do:In [1]: import reIn [2]: from bs4 import BeautifulSoup, SoupStrainerIn [3]: data =    ...: <body>   ...:     <h1>Some Paragraph 1</h1>   ...:     <div>   ...:         <h2>Some Paragraph 2</h2>   ...:     </div>   ...:        ...:     <span>   ...:         <div>   ...:             <h3>Some Paragraph 3</h3>   ...:         </div>   ...:         <h4>Some Paragraph 4</h4>   ...:     </span>   ...:     <h5>Some Paragraph 2</h5>   ...: </body>   ...: In [4]: pattern = re.compile(r^h[0-9]$)In [5]: parse_only = SoupStrainer(pattern)In [6]: soup = BeautifulSoup(data, lxml, parse_only=parse_only)In [7]: print(soup.prettify())<h1> Some Paragraph 1</h1><h2> Some Paragraph 2</h2><h3> Some Paragraph 3</h3><h4> Some Paragraph 4</h4><h5> Some Paragraph 2</h5>