<jupyter_start><jupyter_text>Converting HTML to TextTags: data, python, nlpdate: 2020-08-05T08:00:00+10:00feature_image: /images/jupyter-blog.png   How can we convert HTML into text for processing?Whitespace in HTML [is complicated](https://developer.mozilla.org/en-US/docs/Web/API/Document_Object_Model/Whitespace).<jupyter_code>def html2md(html):
    parser = HTML2Text()
    parser.ignore_images = True
    parser.ignore_anchors = True
    parser.body_width = 0
    md = parser.handle(html)
    return md
def html2plain(html):
    # HTML to Markdown
    md = html2md(html)
    # Normalise custom lists
    md = re.sub(r'(^|\n) ? ? ?\\?[•·–-—-*]( \w)', r'\1  *\2', md)
    # Convert back into HTML
    html_simple = mistletoe.markdown(md)
    # Convert to plain text
    soup = BeautifulSoup(html_simple)
    text = soup.getText()
    # Strip off table formatting
    text = re.sub(r'(^|\n)\|\s*', r'\1', text)
    # Strip off extra emphasis
    text = re.sub(r'\*\*', '', text)
    # Remove trailing whitespace and leading newlines
    text = re.sub(r' *$', '', text)
    text = re.sub(r'\n\n+', r'\n\n', text)
    text = re.sub(r'^\n+', '', text)
    return text<jupyter_output><empty_output><jupyter_text>Example with linebreaks instead of paragraphs<jupyter_code>p = paths[6]
data = read_jsonl(p)
html = data[0]['description']<jupyter_output><empty_output><jupyter_text>The plain HTML<jupyter_code>print(html)<jupyter_output><strong><u>The Client</u></strong><br/><br/>Our client is a secondary education institution in the Eastern Suburbs of Melbourne. They offer rewarding and diverse programs for Australian and overseas students as well as workforce development for Australia's corporate, government, and commercial organisations. They operate seven campuses in Victoria, deliver online learning, and have over 1000 passionate and dedicated team members from diverse backgrounds, making them a high flexible and student-focussed organization.<br/><br/><u><strong>The Job</strong></u><br/>This is a job for a talented Information Security Administrator. You'll be providing crucial services in responding to security exposures, monitoring risk, and reporting on internal information security practices across the entire organization.<br/><br/>You'll be chiefly responsible for resolving security issues and proactively assessing risk and vulnerabilities by conducting gap analyses, user education, and providing reports an[...]<jupyter_text>How it looks in a browser<jupyter_code>HTML(html)<jupyter_output><empty_output><jupyter_text>Beautiful soup runs the sentences together because it doesn't process the `` tags.<jupyter_code>print(BeautifulSoup(html).getText())<jupyter_output>The ClientOur client is a secondary education institution in the Eastern Suburbs of Melbourne. They offer rewarding and diverse programs for Australian and overseas students as well as workforce development for Australia's corporate, government, and commercial organisations. They operate seven campuses in Victoria, deliver online learning, and have over 1000 passionate and dedicated team members from diverse backgrounds, making them a high flexible and student-focussed organization.The JobThis is a job for a talented Information Security Administrator. You'll be providing crucial services in responding to security exposures, monitoring risk, and reporting on internal information security practices across the entire organization.You'll be chiefly responsible for resolving security issues and proactively assessing risk and vulnerabilities by conducting gap analyses, user education, and providing reports and assessments.You will be a key and active member of the Security Team, providing s[...]<jupyter_text>It's better if we replace them with spaces, but we lose separations between headers and content.<jupyter_code>print(BeautifulSoup(html).getText(' '))<jupyter_output>The Client Our client is a secondary education institution in the Eastern Suburbs of Melbourne. They offer rewarding and diverse programs for Australian and overseas students as well as workforce development for Australia's corporate, government, and commercial organisations. They operate seven campuses in Victoria, deliver online learning, and have over 1000 passionate and dedicated team members from diverse backgrounds, making them a high flexible and student-focussed organization. The Job This is a job for a talented Information Security Administrator. You'll be providing crucial services in responding to security exposures, monitoring risk, and reporting on internal information security practices across the entire organization. You'll be chiefly responsible for resolving security issues and proactively assessing risk and vulnerabilities by conducting gap analyses, user education, and providing reports and assessments. You will be a key and active member of the Security Team, provid[...]<jupyter_text>Newlines work for this particular case.<jupyter_code>print(BeautifulSoup(html).getText('\n'))<jupyter_output>The Client
Our client is a secondary education institution in the Eastern Suburbs of Melbourne. They offer rewarding and diverse programs for Australian and overseas students as well as workforce development for Australia's corporate, government, and commercial organisations. They operate seven campuses in Victoria, deliver online learning, and have over 1000 passionate and dedicated team members from diverse backgrounds, making them a high flexible and student-focussed organization.
The Job
This is a job for a talented Information Security Administrator. You'll be providing crucial services in responding to security exposures, monitoring risk, and reporting on internal information security practices across the entire organization.
You'll be chiefly responsible for resolving security issues and proactively assessing risk and vulnerabilities by conducting gap analyses, user education, and providing reports and assessments.
You will be a key and active member of the Security Team, provid[...]<jupyter_text>HTML2Text does an excellent job of converting this into markdown (though notice it's sensitive to spaces around markup in the headings which aren't visible in the HTML.<jupyter_code>md = html2md(html)
print(md)<jupyter_output>**_The Client_**  
  
Our client is a secondary education institution in the Eastern Suburbs of Melbourne. They offer rewarding and diverse programs for Australian and overseas students as well as workforce development for Australia's corporate, government, and commercial organisations. They operate seven campuses in Victoria, deliver online learning, and have over 1000 passionate and dedicated team members from diverse backgrounds, making them a high flexible and student-focussed organization.  
  
 _**The Job**_  
This is a job for a talented Information Security Administrator. You'll be providing crucial services in responding to security exposures, monitoring risk, and reporting on internal information security practices across the entire organization.  
  
You'll be chiefly responsible for resolving security issues and proactively assessing risk and vulnerabilities by conducting gap analyses, user education, and providing reports and assessments.  
  
You will be a key and active [...]<jupyter_text>Round trip it back to HTML.<jupyter_code>html2 = mistletoe.markdown(md)
print(html2)
HTML(html2)
print(BeautifulSoup(html2).get_text(''))
print(html2plain(html))<jupyter_output>The Client
Our client is a secondary education institution in the Eastern Suburbs of Melbourne. They offer rewarding and diverse programs for Australian and overseas students as well as workforce development for Australia's corporate, government, and commercial organisations. They operate seven campuses in Victoria, deliver online learning, and have over 1000 passionate and dedicated team members from diverse backgrounds, making them a high flexible and student-focussed organization.
The Job
This is a job for a talented Information Security Administrator. You'll be providing crucial services in responding to security exposures, monitoring risk, and reporting on internal information security practices across the entire organization.
You'll be chiefly responsible for resolving security issues and proactively assessing risk and vulnerabilities by conducting gap analyses, user education, and providing reports and assessments.
You will be a key and active member of the Security Team, provid[...]<jupyter_text>Example - HTML Tables<jupyter_code>p = paths[16]
data = read_jsonl(p)
[idx for idx, d in enumerate(data) if '<tr>' in d['description']]
html = data[68]['description']
print(html)
HTML(html)
md = html2md(html)
print(md)
html2 = mistletoe.markdown(md)
print(html2)
HTML(html2)
print(BeautifulSoup(html2).get_text(''))
text = html2plain(html)
print(text)<jupyter_output>Location Profile
SCHOOL PROFILE
Greenhills Primary School, established in 1962, is situated in a quiet residential location between the north-eastern Melbourne suburbs of Greensborough and Diamond Creek, within the municipality of Nillumbik. The location of the school, on a well maintained grassed and treed site provides a geographic and social centre for the community. The current school enrolment is 517 students. It’s a great place to be!
The school has undergone a series of major works providing our community with modern, spacious and well resourced teaching facilities, which include classrooms equipped with the latest IT equipment including interactive whiteboards. We also run a iPad Program in Years 5 and 6.
Vision
Greenhills Primary School aims to provide a caring, safe, supportive and nurturing environment, which will foster all students’ educational and behavioural development, to enable them to become effective citizens in our school and, the broader society.
Values
The follow[...]<jupyter_text>Example - Blank bold<jupyter_code>p = paths[21]
data = read_jsonl(p)
[idx for idx, d in enumerate(data) if '<tr>' in d['description']]
html = data[546]['description']
print(html)
HTML(html)
md = html2md(html)
print(md)
html2 = mistletoe.markdown(md)
print(html2)
HTML(html2)
print(BeautifulSoup(html2).get_text(''))
text = html2plain(html)
print(text)<jupyter_output>About the role
Position Title:  Field Organiser
Position Location:  Darwin, NT
Employment Status: Ongoing (subject to probation) / Full Time
Classification and Salary range:

Organiser Level 1 – 2, $74,984 – $97,713 per annum (includes Organiser Expense Allowance paid as salary) + 15.4% superannuation
Darwin Remote Localities Allowance of $5,941 per annum is also payable on a fortnightly basis

Position reports to:  Regional Secretary
Positions reporting to this position are: Nil
CONDITIONS OF EMPLOYMENT

Flexible work practices and access to accrued days off.
Best practice leave provisions including paid primary carer leave (26 weeks) and supporting partner leave (six weeks); and paid family violence leave (20 days pro rata per annum, non-cumulative).
Employer super contributions on paid or unpaid parental leave for a period equal to a maximum of 52 weeks.
A comprehensive Employee Assistance Program.
A strong commitment to training and development.
Health and wellbeing initiatives.
Sa[...]<jupyter_text>Example: Complicated HTML<jupyter_code>p = paths[4]
data = read_jsonl(p)
html = data[0]['description']<jupyter_output><empty_output><jupyter_text>The plain HTML<jupyter_code>print(html)<jupyter_output><div class="job-detail-des">
<iframe allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen="" frameborder="0" height="315" src="https://www.youtube.com/embed/dwoywMmGZEI" width="560"></iframe>
<div style=" "> </div>
<div align="center" style=""><b style="">Organisation Design Specialist</b></div>
<div align="center" style=""><b>TAFE Worker Level 9 – Talent Pool</b></div>
<div style=""><br/>
</div>
<div style="">We are seeking candidates who are interested in joining TAFE NSW’s Organisation Design Specialist Talent Pool.</div>
<div style="">This is a great opportunity for you to be considered for future roles over the next 12 months.</div>
<div style=""><br/>
</div>
<div style=""><br/>
</div>
<div style=""><br/>
</div>
<div style="">• Competitive salary package and access to multiple benefits</div>
<div style="">• Opportunity to join a dynamic team </div>
<div style="">• Build and maintain relationships with key stakeholders</div>
<div style="">[...]<jupyter_text>How it looks in a browser (after removing the video iframe):<jupyter_code>HTML(re.sub('<iframe[^>]*>[^<]*</iframe>', '', html))<jupyter_output><empty_output><jupyter_text>HTML2Text does an excellent job of converting this into markdown.Notice there are some empty bold `**` that are invisible in the rendered HTML.Also note that in CLOSING DATE they've had to insert an additional space between the phrase and the colon to bold it as markdown, which is different to how it reads.<jupyter_code>md = html2md(html)
print(md)
html2 = mistletoe.markdown(md.replace('•', '  *'))
print(html2)
HTML(html2)
text = BeautifulSoup(html2).getText()
print(text)
text = html2plain(html)
print(text)<jupyter_output>Organisation Design Specialist
TAFE Worker Level 9 – Talent Pool
We are seeking candidates who are interested in joining TAFE NSW’s Organisation Design Specialist Talent Pool.
This is a great opportunity for you to be considered for future roles over the next 12 months.

Competitive salary package and access to multiple benefits

Opportunity to join a dynamic team

Build and maintain relationships with key stakeholders

THE OPPORTUNITY
With TAFE NSW you will have the opportunity to grow your professional career in a dynamic and collaborative environment, where you can innovate, create value and proudly play a meaningful role in the once in a generation transformation of Australia’s largest skills and training provider!

THE ROLE
The Design Specialist is responsible for supporting organisation design projects to develop fit for purpose organisation designs aligned to strategy and service delivery models.

Coordinate Organisation Design workshops to ensure organisational structures and j[...]<jupyter_text>Example - Processing Lists<jupyter_code>p = paths[14]
data = read_jsonl(p)
html = data[0]['description']<jupyter_output><empty_output><jupyter_text>The plain HTML<jupyter_code>html<jupyter_output><empty_output><jupyter_text>How it looks in a browser<jupyter_code>HTML(html)<jupyter_output><empty_output><jupyter_text>HTML2Text again does an excellent job of converting this into markdown.Note that there's a mixing of markup and styling in the Key Selection Criteria heading.<jupyter_code>md = html2md(html)
print(md)
html2 = mistletoe.markdown(md)
print(html2)
HTML(html2)
text = BeautifulSoup(html2).getText()
print(text)
text = html2plain(html)
print(text)<jupyter_output>Progressive peak body for Dementia
Full time, fixed term opportunity until June 2020
Attractive salary packaging options available

Dementia Australia is a well-known and respected organisation transforming the experience of people impacted by dementia by elevating their voices and inspiring excellence in support and care free from discrimination.
We are currently seeking a Younger Onset Dementia Support Coordinator to join our Client Services team. This role covers the Metro Melbourne and Gippsland regions and is based with our team in Hawthorn. You will be responsible for the provision of dementia specialist support to assist people living with younger onset dementia, aged under 65 years, to interface with the National Disability Insurance Scheme (NDIS). The Younger Onset Dementia Support Coordinator role plays an active role in assisting clients to implement their NDIS plans, meet identified goals and connect with services and supports that best meet their needs.
To be successful in[...]<jupyter_text>Example - Custom Lists<jupyter_code>p = paths[6]
data = read_jsonl(p)
html = data[2]['description']<jupyter_output><empty_output><jupyter_text>The plain HTML<jupyter_code>md = html2md(html)
print(md)
html2 = mistletoe.markdown(md.replace('·', '  *'))
print(html2)
HTML(html2)
print(html2)
text = BeautifulSoup(html2).getText()
print(text)
text = html2plain(html)
print(text)<jupyter_output>Do you have a fintech background and are hungry for your next move? National BDM role where you can work from home, apply now!
Duties and Responsibilities

Develop sales plans and exceed set KPI's

Generate leads by researching and networking with key stakeholders

Prepare presentations and proposals

Keep abreast of product and industry knowledge

Skills and Experience

5 years' experience selling B2B products or services

Experience selling payment products or services- highly regarded

Experience in the FinTech space- highly regarded

Experience using Salesforce CRM

Experience in hunter sales roles

Excellent written and verbal communication skills

Demonstrated successful negotiation and influencing skills

Fantastic presenting skills

Thank you in advance of your application, we would kindly ask you submit your resume in WORD format
Please note only shortlisted candidates will be contacted<jupyter_text>Example 5 - Actually plain text<jupyter_code>p = paths[10]
data = read_jsonl(p)
html = data[3]['description']<jupyter_output><empty_output><jupyter_text>The plain HTML<jupyter_code>print(html)<jupyter_output>The Opportunity

Do you want to conduct impactful research of strategic importance to Australia?
Incorporate host plant resistance to pests and diseases in the cotton breeding program
Grow your research career with a CSIRO PhD Fellowship

CSIRO Early Research Career (CERC) Postdoctoral Fellowships provide opportunities to scientists and engineers who have completed their doctorate and have less than three years of relevant postdoctoral work experience. These fellowships aim to develop the next generation of future leaders of the innovation system.

In this position you will be the lead researcher for a project titled "Incorporating host plant resistance to pests and diseases in the cotton breeding program. "

The key output from this project is to incorporate host plant resistance traits to verticillium wilt, spider mites and whitefly into advanced lines in the CSIRO cotton breeding program, as well as to improve the methodologies to select for resistance.

As rhe Postdoctoral Fellow i[...]<jupyter_text>How it looks in a browser<jupyter_code>HTML(html)<jupyter_output><empty_output><jupyter_text>HTML2Text again does an excellent job of converting this into markdown.Note that there's a mixing of markup and styling in the Key Selection Criteria heading.<jupyter_code>md = html2md(html)
print(md)
html2 = mistletoe.markdown(md)
print(html2)
HTML(html2)
print(html2)
text = BeautifulSoup(html2).getText()
print(text)
text = html2plain(html)
print(text)<jupyter_output>The Opportunity Do you want to conduct impactful research of strategic importance to Australia? Incorporate host plant resistance to pests and diseases in the cotton breeding program Grow your research career with a CSIRO PhD Fellowship CSIRO Early Research Career (CERC) Postdoctoral Fellowships provide opportunities to scientists and engineers who have completed their doctorate and have less than three years of relevant postdoctoral work experience. These fellowships aim to develop the next generation of future leaders of the innovation system. In this position you will be the lead researcher for a project titled "Incorporating host plant resistance to pests and diseases in the cotton breeding program. " The key output from this project is to incorporate host plant resistance traits to verticillium wilt, spider mites and whitefly into advanced lines in the CSIRO cotton breeding program, as well as to improve the methodologies to select for resistance. As rhe Postdoctoral Fellow in Hos[...]
{'fraction_non_alphanumeric': 0.06621383647798743, 'fraction_numerical': 0.00880503144654088, 'mean_word_length': 5.898993405067685, 'pattern_counts': {'":': 0, '<': 143, '<?xml version=': 0, '>': 143, 'https://': 2, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '25', 'dir': 'jupyter-structured-clean-dedup', 'id': '667361', 'license': 'MIT', 'n_tokens_mistral': 5299, 'n_tokens_neox': 4912, 'n_words': 2389, 'path': 'notebooks/Converting HTML to Text.ipynb', 'repo_name': 'EdwardJRoss/adzuna-salary-prediction', 'size': '19875'}