Q:

Writing clean code in beautifulsoup

While parsing a table on a webpage with little semantic structure, my beautiful soup expressions are getting really ugly. I might be going about it the wrong way and would like to know how I can rewrite my code to make it more readable and less messy?
For example, in a page there are three tables. Relevant data is in the third table. The actual data starts in the second row. The first entry in the row is an index and the data I need is in the second td element. This second td element has two links and my text of interest is within the second a tag. Translating this into beuatifulsoup I wrote 
soup.find_all('table')[2].find_all('tr')[2].find_all('td')[1].find_all('a')[1].text

works fine, and I grab all the 70 elements in the table using the same principle in a list comprehension. 
relevant_data = [ x.find_all('td')[1].find_all('a')[1].text for x in soup.find_all('table')[2].find_all('tr')[2:]]

Is this kind of code OK or is there any scope for improvement?

A:

Using lxml, you can use XPath.
For example:
html = '''
<body>
    <table></table>
    <table></table>
    <table>
        <tr></tr>
        <tr></tr>
        <tr><td></td><td><a>blah1</a><a>blah1-1</a></td></tr>
        <tr><td></td><td><a>blah2</a><a>blah2-1</a></td></tr>
        <tr><td></td><td><a>blah3</a><a>blah3-1</a></td></tr>
        <tr><td></td><td><a>blah4</a><a>blah4-1</a></td></tr>
        <tr><td></td><td><a>blah5</a><a>blah5-1</a></td></tr>
    </table>
    <table></table>
</body>
'''

import lxml.html
root = lxml.html.fromstring(html)
print(root.xpath('.//table[3]/tr[position()>=2]/td[2]/a[2]/text()'))

output:
['blah1-1', 'blah2-1', 'blah3-1', 'blah4-1', 'blah5-1']

