Q:

XPath through python

I have the following html which I have been using:

<html>
<body>
<div class="directions" itemprop="instructions">
<h6>Instructions</h6>
<p>Sharpen your <a href="pencil.html" class="crosslink">pencil</a> (or, alternatively, use your pen)</p>
<p>In a large paper sheet, write your name. When the ink thickens slightly, gently open the <a href="envelop.html" class="crosslink">envelop</a> and insert the <a href="letter.html" class="crosslink" >letter</a> inside folded into 3. Set aside.</p>
<p>Use the pen again to <a href="write.html" class="crosslink">write</a> your name and address into the evelope. Include the destination <a href="address.html" class="crosslink">address</a>.</p>
  <p>Seal the envelop and stamp it</p>
<p class="copyright">Instruction courtesy of John Doe</p>
</div>
</body>
  </html>

The result I'm expecting is getting an array of text elements ordered and without taking into account the html tags.
result=[
'Sharpen your pencil (or, alternatively, use your pen)',
'In a large paper sheet, write your name. When the ink thickens slightly, gently open the envelop and insert the letter inside folded into 3. Set aside',
'Use the pen again to write your name and address into the envelop. Include the destination address',
'Seal the envelop and stamp it'
]

I'm using python to parse the html and get the information pieces I need. with tree.xpath(''//@[itemprop="instructions"]') I'm getting the elements I need. But I can't seem to get the information in the way I want it.
My closest attempt (that still fails) is the following:
for a in tree.xpath('//*[@itemprop="instructions"]'):
    for i in a.xpath('./p'):
        temptext = ""
        for c in i.xpath('text()'):
            temptext += c
        for c in i.xpath('./a'):
            temptext += c.text
        tempIteration.append(temptext)

EDIT FOR CLARITY:
This gets me this result which is incorrect (order is wrong for the 'a' node text).
See the way pencil is at the end of element 1 instead of after 'sharpen your'. Same thing happens in the rest of the lines.
result=[
'Sharpen your (or, alternatively, use your pen)pencil',
'In a large paper sheet, write your name. When the ink thickens slightly, gently open the and insert the inside folded into 3. Set asideenvelopletter',
'Use the pen again to your name and address into the envelop. Include the destination writeaddress',
'Seal the envelop and stamp it',
'Instruction Courtesy of John Doe'
]

I haven't been able to get this working so any help would be greatly appreciated.

A:

You can use the getchildren() method and text and tail properties of elements.
I never used lxml but from the documentation here I got it to work in the example below.
from lxml import etree

html='''<html>
<body>
<div class="directions" itemprop="instructions">
<h6>Instructions</h6>
<p>Sharpen your <a href="pencil.html" class="crosslink">pencil</a> (or, alternatively, use your pen)</p>
<p>In a large paper sheet, write your name. When the ink thickens slightly, gently open the <a href="envelop.html" class="crosslink">envelop</a> and insert the <a href="letter.html" class="crosslink" >letter</a> inside folded into 3. Set aside.</p>
<p>Use the pen again to <a href="write.html" class="crosslink">write</a> your name and address into the evelope. Include the destination <a href="address.html" class="crosslink">address</a>.</p>
  <p>Seal the envelop and stamp it</p>
<p class="copyright">Instruction courtesy of John Doe</p>
</div>
</body>
  </html>'''

tree=etree.HTML(html)
result=[]
for a in tree.xpath('//*[@itemprop="instructions"]'):
    for i in a.xpath('./p'):
        temptext = ""
        temptext += i.text
        for j in i.getchildren():
            temptext += j.text
            temptext += j.tail
        result.append(temptext)

print result

which yields
[
'Sharpen your pencil (or, alternatively, use your pen)', 
'In a large paper sheet, write your name. When the ink thickens slightly, gently open the envelop and insert the letter inside folded into 3. Set aside.', 
'Use the pen again to write your name and address into the evelope. Include the destination address.', 
'Seal the envelop and stamp it', 
'Instruction courtesy of John Doe'
]

you can then just do result[:-1] to throw away the last one

