Q:

How do I find the desired XPath for an HTML element?

I am trying to create a Python application that scrapes the HTML from a website using lxml and gathers the countries and their corresponding capitals. I am scraping the HTML from: http://www.manythings.org/vocabulary/lists/2/words.php?f=countries_and_capitals and I can't figure out how to obtain all of the Countries so that I may put them in a List. This is what I have so far:
from lxml import html
import requests

page = requests.get("http://www.manythings.org/vocabulary/lists/2/words.php?f=countries_and_capitals")
tree = html.fromstring(page.content)

countries = tree.xpath('//*[@id="yui-main"]/div/div[2]/div/div[1]/ul/li[1]/b')
capitals = tree.xpath('//*[@id="yui-main"]/div/div[2]/div/div[1]/ul/li[1]/i')

print 'Countries: ', countries
print 'Capitals: ', capitals

Right now the output is two empty lists, and I'm fairly certain it is because the XPaths are incorrect but I'm not familiar enough with XPath nor HTML to correct it. I would much rather be led towards the answer than fed the answer.

A:

This was an interesting problem. Turns out your X-Path and HTML were  correct - running them using Chrome debug tools selected the appropriate elements. However, the issue becomes apparent when debugging via a python interactive shell - the yui-main div doesn't actually exist. 
The webpage is updated dynamically using JavaScript - the content is loaded into the yui-main div at runtime. The xml parser will not execute JavaScript, so your parsed tree will never have the yui-main div.
I confirmed this by simply turning off JavaScript in the browser and visiting the page.
It's trivial to come up with an XPath selector afterwards:
countries = tree.xpath('//li/b/text()')
capitals = tree.xpath('//li/i/text()')

