Q:

xpathSApply function in R only returning one value when multiple are expected

I'm looking to scrape the article links from this website: http://america.aljazeera.com/topics/topic/categories/us.html
I'm simplifying my task by ignoring the pagination and am only interested in the first 10 articles that are listed and currently have the following syntax:
library(RCurl)
library(XML)
response <- getURL('http://america.aljazeera.com/topics/topic/categories/us.html')
html <- htmlParse(response)
xpath <- "//div[@class='story-holder']//a"
xpathSApply(html, xpath, xmlGetAttr, 'href')

I would have expected to get all of the article links, the links in the images, and the links for the tags on each article (these will be parsed later). However, I'm only getting the first link that is embedded in the thumbnail of the first article. Any idea why it's not returning more results?
Thanks!

A:

That page has invalid HTML markup which is confusing the XML parser. Specifically it has some self-closing div's which seem to be throwing everything off. You can try a more specific xpath expression which avoids the "bad" parts. If you just want the article links, maybe: xpath <- "//div[@class='media-body']//h3/a"

