Q:

Which programming language to scrape data from web and do api calls at the same time?

My project deals with scraping a lot of data from sites that don't have API or calling APIs if there is one. Using multiple threads to improve speed and work real time. Which would be the better programming language for this? I'm comfortable with Python. But, threading is an issue. Thus, thinking of using JS in node.js. Thus, which should I choose?

A:

Threading is an issue in python only if you want to compute multiple things in parallel. If you just want to do a lot of requests, the limitation of the interpreter (only one thread interpreting python at one point) won't be a problem.
In fact, to make a lot of requests simultaneously, you don't even have to use a lot of threads. You can use an async requests library, like requests.async.
If you have some heavy computation to do with the result from the requests, you can always parallelize it in python using multiprocessing, which enable you to bypass the thread limitation I talked earlier.

A:

In python you are able to multi-thread your scrappers. I've used Beautiful Soup in past, but there are alternatives. 

Scrappy 
Mechanize
lxml

Since I have experience using Beautiful Soup, a very simple example to multi-process a scrapper using that is below. 
from BeautifulSoup import BeautifulSoup
from multiprocessing import Process, JoinableQueue, cpu_count

jobs = []
queue = JoinableQueue()

class scrapperClass(Process):
    def __init__(self,queue):
        Process.__init__(self)
        # Other init things

    def run(self):
        # your scraping code here
        # Perhaps save stuff to a DB?

            page = urllib2.urlopen(fullUrl)      # fullUrl can be passed in via the queue, or other possible methods
            soup = BeautifulSoup(page)
            # Read Beautiful Soup docs for how to parse further

def main():
    numProcesses = 2
    for i in xrange(numProcesses):
        p = scrapperClass(queue)
        jobs.append(p)
        p.start()           # This will call the scapperClass.run() method

if __name__ == "__main__":
    main()

