Few things in life give me pleasure like scraping structured and unstructured data from the Internet and making use of it in my models. For instance, the Data Science Toolkit (or RDSTK for R programmers) allows me to pull lots of good location-based data using IP's or addresses and the tm.webmining.plugin for R's tm package makes scraping financial and news data straightfoward. When going beyond such (semi-) structured data I tend to use XPath.However, I'm constantly getting throttled by limits on the number of queries you're allowed to make. I think Google limits me to about 50,000 requests per 24 hours, which is a problem for Big Data.From a technical perspective getting around these limits is easy -- just switch IP addresses and purge other identifiers from your environment. However, this presents both ethical and financial concerns (I think?).Is there a solution that I'm overlooking?
Ethically and Cost-effectively Scaling Data Scrapes
text mining;scraping
For many APIs (most I've seen) ratelimiting is a function of your API Key or OAuth credentials. (Google, Twitter, NOAA, Yahoo, Facebook, etc.) The good news is you won't need to spoof your IP, you just need to swap out credentials as they hit there rate limit. A bit of shameless self promotion here but I wrote a python package specifically for handling this problem.https://github.com/rawkintrevo/angemilnerhttps://pypi.python.org/pypi/angemilner/0.2.0It requires a mongodb daemon and basically you make a page for each one of your keys. So you have 4 email addresses each with a separate key assigned. When you load the key in you specify the maximum calls per day and minimum time between uses. Load keys:from angemilner import APIKeyLibrarianl= APIKeyLibrarian()l.new_api_key(your_assigned_key1, 'noaa', 1000, .2)l.new_api_key(your_assigned_key2, 'noaa', 1000, .2)Then when you run your scraper for instance the NOAA api:url= 'http://www.ncdc.noaa.gov/cdo-web/api/v2/stations' payload= {  'limit': 1000,        'datasetid':  'GHCND',         'startdate': '1999-01-01' }r = requests.get(url, params=payload, headers= {'token': 'your_assigned_key'})becomes:url= 'http://www.ncdc.noaa.gov/cdo-web/api/v2/stations'payload= {  'limit': 1000,            'datasetid':  'GHCND',            'startdate': '1999-01-01' }r = requests.get(url, params=payload, headers= {'token': l.check_out_api_key('noaa')['key']})so if you have 5 keys, l.check_out_api_key returns the key that has the least uses and waits until enough time has elapsed for it to be used again. Finally to see how often your keys have been used / remaining useage available:pprint(l.summary())I didn't write this for R because most scraping is done in python (most of MY scraping).  It could be easily ported. Thats how you can technically get around rate limiting.  Ethically ...UPDATE The example uses Google Places API here