I am trying to crawl a lyrics website and have made pretty good script (although slow running since this is my first time)#!/bin/bashtouch visitedtouch tobe>visited>tobeurl=$(echo http://www.azlyrics.com/)UA=$(echo Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_3) AppleWebKit/537.75.14 (KHTML, like Gecko) Version/7.0.3 Safari/7046A194A)wget -q $url --user-agent=$(echo UA) -O index.htmlcat index.html | grep \http.*.html\ -o | cut -f2 -d\ >> temp_tobefor i in `cat temp_tobe`;do    basename $(echo $i) >> tobedonerm temp_tobewhile true;dofor i in `cat tobe`;do    echo getting $url -> $i    wget -q $url$i --user-agent=$(echo UA)    echo $i >> visited    #PUT THE NEW LINK TO TOBE    cat $(basename $(echo $url$i)) | grep \.*.html\ -o | cut -f2 -d\ >> tobe    #SORT VISITED    cat visited | sort | uniq > tmp && mv tmp visited    #SORT TOBE    cat tobe | sort | uniq > tmp && mv tmp tobe    #REMOVE THE INTERSECTION FROM TOBE    comm -23 tobe visited > tmp && mv tmp tobedonedoneBut i think i have been blacklisted to do this since i am not able to access that site even from the browser. I starte my VPN but still not able access the site (THIS IS INSANE!!!) saying The connection was reset everytimeI have heard of a technique in which you constantly change your IP (piggybacking behind other users online)Can you enlighten me as to how to achieve this (Preferably  THIS)OR Any other method to still access the websitePlease ask for clarification if needed
How to constantly change IP to avoid getting blacklisted by website for crawling?
bash;ip
I can suggest one way to do that, maybe it's not the best but it will do the job.Use tool for tor called torify or torsocks in you script before wget:torify wget -q $url --user-agent=$(echo UA) -O index.htmlthis command will let the request go throw tor server, but, first of all you have to download tor server then lunch it, after that fill free to do what you want with torify:To install it, for example on CentOS or Ubunutu:yum install torapt-get install torby default torify will be installed.