Recently, I wanted to change my vim colors to something new. So I went to the vim colors website and then I decided that I wanted to download ALL the colors. So I wrote this simple Python script that will download all the files from: Vim ColorsI'm still learning Python and it would be amazing to get some feedback / suggestions if possible.import osimport jsonimport requestsfrom bs4 import BeautifulSoupclass Spider:    def __init__(self, total_pages=40):        self.base_url = http://vimcolors.com/?page=        self.total_pages = total_pages        self.download_dir = 'colors/'        # If we don't have the download directory        if not os.path.isdir(self.download_dir):            print(self.download_dir, 'does not exist, trying to create it...')            # create it...            os.mkdir(self.download_dir)    def download(self, name, url):        # If we have already downloaded this file, just skip        if os.path.isfile(self.download_dir + name):            print('File:', name, 'already exists; skipping.')            return        try:            # Get the response            response = requests.get(url)            # If response is 404 (Not Found), just exit            if response.status_code == 404:                raise Exception('File not found')            # Create the file            with open(self.download_dir + name, 'wb') as file_path:                # Write content to the file                file_path.write(response.content)                # Confirm the download                print('Downloaded', name)        except:            # This is a very generic error, perhaps I'll change it sometime :)            print('Could not download the file', name)            pass    def crawl(self):        def repo_formatter(scheme):            return scheme['github_repo']['address'].replace('github.com', 'raw.githubusercontent.com') \                   + '/master/colors/' + scheme['name'] + '.vim'        # Loop over all the pages        for page in range(self.total_pages):            page_source = requests.get(self.base_url + str(page + 1))            plain_text = page_source.text            soup = BeautifulSoup(plain_text, 'lxml')            # Get the data            json_data = json.loads(soup.find('div', {'id': 'data'}).attrs['data-colorschemes'])            # Download the files            for data in json_data['colorschemes']:                self.download(data['name'] + '.vim', repo_formatter(data))colors_spider = Spider()colors_spider.crawl()
VIM colors downloader in Python
python;python 3.x;web scraping;beautifulsoup;network file transfer
Very nice code in general! Here are a few points, though:Think about using os.makedirs instead of os.mkdir. The former allows creating nested directories (so os.makedirs(~/long/new/path) will create all other folders in betweetn as well, if needed).You should never have a bare except. This prevents you to e.g. CTRL-C to abort the program, since it will also be caught. Use at least except Exception, which all normal exceptions inherit from (but not some special exceptions like pressing CTRL-C).In general it is usually not the best idea to have a lot of string addition. You could use format instead in repo_formatter:def repo_formatter(scheme):    base_url = scheme['github_repo']['address'].replace('github.com', 'raw.githubusercontent.com')    return '{}/master/colors/{}.vim'.format(base_url, scheme['name'])You have some very obvious comments in your code, especially in download. Comments should explain why you do something and not what. The latter should be obvious from your code! And in that function it is really easy to follow what happens, because you have chosen appropriate variable names :)