<jupyter_start><jupyter_text>練習- 觀察 https://afuntw.github.io/Test-Crawling-Website/pages/portfolio/index.html 並撰寫爬蟲程式- 下載 2018/01/29 14:39:10 之後修改過的圖片- 以正確的圖片格式存檔<jupyter_code>import requests
import os

from PIL import Image
from bs4 import BeautifulSoup
from datetime import datetime
from time import ctime
from pprint import pprint

url = 'https://afuntw.github.io/Test-Crawling-Website/pages/portfolio/index.html'
last_modified = datetime(2018, 1, 29, 14, 39, 10)
resp = requests.get(url)
soup = BeautifulSoup(resp.text, 'lxml')
imgs = soup.find_all('img')
imgs = [i['src'] for i in imgs]
results = os.path.abspath('../results')
if not os.path.exists(results):
    os.makedirs(results)

for i in imgs:
    # check header only
    check_resp = requests.head(i)
    check_head = dict(check_resp.headers)
    if 'Last-Modified' in check_head:
        check_modified = check_head['Last-Modified']
        check_modified = datetime.strptime(check_modified, '%a, %d %b %Y %H:%M:%S GMT')
        check_not_modified = check_modified < last_modified
        if check_not_modified:
            continue
    
    img_resp = requests.get(i, stream=True)
    image = Image.open(img_resp.raw)
    filename = os.path.basename(i)
    print('catch the filename {} and the real format is {}'.format(filename, image.format))
    
    real_filename = '{}.{}'.format(
        filename.split('.')[0],
        image.format.lower()
    )
    save_filename = os.path.join(results, real_filename)
    print('catch the real filename {}'.format(real_filename))
    
    image.save(save_filename)
    print('save image at {}'.format(save_filename))<jupyter_output>catch the filename IDPxvSl.jpg and the real format is PNG
catch the real filename IDPxvSl.png
save image at /home/dirl/github/Python-Crawling-Tutorial/results/IDPxvSl.png
catch the filename UKxK6FZ.gif and the real format is PNG
catch the real filename UKxK6FZ.png
save image at /home/dirl/github/Python-Crawling-Tutorial/results/UKxK6FZ.png
{'fraction_non_alphanumeric': 0.11824153612935827, 'fraction_numerical': 0.01616978271854472, 'mean_word_length': 3.900990099009901, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 4, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 1}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '1', 'dir': 'jupyter-structured-clean-dedup', 'id': '332129', 'license': 'Apache-2.0', 'n_tokens_mistral': 740, 'n_tokens_neox': 714, 'n_words': 162, 'path': '01_files_website/04_image_crawling_check_last_modified.ipynb', 'repo_name': 'afunTW/Python-Crawling-Tutorial', 'size': '1979'}