<jupyter_start><jupyter_text>Scraper für Krypto-Kurse Wir interessieren uns in diesem Notebook für Krypto-Coins.Die Webseite https://coinmarketcap.com/ führt Marktdaten zu den hundert wichtigsten Coins auf.Mit einem einfachen Scraper werden wir diese Daten beschaffen und rudimentär analysieren.  Vorbereitung<jupyter_code>import requests
from bs4 import BeautifulSoup
import pandas as pd
import time<jupyter_output><empty_output><jupyter_text>Scraper<jupyter_code>path = ''<jupyter_output><empty_output><jupyter_text>Liste von allen Kryptowährungen Zuerst kucken wir auf der Seite, welches die 100 grössten Kryptowährungen sind, und laden uns Namen und Links derselbigen.<jupyter_code>base_url = 'https://coinmarketcap.com/'
response = requests.get(base_url)
doc = BeautifulSoup(response.text, "html.parser")
currencies = doc.find_all('a', class_='currency-name-container link-secondary')
currencies[0]
len(currencies)
currency_list = []
for currency in currencies:
    this_currency = {}
    this_currency['name'] = currency.text
    this_currency['link'] = currency['href']
    currency_list.append(this_currency)
df_currencies = pd.DataFrame(currency_list)
df_currencies.head(2)
df_currencies['link'] = df_currencies['link'].str.extract('/currencies/(.+)/')
df_currencies.head(2)
df_currencies.to_csv(path + 'currencies.csv', index=False)<jupyter_output><empty_output><jupyter_text>Daten von den einzelnen Währungen Zuerst testen wir mit einer Probewährung aus, wie wir an die Informationen kommen.<jupyter_code>base_url = 'https://coinmarketcap.com/currencies/bitcoin/historical-data/?start=20181012&end=20191012'
response = requests.get(base_url)
doc = BeautifulSoup(response.text, "html.parser")
days = doc.find_all('tr', class_='text-right')
days_list = []
cells = days[0].find_all('td')
cells
this_day = {}
this_day['date'] = cells[0].text
this_day['open'] = cells[1].text
this_day['high'] = cells[2].text
this_day['low'] = cells[3].text
this_day['close'] = cells[4].text
this_day['volume'] = cells[5].text
this_day['marketcap'] = cells[6].text
this_day
for day in days:
    this_day = {}
    cells = day.find_all('td')
    this_day['date'] = cells[0].text
    this_day['open'] = cells[1].text
    this_day['high'] = cells[2].text
    this_day['low'] = cells[3].text
    this_day['close'] = cells[4].text
    this_day['volume'] = cells[5].text
    this_day['marketcap'] = cells[6].text
    days_list.append(this_day)
df = pd.DataFrame(days_list)
df.head(2)<jupyter_output><empty_output><jupyter_text>Nun wenden wir den Scraper auf alle Währungen an<jupyter_code>df_currencies = pd.read_csv(path + 'currencies.csv')
df_currencies.head(2)
len(df_currencies)
currencies = df_currencies.to_dict(orient='records')
url_start = 'https://coinmarketcap.com/currencies/'
url_end = '/historical-data/?start=20181012&end=20191012'
for currency in currencies:
    print ('working on: ' + currency['name'])
    
    url = url_start + currency['link'] + url_end
    response = requests.get(url)
    doc = BeautifulSoup(response.text, "html.parser")
    
    # print (doc)
    
    days = doc.find_all('tr', class_='text-right')
    days_list = []
    
    this_day = {}
    for day in days:
        this_day = {}
        cells = day.find_all('td')
        this_day['date'] = cells[0].text
        this_day['open'] = cells[1].text
        this_day['high'] = cells[2].text
        this_day['low'] = cells[3].text
        this_day['close'] = cells[4].text
        this_day['volume'] = cells[5].text
        this_day['marketcap'] = cells[6].text
        days_list.append(this_day)
        
    df = pd.DataFrame(days_list)
    filename = currency['name'] + '.csv'
    df.to_csv(path + 'data/' + filename, index=False)
    
    time.sleep(10)
    
print('Done')<jupyter_output>working on: Bitcoin
working on: Ethereum
working on: XRP
working on: Tether
working on: Bitcoin Cash
working on: Litecoin
working on: EOS
working on: Binance Coin
working on: Bitcoin SV
working on: Stellar
working on: TRON
working on: Cardano
working on: UNUS SED LEO
working on: Monero
working on: Chainlink
working on: Huobi Token
working on: IOTA
working on: Dash
working on: Tezos
working on: Ethereum Classic
working on: Cosmos
working on: NEO
working on: Maker
working on: USD Coin
working on: Crypto.com Coin
working on: NEM
working on: Ontology
working on: Dogecoin
working on: Zcash
working on: Basic Attenti...
working on: Paxos Standard
working on: HedgeTrade
working on: VeChain
working on: TrueUSD
working on: Qtum
working on: Decred
working on: Ravencoin
working on: 0x
working on: V Systems
working on: ZB
working on: Bitcoin Gold
working on: Holo
working on: ABBC Coin
working on: EDUCare
working on: OmiseGO
working on: Swipe
working on: DigiByte
working on: Centrality
working on: A[...]
{'fraction_non_alphanumeric': 0.12408300146719765, 'fraction_numerical': 0.013833577866275414, 'mean_word_length': 4.876847290640394, 'pattern_counts': {'":': 0, '<': 20, '<?xml version=': 0, '>': 20, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '399518', 'license': 'MIT', 'n_tokens_mistral': 1711, 'n_tokens_neox': 1642, 'n_words': 405, 'path': '11 Pandas Teil 3/dataprojects/Krypto/Scraper.ipynb', 'repo_name': 'manuelapaganini/20_21_Workfiles', 'size': '4771'}