<jupyter_start><jupyter_text>Jupyter NoteBook overviewThis Jupyter-notebook is a beginner-level introduction to Jupyter-notebook (formerly known as IPython) for interactive Python programming, high-performance numerical computing, and data visualization. It assumes nothing more than familiarity with Python. It targets developers, students, teachers, hobbyists who know Python a bit, and who want to learn IPython for the extended console, the Notebook, and for more advanced scientific applications. Code ExampleMost files are Jupyter Notebooks (.ipynb extension with JSON data). There are also some Python external scripts (.py extension).The following modules are used in at least one example:* Python 3.7,0* Jupyter-notebook 5.6* NumPy* Pandas* SciPy* Matplotlib* Matplotlib basemap* html5lib* PyQt or PySide* Cython* Crycompare Section 1 Data Analytics: cryptocurrency price oscillation Data Workflow and analysis    -This Jupyter Notebook renders cryptocurrency price oscillation  prediction        * Data Set Source: https://github.com/amisolution/ERC20-AMIS/blob/master/dataset/export-token-0x949bEd886c739f1A3273629b3320db0C5024c719.csv                * Data Set Information: https://github.com/amisolution/ERC20AMIS                - Example 1 : View this Jupyter Notebook for Cryptocurrency Data Modelling & Analysis inside https://nbviewer.jupyter.org/github/amis-erc20/cryptocompare-notebook- Follow the Link to access [Example 1 Jupyter Notebook for Cryptocurrency Data Modelling & Analysis](https://nbviewer.jupyter.org/github/amis-erc20/cryptocompare-notebook/cryptocompare.ipynb)https://github.com/amis-erc20/cryptocompare-notebook- Example 2 : View Jupyter Notebook for Cryptocurrency Data Modelling & Analysis in action inside https://nbviewer.jupyter.org/github/amis-erc20/cryptocompare-notebook- Follow the Link to access [Example 2  Jupyter Notebook for Cryptocurrency Data Modelling & Analysis](https://nbviewer.jupyter.org/github/amis-erc20/cryptocompare-notebook/cryptocompare.ipynb)Notes:    Attribute Information:                * date: Date in format dd/mm/yyyy                * time: time in format hh:mm:ss                * Volume in: Volume in                * Volume out: Volume out                * Price     Content Section 1 Data Analytics: Altcoin and cryptocurrency prediction Data Workflow and analysis    -This Jupyter Notebook analyses cryptocurrency trends and predictionTime series have been included in the data set for testing the regression models and to filter out non predictive attributes (parameters).For more information about the Jupyter Notebook scripts and figures, please refer to the paper and to the following github repository:[Web Link](https://github.com/amisolution/ERC20AMIS/)                Section 2        * Data Set Source: Quandl                * Data Set Information: https://github.com/amisolution/ERC20AMIS/dataset/ Section 3        * Data Set Source: CoinMarketCap                * Data Set Information: https://github.com/amisolution/ERC20AMIS/dataset/ Section 4        * Data Set Source: CryptoCompare                * Data Set Information: https://github.com/amisolution/ERC20AMIS/dataset/ Section 5        * Data Set Source: Etherscan                * Data Set Information: https://github.com/amisolution/ERC20AMIS/dataset/  Section 1  Data Analytics: Altcoin and cryptocurrency prediction  Project setupLet's first import the required dependencies.<jupyter_code>!pip install pandas
!pip install sklearn
!pip install matplotlib
!pip install joblib
!pip install crycompare
!pip install ClusterLib
!pip install lxml
!pip install html5lib<jupyter_output>Requirement already satisfied: pandas in /srv/conda/lib/python3.7/site-packages (0.24.2)
Requirement already satisfied: python-dateutil>=2.5.0 in /srv/conda/lib/python3.7/site-packages (from pandas) (2.8.0)
Requirement already satisfied: numpy>=1.12.0 in /srv/conda/lib/python3.7/site-packages (from pandas) (1.16.2)
Requirement already satisfied: pytz>=2011k in /srv/conda/lib/python3.7/site-packages (from pandas) (2018.9)
Requirement already satisfied: six>=1.5 in /srv/conda/lib/python3.7/site-packages (from python-dateutil>=2.5.0->pandas) (1.12.0)
Requirement already satisfied: sklearn in /srv/conda/lib/python3.7/site-packages (0.0)
Requirement already satisfied: scikit-learn in /srv/conda/lib/python3.7/site-packages (from sklearn) (0.20.3)
Requirement already satisfied: scipy>=0.13.3 in /srv/conda/lib/python3.7/site-packages (from scikit-learn->sklearn) (1.2.1)
Requirement already satisfied: numpy>=1.8.2 in /srv/conda/lib/python3.7/site-packages (from scikit-learn->sklearn) (1.16.2)
R[...]<jupyter_text>Retrieve the data from the data source<jupyter_code>import coinmarketcap

import bs4 as bs
import datetime as dt
import os
import pandas as pd
#import pandas_datareader as web
import pickle
import requests

def save_cmc_tickers():
    resp = requests.get('https://coinmarketcap.com')
    soup = bs.BeautifulSoup(resp.text, "lxml")
    table = soup.find('table', {'class': 'table floating-header'})
    #table1 = soup.find('table', {'class': 'no-wrap text-right'})
    tickers = []
    for row in table.findAll('tr')[1:]:
        ticker = row.findAll('a')[0].text
        price = row.findAll('a', {'class': 'price'})[0].text
        tickers.append(ticker)
        tickers.append(price)

    with open("cmctickers.pickle", "wb") as f:
        pickle.dump(tickers, f)

    print(tickers)

    return tickers

    #save_cmc_tickers()

    def get_data_from_cmc(reload_cmc100=False):

        if reload_cmc100:
            tickers = save_cmc_tickers()
        else:
            with open("cmctickers.pickle", "rb") as f:
                tickers = pickle.load(f)

    if not os.path.exists('stock_dfs'):
        os.makedirs('stock_dfs')

    start = dt.datetime(2010, 1, 1)
    end = dt.datetime(2017, 12, 31)

    for ticker in tickers:
        print(ticker)
        if not os.path.exists('stock_dfs/{}.csv'.format(ticker)):
            df = pd.read_html(ticker, 'coinmarketcap', start, end)
            df.to_csv('stock_dfs/{}.csv'.format(ticker))
        else:
            print("Already have {}".format(ticker))

#get_data_from_cmc()
#print()
coinmarketcap.core.Market()
!conda list
import pandas as pd
import matplotlib as plt
%matplotlib inline
def CryptoData(symbol, frequency):
    #Params: String symbol, int frequency = 300,900,1800,7200,14400,86400
    #Returns: df from first available date
    url ='https://poloniex.com/public?command=returnChartData&currencyPair='+symbol+'&end=9999999999&period='+str(frequency)+'&start=0'
    df = pd.read_json(url)
    df.set_index('date',inplace=True)
    return df
df = CryptoData('USDT_BTC', 86400)['close']
df.head()
df.tail()
df.plot()
df.pct_change().describe()
df.pct_change().hist(bins=100)
import coinmarketcap

import bs4 as bs
import datetime as dt
import os
import pandas as pd
#import pandas_datareader as web
import pickle
import requests

def save_cmc_tickers():
    resp = requests.get('https://coinmarketcap.com')
    soup = bs.BeautifulSoup(resp.text, "lxml")
    table = soup.find('table', {'class': 'table floating-header'})
    #table1 = soup.find('table', {'class': 'no-wrap text-right'})
    tickers = []
    for row in table.findAll('tr')[1:]:
        ticker = row.findAll('a')[0].text
        price = row.findAll('a', {'class': 'price'})[0].text
        tickers.append(ticker)
        tickers.append(price)

    with open("cmctickers.pickle", "wb") as f:
        pickle.dump(tickers, f)

    print(tickers)

    return tickers

    #save_cmc_tickers()

    def get_data_from_cmc(reload_cmc100=False):

        if reload_cmc100:
            tickers = save_cmc_tickers()
        else:
            with open("cmctickers.pickle", "rb") as f:
                tickers = pickle.load(f)

    if not os.path.exists('stock_dfs'):
        os.makedirs('stock_dfs')

    start = dt.datetime(2010, 1, 1)
    end = dt.datetime(2017, 12, 31)

    for ticker in tickers:
        print(ticker)
        if not os.path.exists('stock_dfs/{}.csv'.format(ticker)):
            df = pd.read_html(ticker, 'coinmarketcap', start, end)
            df.to_csv('stock_dfs/{}.csv'.format(ticker))
        else:
            print("Already have {}".format(ticker))

#get_data_from_cmc()
#print()
coinmarketcap.core.Market()
def CryptoData(symbol, frequency):
    #Params: String symbol, int frequency = 300,900,1800,7200,14400,86400
    #Returns: df from first available date
    url ='https://min-api.cryptocompare.com/data/top/exchanges/full?fsym=AMIS&tsym=ETH'
    df = pd.read_json(url)
    df.set_index('date',inplace=True)
    return df
df.head()
df.tail()
def CryptoData(symbol, frequency):
    #Params: String symbol, int frequency = 300,900,1800,7200,14400,86400
    #Returns: df from first available date
    url ='https://min-api.cryptocompare.com/data/top/exchanges/full?fsym=AMIS&tsym=ETH'
    df = pd.read_json(url)
#    df.set_index('date',inplace=True)
    return df
#df = CryptoData('AMIS_ETH', 86400)['close']
df = CryptoData('AMIS','ETH')
df.head()
import html5lib
import pandas as pd
import lxml
#url = 'https://min-api.cryptocompare.com/data/generateAvg?fsym=AMIS&tsym=ETH&e=etherdelta'
url = 'https://github.com/amisolution/ERC20-AMIS/blob/master/dataset/export-token-0x949bEd886c739f1A3273629b3320db0C5024c719.csv'
df=pd.read_csv(url)
df[0]
import pandas as pd
import lxml
url = 'https://coinmarketcap.com/all/views/all/'
df=pd.read_html(url, header=None)
df[0]<jupyter_output><empty_output><jupyter_text>Now let's retrieve some data from the AMIS dataset<jupyter_code># %%timeit
#url= 'https://raw.githubusercontent.com/amisolution/ERC20AMIS/dataset/'
URL = 'https://github.com/amisolution/ERC20-AMIS/blob/master/dataset/export-token-0x949bEd886c739f1A3273629b3320db0C5024c719.csv'
import pandas as pd
import html5lib
df.head()
df.tail()

df = pd.read_html(url)
df = df[0].loc[:, ~df[0].columns.str.contains('^Unnamed')]
df.head(2)
from crycompare import price as price
print(price.coin_snapshot('amis', 'eth'))
import crycompare
crycompare.price.coin_list()
df = pd.read_html(url)
df = df[0].loc[:, ~df[0].columns.str.contains('^Unnamed')]
df.head(2)
import clusterlib
# step 1 https://gmarti.gitlab.io/cryptocurrency/2017/08/25/download-cryptocoins-api-python.html
%load_ext autoreload
%autoreload 2

import numpy as np
import pandas as pd
from joblib import Parallel, delayed
import operator
import matplotlib.pyplot as plt

#from crycompare import *
#from ClusterLib import *
from crycompare import  price as p
#from ClusterLib.clusterlib import *
#from ClusterLib.distlib import *
#from clusterlib.clusterlib import *
from clusterlib import *
#from clusterlib.distlib import *

%matplotlib inline
# step 2 https://gmarti.gitlab.io/cryptocurrency/2017/08/25/download-cryptocoins-api-python.html
#p = crycompare.price.price(from_curr=Eth,to_curr=AMIS)
p = crycompare.price.price('amis', 'eth')

coinList = crycompare.price.coin_list()
#crycompare.price.coin_list()
coins = sorted(list( coinList['Data'].keys() ))
# step 3 https://gmarti.gitlab.io/cryptocurrency/2017/08/25/download-cryptocoins-api-python.html
#h = crycompare.price.History()
h = crycompare.price.price_historical('amis', 'eth', 'CCCAGG')
#h = crycompare.price_historical('amis', 'eth', 'CCCAGG')
df_dict = {}
for coin in coins:
#    histo = h.histoDay(coin,'USD',allData=True)
#    histo = h.crycompare.price.day_avg(coin,'USD',allData=True)
    histo = crycompare.price.day_avg('amis', 'eth')
#    histo = crycompare.price_historical
    if histo['Data']:
        df_histo = pd.DataFrame(histo['Data'])
        df_histo['time'] = pd.to_datetime(df_histo['time'],unit='s')
        df_histo.index = df_histo['time']
        del df_histo['time']
        del df_histo['volumefrom']
        del df_histo['volumeto']
        
        df_dict[coin] = df_histo
# Step 4
crypto_histo = pd.concat(df_dict.values(), axis=1, keys=df_dict.keys())
# Step 5
histo_coins = [elem for elem in crypto_histo.columns.levels[0] if not elem == 'MYC']
# Step 6
histo_length = {}
for coin in histo_coins:
    histo_length[coin] = np.sum( ~np.isnan(crypto_histo[coin]['close'].values) )
    
sorted_length = sorted(histo_length.items(), key=operator.itemgetter(1),reverse=True)
# Step 7
# we keep the 300 coins having the longest time series of historical prices
sub_coins = [sorted_length[i][0] for i in range(300)]

sub_crypto_histo = crypto_histo[sub_coins]
sub_crypto_histo.tail()
N = len(sub_coins)
recent_histo = sub_crypto_histo[-1000:]
returns_dict = {}
for coin in sub_coins:
    coin_histo = recent_histo[coin]
    coin_returns = pd.DataFrame(np.diff(np.log(coin_histo.get_values()),axis=0))
    returns_dict[coin] = coin_returns

recent_returns = pd.concat(returns_dict.values(),axis=1,keys=returns_dict.keys())
recent_returns.index = recent_histo.index[1:]
recent_returns = recent_returns.replace([np.inf, -np.inf], np.nan)
recent_returns = recent_returns.fillna(value=0)
recent_returns.isnull().values.any()
plt.figure(figsize=(40,10))
for coin in crycompare.price.sub_coins:
    plt.plot(recent_returns[coin])
#plt.legend(sub_coins,loc='upper left')
plt.xlabel('time',fontsize=18)
plt.ylabel('returns \'X/USD\'',fontsize=18)
plt.show()
#### End of https://gmarti.gitlab.io/cryptocurrency/2017/08/25/download-cryptocoins-api-python.html
%ls<jupyter_output>cryptocompare.ipynb  LICENSE  README.md
{'fraction_non_alphanumeric': 0.1251684383889804, 'fraction_numerical': 0.03248989369666118, 'mean_word_length': 4.1738962044926415, 'pattern_counts': {'":': 0, '<': 11, '<?xml version=': 0, '>': 21, 'https://': 26, 'lorem ipsum': 0, 'www.': 0, 'xml': 5}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '257460', 'license': 'BSD-2-Clause', 'n_tokens_mistral': 4854, 'n_tokens_neox': 4449, 'n_words': 1032, 'path': 'cryptocompare.ipynb', 'repo_name': 'amis-erc20/cryptocompare-notebook', 'size': '13358'}