<jupyter_start><jupyter_text>Overview- Create config file for prometheus- Create config file for influx db- Create volume with config files- Deploy Influx- Deploy Prometheus- Create some basic alert when a target is missing- Test  Get repo with docker-compose filesSource [6za/monitoring-tools](https://github.com/6za/monitoring-tools)<jupyter_code>%%bash

mkdir ~/repos
cd ~/repos
#rm -rf ~/repos/monitoring-tools
git clone https://github.com/6za/monitoring-tools.git
ls  ~/repos/monitoring-tools/prometheus<jupyter_output>prometheus-template.yml<jupyter_text>Deploy Influx- Select node o host- Deploy it - Create prometheusdbReference: - https://hub.docker.com/_/influxdb- https://docs.influxdata.com/influxdb/v1.7/supported_protocols/prometheus/<jupyter_code>import pandas as pd
hosts = pd.read_csv("../common/hosts.csv")
supressed_columns = ['ip','user']
influx_hosts = hosts[(hosts.hostname == "pi-node7" )]
influx_hosts['ip'].to_csv(r'influx-ip.txt', header=False, index=None, sep=' ')
%%bash
export HOME_DIR=$PWD
echo $HOME_DIR
source ../common/env.sh 
while read ip; do
    cd ~/repos/monitoring-tools
    export DOCKER_HOST="tcp://$ip:2376"
    docker-compose -f docker-compose-influx.yaml down 2>/dev/null
    docker-compose -f docker-compose-influx.yaml up -d 2>/dev/null
    # Wait DB is ready
    while [[ "$(curl -s -o /dev/null -w '%{http_code}' -XPOST http://$ip:8086/query --data-urlencode 'q=CREATE DATABASE prometheus' )" != "200" ]]; do
        echo "Waiting influx to be ready..."
        sleep 15
    done
done <influx-ip.txt
cd $HOME_DIR<jupyter_output>/root/05-monitoring-stack
Waiting influx to be ready...
Waiting influx to be ready...<jupyter_text>Create DB on influxDB InfluxDB created and DB created for `prometheus` metrics. This call need to return 200, to confirm creation was done. Deploy Prometheus- Create Prometheus targets files based on a template- Fill template with hosts/monitors known- Create volume with configs- Deploy docker-compose with proper volume mounts<jupyter_code>import yaml


with open('/root/repos/monitoring-tools/prometheus/prometheus-template.yml',"r") as file:
    # The FullLoader parameter handles the conversion from YAML
    # scalar values to Python the dictionary format
    config = yaml.load(file, Loader=yaml.FullLoader)

    influx_config = influx_hosts.head()['ip'].iloc[0]
    config['remote_write'] = [{'url': "http://"+influx_config+ ":8086/api/v1/prom/write?db=prometheus"}]
    config['remote_read']= [{'url': "http://"+influx_config+ ":8086/api/v1/prom/read?db=prometheus"}]
    #Simple Node Exporter
    for index, row in hosts.iterrows():
        docker_host_config = {'job_name': 'node-exporter-'+row['hostname'], 'metrics_path': '/metrics', 'static_configs': [{'targets': [row['ip']+':9100']}]}
        config['scrape_configs'].append(docker_host_config)
    #Simple Docker-Host Exporter       
    for index, row in hosts.iterrows():
        docker_host_config = {'job_name': 'docker-hosts-exporter-'+row['hostname'], 'static_configs': [{'targets': [row['ip']+':9323']}]}
        config['scrape_configs'].append(docker_host_config)    
    #sensor-exporter  on i86 - port 9255
    for index, row in hosts[(hosts.arch == "x86_64")].iterrows():                
        docker_host_config = {'job_name': 'sensor-exporter-'+row['hostname'], 'metrics_path': '/metrics', 'static_configs': [{'targets': [row['ip']+':9255']}]}
        config['scrape_configs'].append(docker_host_config)    
    with open(r'prometheus.yml', 'w') as outputfile:
        documents = yaml.dump(config, outputfile)<jupyter_output><empty_output><jupyter_text>Create Volume on all nodes running prometheus> There is a weird version on dockerhub that breaks if used the last tag. > the known working tag for arm is `prom/prometheus:v2.12.0`<jupyter_code>#hosts[(hosts.arch == "x86_64") & (hosts.hostname == "nuc-01" )]['ip'].to_csv(r'prom-ip.txt', header=False, index=None, sep=' ')
hosts[(hosts.hostname == "pi-node7" )]['ip'].to_csv(r'prom-ip.txt', header=False, index=None, sep=' ')
%%bash
export HOME_DIR=$PWD
echo $HOME_DIR
source ../common/env.sh 
while read ip; do
    cd ~/repos/monitoring-tools
    export DOCKER_HOST="tcp://$ip:2376"
    docker-compose -f docker-compose-prometheus.yaml down 2>/dev/null
done <prom-ip.txt
cd $HOME_DIR
%%bash
source ../common/env.sh 
while read ip; do
    export DOCKER_HOST="tcp://$ip:2376"
    #Create Volume 
    docker volume rm prometheus-conf
    docker volume create --name prometheus-conf
    docker run -d -v prometheus-conf:/root/conf --name init-conf-prom busybox tail -f /dev/null
    # populate volume
    docker cp prometheus.yml init-conf-prom:/root/conf
    docker stop init-conf-prom
    docker rm init-conf-prom
done <prom-ip.txt<jupyter_output>prometheus-conf
prometheus-conf
676defce69e4ecb33d1e5df0aa2f09a20ed876bc208192168d6dec1942414b3b
init-conf-prom
init-conf-prom<jupyter_text>Deploy Prometheus on all target nodes<jupyter_code>%%bash
export HOME_DIR=$PWD
echo $HOME_DIR
source ../common/env.sh 
while read ip; do
    cd ~/repos/monitoring-tools
    export DOCKER_HOST="tcp://$ip:2376"
    docker-compose -f docker-compose-prometheus.yaml up -d 2>/dev/null
done <prom-ip.txt
cd $HOME_DIR
hosts['ip'].to_csv(r'ip.txt', header=False, index=None, sep=' ')
%%bash
source ../common/env.sh 
echo "ip|name|id|image|ports|runnning" > containers.csv
while read ip; do
    export DOCKER_HOST="tcp://$ip:2376"
    docker ps  --format "$ip|{{.Names}}|{{.ID}}|{{.Image}}|{{.Ports}}|{{.RunningFor}}"  >> containers.csv 2>/dev/null || :
done <ip.txt
containers = pd.read_csv("containers.csv", sep = '|')
pd.merge(containers, hosts, on='ip', how='inner').drop(columns=supressed_columns)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.15090050708165764, 'fraction_numerical': 0.022031823745410038, 'mean_word_length': 4.432098765432099, 'pattern_counts': {'":': 2, '<': 23, '<?xml version=': 0, '>': 28, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '372835', 'license': 'Apache-2.0', 'n_tokens_mistral': 2132, 'n_tokens_neox': 1940, 'n_words': 456, 'path': '05-monitoring-stack/run-book-monitoring-stack.ipynb', 'repo_name': '6za/runbooks-jupyter', 'size': '5719'}