<jupyter_start><jupyter_text>Field Notes: Ambari Blueprints![Take This](http://nola.liberty.me/wp-content/uploads/sites/1472/2014/11/dangerousgif.gif)  Nerd Alert: Presenting from ipython* https://github.com/damianavila/RISE  Not Zeppelin  whoami<NAME>Partner Engineering, EMEA![Me](http://i.imgur.com/8uBbEpH.jpg)  Today* Requirements for Blueprints* Refresher on Ambari Stacks* Blueprint & Cluster Template* Deploying the Blueprint & Cluster* Field Notes *(sort of)** Questions  Not Today Deploying with Ambari:- Infrastructure & Node Prep- Deploying Ambari Server & Agents- Ambari considerations (java, database, …)- Lessons learned from large scale deployments Ongoing operations:- General overview of the API- Configuration Groups- Adding nodes to a config group  What You'll Need* Ambari Server & Agents Installed* Ambari Agents Registered to Ambari Server (/api/v1/hosts)* HDP prereqs(networking, OS repos, …)* If using separate or non-default SQL databases, configure them first.* Access & credentials to Ambari Server* Blueprint (JSON)* Cluster Description (JSON)   * https://wiki.hortonworks.com/display/SE/SE+Cloud  * http://github.com/HortonworksUniversity/Ops_Labs/1.1.0/build/security/ambari-bootstrap  * CloudBreak    * Note: HA Blueprints not supported *(simple validation bug should be fixed soon)*<jupyter_code>r = s.get(api_url + '/api/v1/hosts')
print(json.dumps(r.json(), indent=2))<jupyter_output>{
  "href": "http://sroberts-bp02.cloud.hortonworks.com:8080/api/v1/hosts",
  "items": [
    {
      "href": "http://sroberts-bp02.cloud.hortonworks.com:8080/api/v1/hosts/sroberts-bp01.cloud.hortonworks.com",
      "Hosts": {
        "host_name": "sroberts-bp01.cloud.hortonworks.com"
      }
    },
    {
      "href": "http://sroberts-bp02.cloud.hortonworks.com:8080/api/v1/hosts/sroberts-bp02.cloud.hortonworks.com",
      "Hosts": {
        "host_name": "sroberts-bp02.cloud.hortonworks.com"
      }
    },
    {
      "href": "http://sroberts-bp02.cloud.hortonworks.com:8080/api/v1/hosts/sroberts-bp03.cloud.hortonworks.com",
      "Hosts": {
        "host_name": "sroberts-bp03.cloud.hortonworks.com"
      }
    },
    {
      "href": "http://sroberts-bp02.cloud.hortonworks.com:8080/api/v1/hosts/sroberts-bp04.cloud.hortonworks.com",
      "Hosts": {
        "host_name": "sroberts-bp04.cloud.hortonworks.com"
      }
    },
    {
      "href": "http://sroberts-bp02.cloud.hortonworks.com:808[...]<jupyter_text>Reminder: Ambari Stacks* Stack: HDP, PHD, ...    * Versions: 2.2      * Services: HDFS, SPARK        * Components: NODEMANAGER  Service & Component List<jupyter_code>r = s.get(api_url + '/api/v1/stacks/HDP/versions/2.2/services')

stackservicecomponents = {}
for a in r.json()['items']:
    r = s.get(a['href'] + '/components')
    components = []
    for b in [a['StackServiceComponents'] for a in r.json()['items']]:
        service = b['service_name']
        components.append(b['component_name'])
    stackservicecomponents[service] = ' '.join(components)
pd.DataFrame.from_dict(stackservicecomponents, orient='index').sort()<jupyter_output><empty_output><jupyter_text>A Blueprint isJSON document with 3 sections:* `Blueprints`: Ambari Stack to use * `host_groups`: Grouping of hosts and the Ambari Stack's components to deploy * `configurations`: (optional) Specific configurations to pass through for the Ambari Stack ```json{  "Blueprints": {    "stack_name": "HDP", "stack_version": "2.2"  },    "host_groups": [    { "name": "master_1", "components": [ { "name": "NAMENODE" }, ... ] },    { "name": "slave_1", "components": [ { "name": "DATANODE" }, ... ] }  ],    "configurations": [    { "hive-site": { "hive.execution.engine": "tez" }  ]}```  Blueprint example* With special configurations:  - HA HDFS  - HA YARN Resource Manager  - Set different HDFS dirs depend on host_group  - and Oozie using Postgresql instead of Derby<jupyter_code>blueprint = json.loads(open('blueprints/blueprint-config-example.json').read())
#blueprint = json.loads(open('blueprints/blueprint-hdfs-ha.json').read())
#blueprint = json.loads(open('blueprints/blueprint-yarn-ha.json').read())
#blueprint = json.loads(open('blueprints/hdp-all.json').read())
print(json.dumps(blueprint, indent=2))
host_groups = {}
for group in blueprint['host_groups']:
    host_group = group['name']
    components = []
    for component in group['components']:
        components.append(component['name'])
    host_groups[host_group] = components<jupyter_output><empty_output><jupyter_text>Host Groups & Components<jupyter_code>display(pd.DataFrame.from_dict(OrderedDict(sorted(host_groups.items())), orient='index').T.sort())<jupyter_output><empty_output><jupyter_text>Upload BlueprintPOST /api/v1/blueprints/blueprintname<jupyter_code>## Upload the Blueprint

body = blueprint
r = s.post(api_url + '/api/v1/blueprints/testblueprint', data=json.dumps(body))
print(r.status_code) # should return 201
#print(json.dumps(r.json(), indent=2))
r = s.get(api_url + '/api/v1/blueprints/testblueprint')
print(json.dumps(r.json(), indent=2))<jupyter_output><empty_output><jupyter_text>Cluster Description  Setup Oozie databaseOn Ambari Server:```bashecho "host all all 172.24.0.0/16 trust" >> /var/lib/pgsql/data/pg_hba.conf/etc/init.d/postgresql restartsudo -u postgres psqlCREATE DATABASE oozie;CREATE USER oozie WITH PASSWORD '<PASSWORD>';GRANT ALL PRIVILEGES ON DATABASE oozie TO oozie;```<jupyter_code>cluster = {
  "blueprint": "testblueprint",
  "configurations": [
    { "oozie-site": {
          "oozie.db.schema.name" : "oozie",
          "oozie.service.JPAService.create.db.schema" : "true",
          "oozie.service.JPAService.jdbc.driver" : "org.postgresql.Driver",
          "oozie.service.JPAService.jdbc.username" : "oozie",
          "oozie.service.JPAService.jdbc.password" : "<PASSWORD>",
          "oozie.service.JPAService.jdbc.url" : "jdbc:postgresql://sroberts-bp02.cloud.hortonworks.com:5432/oozie?createDatabaseIfNotExist=true"
    }},
    { "yarn-site" : {
          "yarn.resourcemanager.hostname.rm1": "sroberts-bp04.cloud.hortonworks.com",
          "yarn.resourcemanager.hostname.rm2": "sroberts-bp05.cloud.hortonworks.com"
    }}          
  ],
  "default_password": "<PASSWORD>",
  "host_groups": [
    { "hosts": [
        { "fqdn": "sroberts-bp02.cloud.hortonworks.com" }
      ], "name": "gateway"
    },
    { "hosts": [
        { "fqdn": "sroberts-bp03.cloud.hortonworks.com" }
      ], "name": "master_1"
    },
    { "hosts": [
        { "fqdn": "sroberts-bp04.cloud.hortonworks.com" }
      ], "name": "master_2"
    },
    { "hosts": [
        { "fqdn": "sroberts-bp05.cloud.hortonworks.com" }
      ], "name": "master_3"
    },
    { "configurations": [
       { "yarn-site": {
         "yarn.nodemanager.local-dirs": "/mnt/hdfs0/yarn/local,/mnt/hdfs1/yarn/local,/mnt/hdfs2/yarn/local",
         "yarn.nodemanager.log-dirs": "/mnt/hdfs0/yarn/log,/mnt/hdfs1/yarn/log,/mnt/hdfs2/yarn/log"
        }
       },{ "hdfs-site": { "dfs.datanode.data.dir": "/mnt/hdfs0/data,/mnt/hdfs1/data,/mnt/hdfs2/data"}}
      ],
      "hosts": [
        { "fqdn": "sroberts-bp06.cloud.hortonworks.com" }
      ],
      "name": "slave_standard"     
    },
    { "hosts": [
        { "fqdn": "sroberts-bp07.cloud.hortonworks.com" },
        { "fqdn": "sroberts-bp08.cloud.hortonworks.com" }
      ], "name": "slave_archive"
    }
  ]
}
body = cluster
r = s.post(api_url + '/api/v1/clusters/mycluster', data=json.dumps(body))
print(r.status_code) ## Should return 202
print(json.dumps(r.json(), indent=2))
r = s.get(api_url + '/api/v1/clusters/mycluster/requests/1')
print(json.dumps(r.json()['Requests'], indent=2))<jupyter_output>{
  "type": "INTERNAL_REQUEST",
  "request_status": "IN_PROGRESS",
  "request_context": "Install and start all services",
  "create_time": 1430343177885,
  "cluster_name": "mycluster",
  "completed_task_count": 0,
  "queued_task_count": 34,
  "inputs": null,
  "timed_out_task_count": 0,
  "end_time": -1,
  "operation_level": null,
  "aborted_task_count": 0,
  "failed_task_count": 0,
  "request_schedule": null,
  "resource_filters": [],
  "task_count": 74,
  "progress_percent": 5.5540540540540535,
  "id": 1,
  "exclusive": false,
  "start_time": 1430343178728
}<jupyter_text>Export Blueprint/api/v1/blueprints/api/v1/clusters/clustername?format=blueprint  Field Notes: How to ...* Separate log locations* HA  Field Notes: Separate Databases    Example: PostgreSQL for Oozie1. Prepare the database (see below)2. Add appropriate configuration to Blueprint or Cluster template  Field Notes: Blueprint Schema Changes from 1.7 to 2.0Ambari Metrics replaces Ganglia & Nagios| 1.7                          | 2.0               ||------------------------------|-------------------|| NAGIOS_SERVER GANGLIA SERVER | METRICS_COLLECTOR || GANGLIA_MONITOR              | METRICS_MONITOR   |  Field Notes: HDFS dirs- Blueprints will not detect your mount points.- It will use the default path (/hadoop/...) unless set.- Add to your Blueprint or Cluster template.- They can be added globally or different for each host-group.```json{ "hdfs-site": { "dfs.datanode.data.dir": "/mnt/hdfs0/data,/mnt/hdfs1/data,/mnt/hdfs2/data"}}```  Field Notes: TimeoutsRaise the limits if running with limited networking or on slow hardware``` grep agent.*timeout /etc/ambari-server/conf/ambari.propertiesagent.package.install.task.timeout=1800agent.task.timeout=900```  Consideration: Local Repositories/api/v1/stacks/HDP/versions/2.2/operating_systems/ ```for repo in HDP-2.2 HDP-UTILS-1.1.0.20; do    curl -sSu admin http://${ambari_server}:8080/api/v1/stacks/HDP/versions/2.2/operating_systems/redhat6/repositories/${repo} -o /tmp/update-repo.txt    sed -ir -e 's/\(public\|private\)-repo-1.hortonworks.com/repo.cloud.hortonworks.com/g' -e '/^  "href"/d' /tmp/update-repo.txt    curl -sSu admin -H x-requested-by:sean http://${ambari_server}:8080/api/v1/stacks/HDP/versions/2.2/operating_systems/redhat6/repositories/${repo} -T /tmp/update-repo.txtdone```  Field Notes: Stack Advisor does not run on Blueprints! But this undocumented API helps: `/api/v1/stacks/HDP/versions/2.2/recommendations`<jupyter_code>body = {
      "recommend" : "host_groups",
      "services" : [ "AMBARI_METRICS","FALCON","FLUME","HBASE","HDFS","HIVE","KAFKA","KNOX","MAPREDUCE2","OOZIE","PIG","SLIDER","SPARK","SQOOP","STORM","TEZ","YARN","ZOOKEEPER" ],
      "hosts" : [ "sroberts-bp02.cloud.hortonworks.com","sroberts-bp03.cloud.hortonworks.com","sroberts-bp04.cloud.hortonworks.com","sroberts-bp05.cloud.hortonworks.com","sroberts-bp06.cloud.hortonworks.com","sroberts-bp07.cloud.hortonworks.com","sroberts-bp08.cloud.hortonworks.com" ]
}

r = s.post(api_url + '/api/v1/stacks/HDP/versions/2.2/recommendations', data=json.dumps(body))
print(json.dumps(r.json(), indent=2))
body = {
      "recommend" : "configurations",
      "services" : [ "AMBARI_METRICS","FALCON","FLUME","HBASE","HDFS","HIVE","KAFKA","KNOX","MAPREDUCE2","OOZIE","PIG","SLIDER","SPARK","SQOOP","STORM","TEZ","YARN","ZOOKEEPER" ],
      "hosts" : [ "sroberts-bp02.cloud.hortonworks.com","sroberts-bp03.cloud.hortonworks.com","sroberts-bp04.cloud.hortonworks.com","sroberts-bp05.cloud.hortonworks.com","sroberts-bp06.cloud.hortonworks.com","sroberts-bp07.cloud.hortonworks.com","sroberts-bp08.cloud.hortonworks.com" ]
}

r = s.post(api_url + '/api/v1/stacks/HDP/versions/2.2/recommendations', data=json.dumps(body))
print(json.dumps(r.json(), indent=2))<jupyter_output>{
  "resources": [
    {
      "services": [
        "KAFKA",
        "PIG",
        "SPARK",
        "MAPREDUCE2",
        "YARN",
        "FALCON",
        "SLIDER",
        "HIVE",
        "TEZ",
        "ZOOKEEPER",
        "STORM",
        "SQOOP",
        "HBASE",
        "OOZIE",
        "FLUME",
        "KNOX",
        "HDFS",
        "AMBARI_METRICS"
      ],
      "recommendations": {
        "blueprint": {
          "host_groups": [],
          "configurations": {
            "hadoop-env": {
              "properties": {
                "namenode_opt_maxnewsize": "512",
                "namenode_opt_newsize": "512",
                "namenode_heapsize": "2048"
              }
            },
            "hbase-site": {
              "properties": {
                "hbase.regionserver.global.memstore.upperLimit": "0.4"
              }
            },
            "hbase-env": {
              "properties": {
                "hbase_master_heapsize": "1024",
                "hbase_r[...]
{'fraction_non_alphanumeric': 0.17077676921841112, 'fraction_numerical': 0.024660615310466705, 'mean_word_length': 3.520697167755991, 'pattern_counts': {'":': 99, '<': 30, '<?xml version=': 0, '>': 32, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 2, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '34974', 'license': 'Apache-2.0', 'n_tokens_mistral': 4803, 'n_tokens_neox': 4510, 'n_words': 701, 'path': 'build/security/ambari-bootstrap-master/api-examples/ambari-blueprints_field-notes.ipynb', 'repo_name': 'Virtuant/hadoop-admin-2019', 'size': '12455'}