How to build HA Mesosphere DC/OS Cluster

By Bikram Singh / Jun 30,2018

Overview

This article will be part one of Mesosphere DC/OS series. In this part I will cover the installation of Mesosphere DC/OS cluster and making sure we have a DC/OS cluster up and running in a High available mode. We will be using Mesosphere DC/OS community edition. Focus of this article is to learn the different components and how these work together during installation so we will be using advanced CLI mode to install DC/OS, this way we will have complete freedom to use any configuration parameter during installation which is not possible using GUI. If you are new to Mesosphere DC/OS I highly recommend you go through the DC/OS documentation to understand how it works and its architecture. I will not be covering any concepts or architecture in this blog. Mesosphere DC/OS overview and architecture

Cluster Components

Mesosphere DC/OS community edition v1.10
Centos 7.3_x86_64 – Base OS
Zookeeper v3.4.10

Prerequisites

To complete this article, you will need below Infrastructure:

CentOS x86_64_7.3.1611-minimal for all the DC/OS nodes.
1x Bootstrap node – This node will be used to install DC/OS
3x Mesos masters
2x Mesos private agents – You can add more if needed
1x Mesos private agents – You can add more if needed
3x Zookepeer nodes
Nodes can be Phyical or virtual. I am using them as VMs
A load balancer to balance the following TCP ports to all master nodes: 80, 443, 8080, 8181, 2181, 5050 – I am using openstack Neutron Loadbalancer
Enable password less SSH to all the nodes from bootstrap node. Copy SSH copy from bootstrap node to all the other nodes
Firewall must be disabled on all the nodes
NTP must be enabled on all the nodes
Docker must be installed on all the nodes
Disable sudo password prompts

Software prerequisites

DC/OS Software dependencies

yum -y --tolerant install perl tar xz unzip curl bind-utils net-tools ipset libtool-ltdl rsync nfs-utils

Disable SELINUX

echo ">>> Disabling SELinux"
sed -i 's/SELINUX=enforcing/SELINUX=permissive/g' /etc/selinux/config
setenforce permissive

For deatiled prerequites please visit – DC/OS system requirements

Mesosphere DC/OS Master nodes

mesos-master-1.cloud.itp-inc.com -- 172.16.10.15
mesos-master-2.cloud.itp-inc.com -- 172.16.10.18
mesos-master-3.cloud.itp-inc.com -- 172.16.10.26

Mesosphere DC/OS private agent nodes

mesos-agent-1.cloud.itp-inc.com -- 172.16.10.19
mesos-agent-2.cloud.itp-inc.com -- 172.16.10.44

Mesosphere DC/OS public agent nodes

mesos-zk-1.cloud.itp-inc.com -- 172.16.10.15
mesos-zk-2.cloud.itp-inc.com -- 172.16.10.18
mesos-zk-3.cloud.itp-inc.com -- 172.16.10.26

Mesosphere DC/OS Architecture

DC/OS is a platform for running distributed containerized software, like apps, jobs, and services. As a platform, DC/OS is distinct from and agnostic to the infrastructure layer. This means that the infrastructure may consist of virtual or physical hardware as long as it provides compute, storage, and networking.

How to build HA Mesosphere DC/OS Cluster

Software Layer
At the software layer, DC/OS provides package management and a package repository to easily install and manage multiple types of services: databases, message queues, stream processors, artifact repositories, monitoring solutions, continuous integration tools, source control management, log aggregators, etc. In addition to these packaged apps and services, the user may install their own custom apps, services, and scheduled jobs.

Platform Layer
At the platform layer there are dozens of components grouped into the following categories:

Cluster Management
Container Orchestration
Container Runtimes
Logging and Metrics
Networking
Package Management
IAM and Security (Available with DC/OS ENTERPRISE edition)
Storage

These components are divided across multiple node types:

Master Nodes
Private Agent Nodes
Public Agent Nodes

Infrastructure Layer
At the infrastructure layer, DC/OS can be installed on public clouds, private clouds, or on-premises hardware. Some of these install targets have automated provisioning tools, but almost any infrastructure can be used, as long as it includes multiple x86 machines on a shared IPv4 network.

Source : DC/OS Architecture

DC/OS Cluster Architecture

Below is the high level architecture of our cluster which we are going to build. We will be using 3 mesos masters and 3 agent nodes. We will use “exhibitor_storage_backend” as zookeper and for zookeeper will build a serate Zookeeper cluster outside of mesos, which needs to be build before we start DC/OS installation. I have build the complete mesos cluster on top of openstack.

How to build HA Mesosphere DC/OS Cluster

Note: I have created a openstack neutron load balancer to load-balance the following TCP ports to all mesos master nodes: 80, 443, 8080, 8181, 2181, 5050.

Configure the DC/OS cluster

Configure Zookeeper cluster

Run below commands on all the 3 zookeeper nodes

yum install wget -y
yum install java-1.8.0-openjdk -y
wget http://apache.org/dist/zookeeper/current/zookeeper-3.4.10.tar.gz
tar xvzf zookeeper-3.4.10.tar.gz
useradd zookeeper
cp -r zookeeper-3.4.10/ /opt
chown -R zookeeper:zookeeper /opt/zookeeper-3.4.10/
ln -s /opt/zookeeper-3.4.10 /opt/zookeeper
chown -h zookeeper:zookeeper /opt/zookeeper
mkdir -p /var/lib/zookeeper
chown -R zookeeper:zookeeper /var/lib/zookeeper
cd /opt/zookeeper/conf
cp zoo_sample.cfg zoo.cfg
mkdir -p /var/lib/zookeeper/data
echo 1 > /var/lib/zookeeper/data/myid-->change value to 2&3 on respectively
chown zookeeper:zookeeper /var/lib/zookeeper/data

cat > /etc/systemd/system/zookeeper.service <<EOF
[Unit]
Description=Apache Zookeeper server 
Documentation=http://zookeeper.apache.org
Requires=network.target remote-fs.target 
After=network.target remote-fs.target

[Service]
Type=forking
User=zookeeper
Group=zookeeper
ExecStart=/opt/zookeeper/bin/zkServer.sh start
ExecStop=/opt/zookeeper/bin/zkServer.sh stop
ExecReload=/opt/zookeeper/bin/zkServer.sh restart
WorkingDirectory=/var/lib/zookeeper

[Install]
WantedBy=multi-user.target
EOF

Update data directory and servers in zoo.cfg. Add below to
/opt/zookeeper/conf/zoo.cfg

server.1=zk-1.scaleon.io:2888:3888
server.2=zk-2.scaleon.io:2888:3888
server.3=zk-3.scaleon.io:2888:3888
autopurge.snapRetainCount=10
autopurge.purgeInterval =12

Start the services on all 3 nodes

systemctl daemon-reload
systemctl start zookeeper.service
systemctl status zookeeper.service
systemctl enable zookeeper.service

Configure mesos bootstrap node
Create a directory named genconf on your bootstrap node and within that directory create an ip-detect scipt as below

mkdir  genconf
cd genconf

cat ip-detect
#!/usr/bin/env bash
set -o nounset -o errexit
export PATH=/usr/sbin:/usr/bin:$PATH
echo $(ip addr show eth0 | grep -Eo '[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}' | head -1)

Configure a config.yaml configuration file in getconf directory with below configuration. You can modify your configuration file based on what options you need . My cluster configuration includes DC/OS cluster with three masters, an Exhibitor/ZooKeeper backed by ZooKeeper, masters that have an HTTP load balancer in front of them, one public agent node, 2 private agents, DNS servers and two virtual networks . You can refer below link for sample configurations

DC/OS sample configurations
Configuration refrence

[root@mesos-bootstrap genconf]# cat config.yaml 
---
agent_list:
- 172.16.10.19
- 172.16.10.44
bootstrap_url: file:///opt/dcos_install_tmp
cluster_name: mesos_dev_cluster
# DC/OS Enterprise Only
#customer_key: <customer-key>
exhibitor_storage_backend: zookeeper
exhibitor_zk_hosts: mesos-zk-1:2181,  mesos-zk-2:2181,  mesos-zk-3:2181
exhibitor_zk_path: /var/lib/zookeeper/data
log_directory: ./genconf/logs
ip_detect_public_filename: ./genconf/ip-detect
master_discovery: master_http_loadbalancer
master_list:
- 172.16.10.15
- 172.16.10.18
- 172.16.10.26
num_masters: 3
public_agent_list:
- 172.16.10.28
exhibitor_address: 172.16.10.94
process_timeout: 1800
resolvers:
- 10.9.60.20
- 10.9.60.21
ssh_key_path: ./genconf/id_rsa
ssh_port: '22'
ssh_user: root
dcos_overlay_enable: true
dcos_overlay_mtu: 8000
dcos_overlay_config_attempts: 6
dcos_overlay_network:
  vtep_subnet: 10.0.1.0/24
  vtep_mac_oui: 70:B3:D5:00:00:00
  overlays:
    - name: dcos
      subnet: 10.2.0.0/16
      prefix: 24
    - name: dcos-1
      subnet: 10.3.0.0/16
      prefix: 24
[root@mesos-bootstrap genconf]#

Default process_timeout is 120 increase it to higher value if your installation fails . I have kept it to 1800 secs.

Download the DC/OS installer to your root directory.

[root@mesos-bootstrap ]# curl -O https://downloads.dcos.io/dcos/stable/dcos_generate_config.sh
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  813M  100  813M    0     0  22.2M      0  0:00:36  0:00:36 --:--:-- 23.6M
[root@mesos-bootstrap ]#

Run the DC/OS installer shell script on your bootstrap node to generate a customized DC/OS build using config.yaml file.

[root@mesos-bootstrap ~]# sudo bash dcos_generate_config.sh --genconf
Extracting image from this script and loading into docker daemon, this step can take a few minutes
dcos-genconf.2d45a8f9e277a60007-5c14d12c3c8eecebcd.tar
e53f74215d12: Loading layer  5.06 MB/5.06 MB
e2060b517069: Loading layer 23.09 MB/23.09 MB
4725c710f46f: Loading layer 4.075 MB/4.075 MB
6d2ecb782005: Loading layer 167.9 MB/167.9 MB
6f9254a986e9: Loading layer  2.56 kB/2.56 kB
389f0375642c: Loading layer 24.48 MB/24.48 MB
9e412189cae6: Loading layer 628.4 MB/628.4 MB
e405c76f6591: Loading layer 4.096 kB/4.096 kB
39cad7f44f92: Loading layer 3.072 kB/3.072 kB
c595cb0256ae: Loading layer 6.656 kB/6.656 kB
e5ba695bce1b: Loading layer 1.536 kB/1.536 kB
Loaded image: mesosphere/dcos-genconf:2d45a8f9e277a60007-5c14d12c3c8eecebcd
====> EXECUTING CONFIGURATION GENERATION
Generating configuration files...
Package filename: packages/dcos-config/dcos-config--setup_b0f13aa0ffd564212a68220e31a5695c3b5c5ba3.tar.xz
Package filename: packages/dcos-metadata/dcos-metadata--setup_b0f13aa0ffd564212a68220e31a5695c3b5c5ba3.tar.xz
Generating Bash configuration files for DC/OS
[root@mesos-bootstrap ~]#

Run a preflight script to validate that your cluster is installable.

[root@mesos-bootstrap ~]# sudo bash dcos_generate_config.sh --preflight
====> EXECUTING_PREFLIGHT
====> START run_preflight
====> STAGE preflight
====> STAGE preflight_cleanup
====> STAGE preflight
====> STAGE preflight
====> STAGE preflight
====> STAGE preflight
====> STAGE preflight
====> STAGE preflight_cleanup
====> STAGE preflight_cleanup
====> STAGE preflight_cleanup
====> STAGE preflight_cleanup
====> STAGE preflight_cleanup
====> OUTPUT FOR run_preflight
====> 172.16.10.15:22 COMMAND PASSED
     TASK:
/usr/bin/ssh -oConnectTimeout=10 -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oBatchMode=yes -oPasswordAuthentication=no -p22 -i ./genconf/id_rsa -tt [email protected] sudo mkdir -p /opt/dcos_install_tmp
     STDERR:
     STDOUT:
====> 172.16.10.15:22 COMMAND PASSED
     TASK:
/usr/bin/ssh -oConnectTimeout=10 -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oBatchMode=yes -oPasswordAuthentication=no -p22 -i ./genconf/id_rsa -tt [email protected] sudo chown root /opt/dcos_install_tmp
     STDERR:
     STDOUT:
====> 172.16.10.15:22 COMMAND PASSED
     TASK:
/usr/bin/scp -oConnectTimeout=10 -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oBatchMode=yes -oPasswordAuthentication=no -P22 -i ./genconf/id_rsa genconf/serve/dcos_install.sh [email protected]:/opt/dcos_install_tmp
     STDERR:
     STDOUT:
====> 172.16.10.15:22 COMMAND PASSED
     TASK:
/usr/bin/ssh -oConnectTimeout=10 -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oBatchMode=yes -oPasswordAuthentication=no -p22 -i ./genconf/id_rsa -tt [email protected] sudo bash /opt/dcos_install_tmp/dcos_install.sh --preflight-only master
     STDERR:
     STDOUT:
====> 172.16.10.15:22 COMMAND PASSED
     TASK:
/usr/bin/ssh -oConnectTimeout=10 -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oBatchMode=yes -oPasswordAuthentication=no -p22 -i ./genconf/id_rsa -tt [email protected] sudo rm -rf /opt/dcos_install_tmp
     STDERR:
     STDOUT:
====> 172.16.10.18:22 COMMAND PASSED
     TASK:
/usr/bin/ssh -oConnectTimeout=10 -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oBatchMode=yes -oPasswordAuthentication=no -p22 -i ./genconf/id_rsa -tt [email protected] sudo mkdir -p /opt/dcos_install_tmp
     STDERR:
     STDOUT:
====> 172.16.10.18:22 COMMAND PASSED
     TASK:
/usr/bin/ssh -oConnectTimeout=10 -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oBatchMode=yes -oPasswordAuthentication=no -p22 -i ./genconf/id_rsa -tt [email protected] sudo chown root /opt/dcos_install_tmp
     STDERR:
     STDOUT:
====> 172.16.10.18:22 COMMAND PASSED
     TASK:
/usr/bin/scp -oConnectTimeout=10 -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oBatchMode=yes -oPasswordAuthentication=no -P22 -i ./genconf/id_rsa genconf/serve/dcos_install.sh [email protected]:/opt/dcos_install_tmp
     STDERR:
     STDOUT:
====> 172.16.10.18:22 COMMAND PASSED
     TASK:
/usr/bin/ssh -oConnectTimeout=10 -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oBatchMode=yes -oPasswordAuthentication=no -p22 -i ./genconf/id_rsa -tt [email protected] sudo bash /opt/dcos_install_tmp/dcos_install.sh --preflight-only master
     STDERR:
     STDOUT:
====> 172.16.10.18:22 COMMAND PASSED
     TASK:
/usr/bin/ssh -oConnectTimeout=10 -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oBatchMode=yes -oPasswordAuthentication=no -p22 -i ./genconf/id_rsa -tt [email protected] sudo rm -rf /opt/dcos_install_tmp
     STDERR:
     STDOUT:
====> 172.16.10.26:22 COMMAND PASSED
     TASK:
/usr/bin/ssh -oConnectTimeout=10 -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oBatchMode=yes -oPasswordAuthentication=no -p22 -i ./genconf/id_rsa -tt [email protected] sudo mkdir -p /opt/dcos_install_tmp
     STDERR:
     STDOUT:
====> 172.16.10.26:22 COMMAND PASSED
     TASK:
/usr/bin/ssh -oConnectTimeout=10 -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oBatchMode=yes -oPasswordAuthentication=no -p22 -i ./genconf/id_rsa -tt [email protected] sudo chown root /opt/dcos_install_tmp
     STDERR:
     STDOUT:
====> 172.16.10.26:22 COMMAND PASSED
     TASK:
/usr/bin/scp -oConnectTimeout=10 -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oBatchMode=yes -oPasswordAuthentication=no -P22 -i ./genconf/id_rsa genconf/serve/dcos_install.sh [email protected]:/opt/dcos_install_tmp
     STDERR:
     STDOUT:
====> 172.16.10.26:22 COMMAND PASSED
     TASK:
/usr/bin/ssh -oConnectTimeout=10 -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oBatchMode=yes -oPasswordAuthentication=no -p22 -i ./genconf/id_rsa -tt [email protected] sudo bash /opt/dcos_install_tmp/dcos_install.sh --preflight-only master
     STDERR:
     STDOUT:
====> 172.16.10.26:22 COMMAND PASSED
     TASK:
/usr/bin/ssh -oConnectTimeout=10 -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oBatchMode=yes -oPasswordAuthentication=no -p22 -i ./genconf/id_rsa -tt [email protected] sudo rm -rf /opt/dcos_install_tmp
     STDERR:
     STDOUT:
====> 172.16.10.19:22 COMMAND PASSED
     TASK:
/usr/bin/ssh -oConnectTimeout=10 -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oBatchMode=yes -oPasswordAuthentication=no -p22 -i ./genconf/id_rsa -tt [email protected] sudo mkdir -p /opt/dcos_install_tmp
     STDERR:
     STDOUT:
====> 172.16.10.19:22 COMMAND PASSED
     TASK:
/usr/bin/ssh -oConnectTimeout=10 -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oBatchMode=yes -oPasswordAuthentication=no -p22 -i ./genconf/id_rsa -tt [email protected] sudo chown root /opt/dcos_install_tmp
     STDERR:
     STDOUT:
====> 172.16.10.19:22 COMMAND PASSED
     TASK:
/usr/bin/scp -oConnectTimeout=10 -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oBatchMode=yes -oPasswordAuthentication=no -P22 -i ./genconf/id_rsa genconf/serve/dcos_install.sh [email protected]:/opt/dcos_install_tmp
     STDERR:
     STDOUT:
====> 172.16.10.19:22 COMMAND PASSED
     TASK:
/usr/bin/ssh -oConnectTimeout=10 -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oBatchMode=yes -oPasswordAuthentication=no -p22 -i ./genconf/id_rsa -tt [email protected] sudo bash /opt/dcos_install_tmp/dcos_install.sh --preflight-only master
     STDERR:
     STDOUT:
====> 172.16.10.19:22 COMMAND PASSED
     TASK:
/usr/bin/ssh -oConnectTimeout=10 -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oBatchMode=yes -oPasswordAuthentication=no -p22 -i ./genconf/id_rsa -tt [email protected] sudo rm -rf /opt/dcos_install_tmp
     STDERR:
     STDOUT:
====> 172.16.10.44:22 COMMAND PASSED
     TASK:
/usr/bin/ssh -oConnectTimeout=10 -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oBatchMode=yes -oPasswordAuthentication=no -p22 -i ./genconf/id_rsa -tt [email protected] sudo mkdir -p /opt/dcos_install_tmp
     STDERR:
     STDOUT:
====> 172.16.10.44:22 COMMAND PASSED
     TASK:
/usr/bin/ssh -oConnectTimeout=10 -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oBatchMode=yes -oPasswordAuthentication=no -p22 -i ./genconf/id_rsa -tt [email protected] sudo chown root /opt/dcos_install_tmp
     STDERR:
     STDOUT:
====> 172.16.10.44:22 COMMAND PASSED
     TASK:
/usr/bin/scp -oConnectTimeout=10 -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oBatchMode=yes -oPasswordAuthentication=no -P22 -i ./genconf/id_rsa genconf/serve/dcos_install.sh [email protected]:/opt/dcos_install_tmp
     STDERR:
     STDOUT:
====> 172.16.10.44:22 COMMAND PASSED
     TASK:
/usr/bin/ssh -oConnectTimeout=10 -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oBatchMode=yes -oPasswordAuthentication=no -p22 -i ./genconf/id_rsa -tt [email protected] sudo bash /opt/dcos_install_tmp/dcos_install.sh --preflight-only master
     STDERR:
     STDOUT:
====> 172.16.10.44:22 COMMAND PASSED
     TASK:
/usr/bin/ssh -oConnectTimeout=10 -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oBatchMode=yes -oPasswordAuthentication=no -p22 -i ./genconf/id_rsa -tt [email protected] sudo rm -rf /opt/dcos_install_tmp
     STDERR:
     STDOUT:
====> 172.16.10.28:22 COMMAND PASSED
     TASK:
/usr/bin/ssh -oConnectTimeout=10 -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oBatchMode=yes -oPasswordAuthentication=no -p22 -i ./genconf/id_rsa -tt [email protected] sudo mkdir -p /opt/dcos_install_tmp
     STDERR:
     STDOUT:
====> 172.16.10.28:22 COMMAND PASSED
     TASK:
/usr/bin/ssh -oConnectTimeout=10 -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oBatchMode=yes -oPasswordAuthentication=no -p22 -i ./genconf/id_rsa -tt [email protected] sudo chown root /opt/dcos_install_tmp
     STDERR:
     STDOUT:
====> 172.16.10.28:22 COMMAND PASSED
     TASK:
/usr/bin/scp -oConnectTimeout=10 -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oBatchMode=yes -oPasswordAuthentication=no -P22 -i ./genconf/id_rsa genconf/serve/dcos_install.sh [email protected]:/opt/dcos_install_tmp
     STDERR:
     STDOUT:
====> 172.16.10.28:22 COMMAND PASSED
     TASK:
/usr/bin/ssh -oConnectTimeout=10 -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oBatchMode=yes -oPasswordAuthentication=no -p22 -i ./genconf/id_rsa -tt [email protected] sudo bash /opt/dcos_install_tmp/dcos_install.sh --preflight-only master
     STDERR:
     STDOUT:
====> 172.16.10.28:22 COMMAND PASSED
     TASK:
/usr/bin/ssh -oConnectTimeout=10 -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oBatchMode=yes -oPasswordAuthentication=no -p22 -i ./genconf/id_rsa -tt [email protected] sudo rm -rf /opt/dcos_install_tmp
     STDERR:
     STDOUT:
====> ACTION run_preflight COMPLETE
====> SUMMARY FOR run_preflight
6 out of 6 hosts successfully completed run_preflight stage.
====> END OF SUMMARY FOR run_preflight

Install DC/OS on your cluster

[root@mesos-bootstrap ~]# sudo bash dcos_generate_config.sh --deploy
<---snip---->
====> 172.16.10.44:22 COMMAND PASSED
     TASK:
/usr/bin/scp -oConnectTimeout=10 -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oBatchMode=yes -oPasswordAuthentication=no -P22 -i ./genconf/id_rsa genconf/serve/packages/dcos-checks/dcos-checks--0178ae7d3846212dd59ad2b0d174116bccb4c38d.tar.xz [email protected]:/opt/dcos_install_tmp/packages/dcos-checks
     STDERR:
     STDOUT:
====> 172.16.10.44:22 COMMAND PASSED
     TASK:
/usr/bin/ssh -oConnectTimeout=10 -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oBatchMode=yes -oPasswordAuthentication=no -p22 -i ./genconf/id_rsa -tt [email protected] mkdir -p /opt/dcos_install_tmp/packages/adminrouter
     STDERR:
     STDOUT:
====> 172.16.10.44:22 COMMAND PASSED
     TASK:
/usr/bin/scp -oConnectTimeout=10 -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oBatchMode=yes -oPasswordAuthentication=no -P22 -i ./genconf/id_rsa genconf/serve/packages/adminrouter/adminrouter--080df64df233dd63d6f1db8d11fe38a77107adab.tar.xz [email protected]:/opt/dcos_install_tmp/packages/adminrouter
     STDERR:
     STDOUT:
====> 172.16.10.44:22 COMMAND PASSED
     TASK:
/usr/bin/ssh -oConnectTimeout=10 -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oBatchMode=yes -oPasswordAuthentication=no -p22 -i ./genconf/id_rsa -tt [email protected] mkdir -p /opt/dcos_install_tmp/packages/dcos-integration-test
     STDERR:
     STDOUT:
====> 172.16.10.44:22 COMMAND PASSED
     TASK:
/usr/bin/scp -oConnectTimeout=10 -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oBatchMode=yes -oPasswordAuthentication=no -P22 -i ./genconf/id_rsa genconf/serve/packages/dcos-integration-test/dcos-integration-test--08d4ef8ef5f99f1c16149ef596ac342a081db5b9.tar.xz [email protected]:/opt/dcos_install_tmp/packages/dcos-integration-test
     STDERR:
     STDOUT:
====> 172.16.10.44:22 COMMAND PASSED
     TASK:
/usr/bin/ssh -oConnectTimeout=10 -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oBatchMode=yes -oPasswordAuthentication=no -p22 -i ./genconf/id_rsa -tt [email protected] mkdir -p /opt/dcos_install_tmp/packages/check-time
     STDERR:
     STDOUT:
====> 172.16.10.44:22 COMMAND PASSED
     TASK:
/usr/bin/scp -oConnectTimeout=10 -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oBatchMode=yes -oPasswordAuthentication=no -P22 -i ./genconf/id_rsa genconf/serve/packages/check-time/check-time--f1609ae2fff1ddedc1d32635dc40e6ab7d2c48b6.tar.xz [email protected]:/opt/dcos_install_tmp/packages/check-time
     STDERR:
     STDOUT:
====> 172.16.10.44:22 COMMAND PASSED
     TASK:
/usr/bin/ssh -oConnectTimeout=10 -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oBatchMode=yes -oPasswordAuthentication=no -p22 -i ./genconf/id_rsa -tt [email protected] mkdir -p /opt/dcos_install_tmp/packages/python
     STDERR:
     STDOUT:
====> 172.16.10.44:22 COMMAND PASSED
     TASK:
/usr/bin/scp -oConnectTimeout=10 -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oBatchMode=yes -oPasswordAuthentication=no -P22 -i ./genconf/id_rsa genconf/serve/packages/python/python--547c752ada17bf093048a6da3d50ded309446a96.tar.xz [email protected]:/opt/dcos_install_tmp/packages/python
     STDERR:
     STDOUT:
====> 172.16.10.44:22 COMMAND PASSED
     TASK:
/usr/bin/ssh -oConnectTimeout=10 -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oBatchMode=yes -oPasswordAuthentication=no -p22 -i ./genconf/id_rsa -tt [email protected] mkdir -p /opt/dcos_install_tmp/packages/toybox
     STDERR:
     STDOUT:
====> 172.16.10.44:22 COMMAND PASSED
     TASK:
/usr/bin/scp -oConnectTimeout=10 -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oBatchMode=yes -oPasswordAuthentication=no -P22 -i ./genconf/id_rsa genconf/serve/packages/toybox/toybox--0b4707cd828063b523dcc9abada857023d4d7c59.tar.xz [email protected]:/opt/dcos_install_tmp/packages/toybox
     STDERR:
     STDOUT:
====> 172.16.10.44:22 COMMAND PASSED
     TASK:
/usr/bin/ssh -oConnectTimeout=10 -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oBatchMode=yes -oPasswordAuthentication=no -p22 -i ./genconf/id_rsa -tt [email protected] mkdir -p /opt/dcos_install_tmp/packages/docker-gc
     STDERR:
     STDOUT:
====> 172.16.10.44:22 COMMAND PASSED
     TASK:
/usr/bin/scp -oConnectTimeout=10 -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oBatchMode=yes -oPasswordAuthentication=no -P22 -i ./genconf/id_rsa genconf/serve/packages/docker-gc/docker-gc--19fdaeacfc7544c32607a138cc434d00f65b68e0.tar.xz [email protected]:/opt/dcos_install_tmp/packages/docker-gc
     STDERR:
     STDOUT:
====> 172.16.10.44:22 COMMAND PASSED
     TASK:
/usr/bin/ssh -oConnectTimeout=10 -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oBatchMode=yes -oPasswordAuthentication=no -p22 -i ./genconf/id_rsa -tt [email protected] mkdir -p /opt/dcos_install_tmp/packages/dcos-test-utils
     STDERR:
     STDOUT:
====> 172.16.10.44:22 COMMAND PASSED
     TASK:
/usr/bin/scp -oConnectTimeout=10 -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oBatchMode=yes -oPasswordAuthentication=no -P22 -i ./genconf/id_rsa genconf/serve/packages/dcos-test-utils/dcos-test-utils--5f877c1e21b39f57d921069e2a727add3780b0b1.tar.xz [email protected]:/opt/dcos_install_tmp/packages/dcos-test-utils
     STDERR:
     STDOUT:
====> 172.16.10.44:22 COMMAND PASSED
     TASK:
/usr/bin/ssh -oConnectTimeout=10 -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oBatchMode=yes -oPasswordAuthentication=no -p22 -i ./genconf/id_rsa -tt [email protected] mkdir -p /opt/dcos_install_tmp/packages/octarine
     STDERR:
     STDOUT:
====> 172.16.10.44:22 COMMAND PASSED
     TASK:
/usr/bin/scp -oConnectTimeout=10 -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oBatchMode=yes -oPasswordAuthentication=no -P22 -i ./genconf/id_rsa genconf/serve/packages/octarine/octarine--4e37c062d2f145f9c2ce01d30dadf72c2aac5c4a.tar.xz [email protected]:/opt/dcos_install_tmp/packages/octarine
     STDERR:
     STDOUT:
====> 172.16.10.44:22 COMMAND PASSED
     TASK:
/usr/bin/ssh -oConnectTimeout=10 -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oBatchMode=yes -oPasswordAuthentication=no -p22 -i ./genconf/id_rsa -tt [email protected] mkdir -p /opt/dcos_install_tmp/packages/pytest
     STDERR:
     STDOUT:
====> 172.16.10.44:22 COMMAND PASSED
     TASK:
/usr/bin/scp -oConnectTimeout=10 -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oBatchMode=yes -oPasswordAuthentication=no -P22 -i ./genconf/id_rsa genconf/serve/packages/pytest/pytest--07a2c5a7f649f673e0fb05bf90cc5546e45736a3.tar.xz [email protected]:/opt/dcos_install_tmp/packages/pytest
     STDERR:
     STDOUT:
====> 172.16.10.44:22 COMMAND PASSED
     TASK:
/usr/bin/ssh -oConnectTimeout=10 -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oBatchMode=yes -oPasswordAuthentication=no -p22 -i ./genconf/id_rsa -tt [email protected] mkdir -p /opt/dcos_install_tmp/packages/mesos-dns
     STDERR:
     STDOUT:
====> 172.16.10.44:22 COMMAND PASSED
     TASK:
/usr/bin/scp -oConnectTimeout=10 -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oBatchMode=yes -oPasswordAuthentication=no -P22 -i ./genconf/id_rsa genconf/serve/packages/mesos-dns/mesos-dns--600da87080b7634f2380594499004a7ff0b34662.tar.xz [email protected]:/opt/dcos_install_tmp/packages/mesos-dns
     STDERR:
     STDOUT:
====> 172.16.10.44:22 COMMAND PASSED
     TASK:
/usr/bin/ssh -oConnectTimeout=10 -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oBatchMode=yes -oPasswordAuthentication=no -p22 -i ./genconf/id_rsa -tt [email protected] mkdir -p /opt/dcos_install_tmp/packages/java
     STDERR:
     STDOUT:
====> 172.16.10.44:22 COMMAND PASSED
     TASK:
/usr/bin/scp -oConnectTimeout=10 -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oBatchMode=yes -oPasswordAuthentication=no -P22 -i ./genconf/id_rsa genconf/serve/packages/java/java--a3ce082cda36449933b18301c97b5e287c184539.tar.xz [email protected]:/opt/dcos_install_tmp/packages/java
     STDERR:
     STDOUT:
====> 172.16.10.44:22 COMMAND PASSED
     TASK:
/usr/bin/ssh -oConnectTimeout=10 -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oBatchMode=yes -oPasswordAuthentication=no -p22 -i ./genconf/id_rsa -tt [email protected] mkdir -p /opt/dcos_install_tmp/packages/python-kazoo
     STDERR:
     STDOUT:
====> 172.16.10.44:22 COMMAND PASSED
     TASK:
/usr/bin/scp -oConnectTimeout=10 -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oBatchMode=yes -oPasswordAuthentication=no -P22 -i ./genconf/id_rsa genconf/serve/packages/python-kazoo/python-kazoo--8f466a7ed59b8b9843eacecedd19074a7d0c80a6.tar.xz [email protected]:/opt/dcos_install_tmp/packages/python-kazoo
     STDERR:
     STDOUT:
====> 172.16.10.44:22 COMMAND PASSED
     TASK:
/usr/bin/ssh -oConnectTimeout=10 -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oBatchMode=yes -oPasswordAuthentication=no -p22 -i ./genconf/id_rsa -tt [email protected] mkdir -p /opt/dcos_install_tmp/packages/rexray
     STDERR:
     STDOUT:
====> 172.16.10.44:22 COMMAND PASSED
     TASK:
/usr/bin/scp -oConnectTimeout=10 -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oBatchMode=yes -oPasswordAuthentication=no -P22 -i ./genconf/id_rsa genconf/serve/packages/rexray/rexray--da7f17f8a4b772c0bac3f8d289a08abd4ff272b4.tar.xz [email protected]:/opt/dcos_install_tmp/packages/rexray
     STDERR:
     STDOUT:
====> 172.16.10.44:22 COMMAND PASSED
     TASK:
/usr/bin/ssh -oConnectTimeout=10 -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oBatchMode=yes -oPasswordAuthentication=no -p22 -i ./genconf/id_rsa -tt [email protected] mkdir -p /opt/dcos_install_tmp/packages/python-gunicorn
     STDERR:
     STDOUT:
====> 172.16.10.44:22 COMMAND PASSED
     TASK:
/usr/bin/scp -oConnectTimeout=10 -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oBatchMode=yes -oPasswordAuthentication=no -P22 -i ./genconf/id_rsa genconf/serve/packages/python-gunicorn/python-gunicorn--a12bb65151fe101c049af7d6be158955b726e812.tar.xz [email protected]:/opt/dcos_install_tmp/packages/python-gunicorn
     STDERR:
     STDOUT:
====> 172.16.10.44:22 COMMAND PASSED
     TASK:
/usr/bin/ssh -oConnectTimeout=10 -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oBatchMode=yes -oPasswordAuthentication=no -p22 -i ./genconf/id_rsa -tt [email protected] mkdir -p /opt/dcos_install_tmp/packages/libsodium
     STDERR:
     STDOUT:
====> 172.16.10.44:22 COMMAND PASSED
     TASK:
/usr/bin/scp -oConnectTimeout=10 -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oBatchMode=yes -oPasswordAuthentication=no -P22 -i ./genconf/id_rsa genconf/serve/packages/libsodium/libsodium--6b9df3bf76df0789ddd944235399e0a0f7f041a1.tar.xz [email protected]:/opt/dcos_install_tmp/packages/libsodium
     STDERR:
     STDOUT:
====> 172.16.10.44:22 COMMAND PASSED
     TASK:
/usr/bin/ssh -oConnectTimeout=10 -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oBatchMode=yes -oPasswordAuthentication=no -p22 -i ./genconf/id_rsa -tt [email protected] mkdir -p /opt/dcos_install_tmp/packages/libffi
     STDERR:
     STDOUT:
====> 172.16.10.44:22 COMMAND PASSED
     TASK:
/usr/bin/scp -oConnectTimeout=10 -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oBatchMode=yes -oPasswordAuthentication=no -P22 -i ./genconf/id_rsa genconf/serve/packages/libffi/libffi--65417d4d857b8daa813a52749413ff825e6ce478.tar.xz [email protected]:/opt/dcos_install_tmp/packages/libffi
     STDERR:
     STDOUT:
====> 172.16.10.44:22 COMMAND PASSED
     TASK:
/usr/bin/ssh -oConnectTimeout=10 -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oBatchMode=yes -oPasswordAuthentication=no -p22 -i ./genconf/id_rsa -tt [email protected] mkdir -p /opt/dcos_install_tmp/packages/six
     STDERR:
     STDOUT:
====> 172.16.10.44:22 COMMAND PASSED
     TASK:
/usr/bin/scp -oConnectTimeout=10 -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oBatchMode=yes -oPasswordAuthentication=no -P22 -i ./genconf/id_rsa genconf/serve/packages/six/six--44e1bc27be58047605d14d18551576de50495be3.tar.xz [email protected]:/opt/dcos_install_tmp/packages/six
     STDERR:
     STDOUT:
====> 172.16.10.44:22 COMMAND PASSED
     TASK:
/usr/bin/ssh -oConnectTimeout=10 -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oBatchMode=yes -oPasswordAuthentication=no -p22 -i ./genconf/id_rsa -tt [email protected] mkdir -p /opt/dcos_install_tmp/packages/dnspython
     STDERR:
     STDOUT:
====> 172.16.10.44:22 COMMAND PASSED
     TASK:
/usr/bin/scp -oConnectTimeout=10 -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oBatchMode=yes -oPasswordAuthentication=no -P22 -i ./genconf/id_rsa genconf/serve/packages/dnspython/dnspython--d9e685d4cf3932a21f306717b96e8e63a2b5210a.tar.xz [email protected]:/opt/dcos_install_tmp/packages/dnspython
     STDERR:
     STDOUT:
====> 172.16.10.44:22 COMMAND PASSED
     TASK:
/usr/bin/ssh -oConnectTimeout=10 -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oBatchMode=yes -oPasswordAuthentication=no -p22 -i ./genconf/id_rsa -tt [email protected] mkdir -p /opt/dcos_install_tmp/packages/boost-libs
     STDERR:
     STDOUT:
====> 172.16.10.44:22 COMMAND PASSED
     TASK:
/usr/bin/scp -oConnectTimeout=10 -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oBatchMode=yes -oPasswordAuthentication=no -P22 -i ./genconf/id_rsa genconf/serve/packages/boost-libs/boost-libs--a04883bd55bb23fe577a8e60b01bfb0948c18c73.tar.xz [email protected]:/opt/dcos_install_tmp/packages/boost-libs
     STDERR:
     STDOUT:
====> 172.16.10.44:22 COMMAND PASSED
     TASK:
/usr/bin/ssh -oConnectTimeout=10 -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oBatchMode=yes -oPasswordAuthentication=no -p22 -i ./genconf/id_rsa -tt [email protected] mkdir -p /opt/dcos_install_tmp/packages/python-isodate
     STDERR:
     STDOUT:
====> 172.16.10.44:22 COMMAND PASSED
     TASK:
/usr/bin/scp -oConnectTimeout=10 -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oBatchMode=yes -oPasswordAuthentication=no -P22 -i ./genconf/id_rsa genconf/serve/packages/python-isodate/python-isodate--cc66ae2851ee0663200e92ece32525f95a1fb642.tar.xz [email protected]:/opt/dcos_install_tmp/packages/python-isodate
     STDERR:
     STDOUT:
====> 172.16.10.44:22 COMMAND PASSED
     TASK:
/usr/bin/ssh -oConnectTimeout=10 -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oBatchMode=yes -oPasswordAuthentication=no -p22 -i ./genconf/id_rsa -tt [email protected] mkdir -p /opt/dcos_install_tmp/packages/dcos-cni
     STDERR:
     STDOUT:
====> 172.16.10.44:22 COMMAND PASSED
     TASK:
/usr/bin/scp -oConnectTimeout=10 -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oBatchMode=yes -oPasswordAuthentication=no -P22 -i ./genconf/id_rsa genconf/serve/packages/dcos-cni/dcos-cni--cc00e84332454d6c3b3d94f4752a673ba0faac82.tar.xz [email protected]:/opt/dcos_install_tmp/packages/dcos-cni
     STDERR:
     STDOUT:
====> 172.16.10.44:22 COMMAND PASSED
     TASK:
/usr/bin/ssh -oConnectTimeout=10 -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oBatchMode=yes -oPasswordAuthentication=no -p22 -i ./genconf/id_rsa -tt [email protected] mkdir -p /opt/dcos_install_tmp/packages/dcos-metadata
     STDERR:
     STDOUT:
====> 172.16.10.44:22 COMMAND PASSED
     TASK:
/usr/bin/scp -oConnectTimeout=10 -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oBatchMode=yes -oPasswordAuthentication=no -P22 -i ./genconf/id_rsa genconf/serve/packages/dcos-metadata/dcos-metadata--setup_4368e0295c0502c45f91215713059a483c1b0db4.tar.xz [email protected]:/opt/dcos_install_tmp/packages/dcos-metadata
     STDERR:
     STDOUT:
====> 172.16.10.44:22 COMMAND PASSED
     TASK:
/usr/bin/ssh -oConnectTimeout=10 -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oBatchMode=yes -oPasswordAuthentication=no -p22 -i ./genconf/id_rsa -tt [email protected] mkdir -p /opt/dcos_install_tmp/packages/dcos-config
     STDERR:
     STDOUT:
====> 172.16.10.44:22 COMMAND PASSED
     TASK:
/usr/bin/scp -oConnectTimeout=10 -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oBatchMode=yes -oPasswordAuthentication=no -P22 -i ./genconf/id_rsa genconf/serve/packages/dcos-config/dcos-config--setup_4368e0295c0502c45f91215713059a483c1b0db4.tar.xz [email protected]:/opt/dcos_install_tmp/packages/dcos-config
     STDERR:
     STDOUT:
====> 172.16.10.44:22 COMMAND PASSED
     TASK:
/usr/bin/ssh -oConnectTimeout=10 -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oBatchMode=yes -oPasswordAuthentication=no -p22 -i ./genconf/id_rsa -tt [email protected] mkdir -p /opt/dcos_install_tmp/packages/libevent
     STDERR:
     STDOUT:
====> 172.16.10.44:22 COMMAND PASSED
     TASK:
/usr/bin/scp -oConnectTimeout=10 -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oBatchMode=yes -oPasswordAuthentication=no -P22 -i ./genconf/id_rsa genconf/serve/packages/libevent/libevent--b32e88ffa3b38113eb39282199c122406f89539f.tar.xz [email protected]:/opt/dcos_install_tmp/packages/libevent
     STDERR:
     STDOUT:
====> 172.16.10.44:22 COMMAND PASSED
     TASK:
/usr/bin/ssh -oConnectTimeout=10 -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oBatchMode=yes -oPasswordAuthentication=no -p22 -i ./genconf/id_rsa -tt [email protected] mkdir -p /opt/dcos_install_tmp/packages/python-jinja2
     STDERR:
     STDOUT:
====> 172.16.10.44:22 COMMAND PASSED
     TASK:
/usr/bin/scp -oConnectTimeout=10 -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oBatchMode=yes -oPasswordAuthentication=no -P22 -i ./genconf/id_rsa genconf/serve/packages/python-jinja2/python-jinja2--ba3a62d34792b22b70b30bbf9d84755c17404730.tar.xz [email protected]:/opt/dcos_install_tmp/packages/python-jinja2
     STDERR:
     STDOUT:
====> 172.16.10.44:22 COMMAND PASSED
     TASK:
/usr/bin/ssh -oConnectTimeout=10 -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oBatchMode=yes -oPasswordAuthentication=no -p22 -i ./genconf/id_rsa -tt [email protected] mkdir -p /opt/dcos_install_tmp/packages/dcos-history
     STDERR:
     STDOUT:
====> 172.16.10.44:22 COMMAND PASSED
     TASK:
/usr/bin/scp -oConnectTimeout=10 -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oBatchMode=yes -oPasswordAuthentication=no -P22 -i ./genconf/id_rsa genconf/serve/packages/dcos-history/dcos-history--40ed7feba4afb79a91c9aadef4f5af68051be915.tar.xz [email protected]:/opt/dcos_install_tmp/packages/dcos-history
     STDERR:
     STDOUT:
====> 172.16.10.44:22 COMMAND PASSED
     TASK:
/usr/bin/ssh -oConnectTimeout=10 -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oBatchMode=yes -oPasswordAuthentication=no -p22 -i ./genconf/id_rsa -tt [email protected] mkdir -p /opt/dcos_install_tmp/packages/logrotate
     STDERR:
     STDOUT:
====> 172.16.10.44:22 COMMAND PASSED
     TASK:
/usr/bin/scp -oConnectTimeout=10 -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oBatchMode=yes -oPasswordAuthentication=no -P22 -i ./genconf/id_rsa genconf/serve/packages/logrotate/logrotate--b91d427d87bb9a4248e9c71ce9749d168bc95525.tar.xz [email protected]:/opt/dcos_install_tmp/packages/logrotate
     STDERR:
     STDOUT:
====> 172.16.10.44:22 COMMAND PASSED
     TASK:
/usr/bin/ssh -oConnectTimeout=10 -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oBatchMode=yes -oPasswordAuthentication=no -p22 -i ./genconf/id_rsa -tt [email protected] mkdir -p /opt/dcos_install_tmp/packages/cosmos
     STDERR:
     STDOUT:
====> 172.16.10.44:22 COMMAND PASSED
     TASK:
/usr/bin/scp -oConnectTimeout=10 -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oBatchMode=yes -oPasswordAuthentication=no -P22 -i ./genconf/id_rsa genconf/serve/packages/cosmos/cosmos--adc8a4b3724fc89a4a91a26d09293f096af91709.tar.xz [email protected]:/opt/dcos_install_tmp/packages/cosmos
     STDERR:
     STDOUT:
====> 172.16.10.44:22 COMMAND PASSED
     TASK:
/usr/bin/ssh -oConnectTimeout=10 -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oBatchMode=yes -oPasswordAuthentication=no -p22 -i ./genconf/id_rsa -tt [email protected] mkdir -p /opt/dcos_install_tmp/packages/python-requests
     STDERR:
     STDOUT:
====> 172.16.10.44:22 COMMAND PASSED
     TASK:
/usr/bin/scp -oConnectTimeout=10 -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oBatchMode=yes -oPasswordAuthentication=no -P22 -i ./genconf/id_rsa genconf/serve/packages/python-requests/python-requests--021ebc337b23a42268efb7a8d408049698f53d63.tar.xz [email protected]:/opt/dcos_install_tmp/packages/python-requests
     STDERR:
     STDOUT:
====> 172.16.10.44:22 COMMAND PASSED
     TASK:
/usr/bin/ssh -oConnectTimeout=10 -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oBatchMode=yes -oPasswordAuthentication=no -p22 -i ./genconf/id_rsa -tt [email protected] mkdir -p /opt/dcos_install_tmp/packages/marathon
     STDERR:
     STDOUT:
====> 172.16.10.44:22 COMMAND PASSED
     TASK:
/usr/bin/scp -oConnectTimeout=10 -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oBatchMode=yes -oPasswordAuthentication=no -P22 -i ./genconf/id_rsa genconf/serve/packages/marathon/marathon--cbda0e186593ca79499bfc7c583e29129acd864e.tar.xz [email protected]:/opt/dcos_install_tmp/packages/marathon
     STDERR:
     STDOUT:
====> 172.16.10.44:22 COMMAND PASSED
     TASK:
/usr/bin/ssh -oConnectTimeout=10 -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oBatchMode=yes -oPasswordAuthentication=no -p22 -i ./genconf/id_rsa -tt [email protected] mkdir -p /opt/dcos_install_tmp/packages/dvdcli
     STDERR:
     STDOUT:
====> 172.16.10.44:22 COMMAND PASSED
     TASK:
/usr/bin/scp -oConnectTimeout=10 -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oBatchMode=yes -oPasswordAuthentication=no -P22 -i ./genconf/id_rsa genconf/serve/packages/dvdcli/dvdcli--ee85411e3cb9f0988ed54b5cc0789172b887f12f.tar.xz [email protected]:/opt/dcos_install_tmp/packages/dvdcli
     STDERR:
     STDOUT:
====> 172.16.10.44:22 COMMAND PASSED
     TASK:
/usr/bin/ssh -oConnectTimeout=10 -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oBatchMode=yes -oPasswordAuthentication=no -p22 -i ./genconf/id_rsa -tt [email protected] mkdir -p /opt/dcos_install_tmp/packages/cni
     STDERR:
     STDOUT:
====> 172.16.10.44:22 COMMAND PASSED
     TASK:
/usr/bin/scp -oConnectTimeout=10 -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oBatchMode=yes -oPasswordAuthentication=no -P22 -i ./genconf/id_rsa genconf/serve/packages/cni/cni--c3b4dbf04ffcb3c59ff69b62aeae6b1ddb2d2bae.tar.xz [email protected]:/opt/dcos_install_tmp/packages/cni
     STDERR:
     STDOUT:
====> 172.16.10.44:22 COMMAND PASSED
     TASK:
/usr/bin/ssh -oConnectTimeout=10 -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oBatchMode=yes -oPasswordAuthentication=no -p22 -i ./genconf/id_rsa -tt [email protected] mkdir -p /opt/dcos_install_tmp/packages/metronome
     STDERR:
     STDOUT:
====> 172.16.10.44:22 COMMAND PASSED
     TASK:
/usr/bin/scp -oConnectTimeout=10 -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oBatchMode=yes -oPasswordAuthentication=no -P22 -i ./genconf/id_rsa genconf/serve/packages/metronome/metronome--4751bc129503c0b7eed1effa122b7cef48acf93c.tar.xz [email protected]:/opt/dcos_install_tmp/packages/metronome
     STDERR:
     STDOUT:
====> 172.16.10.44:22 COMMAND PASSED
     TASK:
/usr/bin/ssh -oConnectTimeout=10 -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oBatchMode=yes -oPasswordAuthentication=no -p22 -i ./genconf/id_rsa -tt [email protected] mkdir -p /opt/dcos_install_tmp/packages/ncurses
     STDERR:
     STDOUT:
====> 172.16.10.44:22 COMMAND PASSED
     TASK:
/usr/bin/scp -oConnectTimeout=10 -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oBatchMode=yes -oPasswordAuthentication=no -P22 -i ./genconf/id_rsa genconf/serve/packages/ncurses/ncurses--2ff075e29c401fe632969613f471a29a6e5ea133.tar.xz [email protected]:/opt/dcos_install_tmp/packages/ncurses
     STDERR:
     STDOUT:
====> 172.16.10.44:22 COMMAND PASSED
     TASK:
/usr/bin/ssh -oConnectTimeout=10 -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oBatchMode=yes -oPasswordAuthentication=no -p22 -i ./genconf/id_rsa -tt [email protected] mkdir -p /opt/dcos_install_tmp/packages/python-pyyaml
     STDERR:
     STDOUT:
====> 172.16.10.44:22 COMMAND PASSED
     TASK:
/usr/bin/scp -oConnectTimeout=10 -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oBatchMode=yes -oPasswordAuthentication=no -P22 -i ./genconf/id_rsa genconf/serve/packages/python-pyyaml/python-pyyaml--63bb76db1b11b2cccf11a33d3b70a9a031b1131b.tar.xz [email protected]:/opt/dcos_install_tmp/packages/python-pyyaml
     STDERR:
     STDOUT:
====> 172.16.10.44:22 COMMAND PASSED
     TASK:
/usr/bin/ssh -oConnectTimeout=10 -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oBatchMode=yes -oPasswordAuthentication=no -p22 -i ./genconf/id_rsa -tt [email protected] mkdir -p /opt/dcos_install_tmp/packages/python-docopt
     STDERR:
     STDOUT:
====> 172.16.10.44:22 COMMAND PASSED
     TASK:
/usr/bin/scp -oConnectTimeout=10 -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oBatchMode=yes -oPasswordAuthentication=no -P22 -i ./genconf/id_rsa genconf/serve/packages/python-docopt/python-docopt--630f860f2dd27a719196221615c3843e83685fb0.tar.xz [email protected]:/opt/dcos_install_tmp/packages/python-docopt
     STDERR:
     STDOUT:
====> 172.16.10.44:22 COMMAND PASSED
     TASK:
/usr/bin/ssh -oConnectTimeout=10 -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oBatchMode=yes -oPasswordAuthentication=no -p22 -i ./genconf/id_rsa -tt [email protected] mkdir -p /opt/dcos_install_tmp/packages/avro-cpp
     STDERR:
     STDOUT:
====> 172.16.10.44:22 COMMAND PASSED
     TASK:
/usr/bin/scp -oConnectTimeout=10 -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oBatchMode=yes -oPasswordAuthentication=no -P22 -i ./genconf/id_rsa genconf/serve/packages/avro-cpp/avro-cpp--baefc2442ff8d5d7b88b4a47e80cb9c00a38cdc7.tar.xz [email protected]:/opt/dcos_install_tmp/packages/avro-cpp
     STDERR:
     STDOUT:
====> 172.16.10.44:22 COMMAND PASSED
     TASK:
/usr/bin/ssh -oConnectTimeout=10 -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oBatchMode=yes -oPasswordAuthentication=no -p22 -i ./genconf/id_rsa -tt [email protected] mkdir -p /opt/dcos_install_tmp/bootstrap
     STDERR:
     STDOUT:
====> 172.16.10.44:22 COMMAND PASSED
     TASK:
/usr/bin/scp -oConnectTimeout=10 -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oBatchMode=yes -oPasswordAuthentication=no -P22 -i ./genconf/id_rsa genconf/serve/bootstrap/5c14d12c3c8eecebcd9b4869bee6c8cf8214f08e.bootstrap.tar.xz [email protected]:/opt/dcos_install_tmp/bootstrap
     STDERR:
     STDOUT:
====> 172.16.10.44:22 COMMAND PASSED
     TASK:
/usr/bin/ssh -oConnectTimeout=10 -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oBatchMode=yes -oPasswordAuthentication=no -p22 -i ./genconf/id_rsa -tt [email protected] sudo bash /opt/dcos_install_tmp/dcos_install.sh slave
     STDERR:
     STDOUT:
====> 172.16.10.44:22 COMMAND PASSED
     TASK:
/usr/bin/ssh -oConnectTimeout=10 -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oBatchMode=yes -oPasswordAuthentication=no -p22 -i ./genconf/id_rsa -tt [email protected] sudo rm -rf /opt/dcos_install_tmp
     STDERR:
     STDOUT:
====> ACTION install_dcos COMPLETE
====> SUMMARY FOR install_dcos
6 out of 6 hosts successfully completed install_dcos stage.
====> END OF SUMMARY FOR install_dcos
[root@mesos-bootstrap ~]#

Run the DC/OS diagnostic script to verify that services are up and running.

[root@mesos-bootstrap ~]# sudo bash dcos_generate_config.sh --postflight
====> EXECUTING POSTFLIGHT
====> START run_postflight
====> STAGE postflight
====> STAGE postflight
====> STAGE postflight_cleanup
====> STAGE postflight_cleanup
====> STAGE postflight
====> STAGE postflight
====> STAGE postflight_cleanup
====> STAGE postflight
====> STAGE postflight
====> STAGE postflight_cleanup
====> STAGE postflight_cleanup
====> STAGE postflight_cleanup
====> STAGE cluster_postflight
====> OUTPUT FOR run_postflight
====> 172.16.10.28:22 CHECK PASSED
     Overall status: OK
     clock_sync: OK
     components_agent: OK
     curl: OK
     ip_detect_script: OK
     journald_dir_permissions: OK
     mesos_agent_registered_with_masters: OK
     tar: OK
     unzip: OK
     xz: OK
====> 172.16.10.28:22 COMMAND PASSED
     TASK:
/usr/bin/ssh -oConnectTimeout=10 -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oBatchMode=yes -oPasswordAuthentication=no -p22 -i ./genconf/id_rsa -tt [email protected] sudo rm -f /opt/dcos-prereqs.installed
     STDERR:
     STDOUT:
====> 172.16.10.19:22 CHECK PASSED
     Overall status: OK
     clock_sync: OK
     components_agent: OK
     curl: OK
     ip_detect_script: OK
     journald_dir_permissions: OK
     mesos_agent_registered_with_masters: OK
     tar: OK
     unzip: OK
     xz: OK
====> 172.16.10.19:22 COMMAND PASSED
     TASK:
/usr/bin/ssh -oConnectTimeout=10 -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oBatchMode=yes -oPasswordAuthentication=no -p22 -i ./genconf/id_rsa -tt [email protected] sudo rm -f /opt/dcos-prereqs.installed
     STDERR:
     STDOUT:
====> 172.16.10.44:22 CHECK PASSED
     Overall status: OK
     clock_sync: OK
     components_agent: OK
     curl: OK
     ip_detect_script: OK
     journald_dir_permissions: OK
     mesos_agent_registered_with_masters: OK
     tar: OK
     unzip: OK
     xz: OK
====> 172.16.10.44:22 COMMAND PASSED
     TASK:
/usr/bin/ssh -oConnectTimeout=10 -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oBatchMode=yes -oPasswordAuthentication=no -p22 -i ./genconf/id_rsa -tt [email protected] sudo rm -f /opt/dcos-prereqs.installed
     STDERR:
     STDOUT:
====> 172.16.10.15:22 CHECK PASSED
     Overall status: OK
     clock_sync: OK
     components_master: OK
     curl: OK
     ip_detect_script: OK
     journald_dir_permissions: OK
     mesos_master_replog_synchronized: OK
     tar: OK
     unzip: OK
     xz: OK
====> 172.16.10.15:22 COMMAND PASSED
     TASK:
/usr/bin/ssh -oConnectTimeout=10 -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oBatchMode=yes -oPasswordAuthentication=no -p22 -i ./genconf/id_rsa -tt [email protected] sudo rm -f /opt/dcos-prereqs.installed
     STDERR:
     STDOUT:
====> 172.16.10.18:22 CHECK PASSED
     Overall status: OK
     clock_sync: OK
     components_master: OK
     curl: OK
     ip_detect_script: OK
     journald_dir_permissions: OK
     mesos_master_replog_synchronized: OK
     tar: OK
     unzip: OK
     xz: OK
====> 172.16.10.18:22 COMMAND PASSED
     TASK:
/usr/bin/ssh -oConnectTimeout=10 -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oBatchMode=yes -oPasswordAuthentication=no -p22 -i ./genconf/id_rsa -tt [email protected] sudo rm -f /opt/dcos-prereqs.installed
     STDERR:
     STDOUT:
====> 172.16.10.26:22 CHECK PASSED
     Overall status: OK
     clock_sync: OK
     components_master: OK
     curl: OK
     ip_detect_script: OK
     journald_dir_permissions: OK
     mesos_master_replog_synchronized: OK
     tar: OK
     unzip: OK
     xz: OK
====> 172.16.10.26:22 COMMAND PASSED
     TASK:
/usr/bin/ssh -oConnectTimeout=10 -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oBatchMode=yes -oPasswordAuthentication=no -p22 -i ./genconf/id_rsa -tt [email protected] sudo rm -f /opt/dcos-prereqs.installed
     STDERR:
     STDOUT:
====> 172.16.10.28:22 CHECK PASSED
     Overall status: OK
====> ACTION run_postflight COMPLETE
====> SUMMARY FOR run_postflight
6 out of 6 hosts successfully completed run_postflight stage.
====> END OF SUMMARY FOR run_postflight
[root@mesos-bootstrap ~]#

Verification

Lets login to Mesos DC/OS GUI by going to the master or the loadbalancer VIP which is in front of the mesos masters.

http://mesos.itp-inc.com –> This points to VIP

How to build HA Mesosphere DC/OS Cluster

Install the dcos CLI. I have installed it on my latop using below install CLI option from the dashboard which gives a command to run on my latop

How to build HA Mesosphere DC/OS Cluster

[ -d /usr/local/bin ] || sudo mkdir -p /usr/local/bin && \ncurl https://downloads.dcos.io/binaries/cli/darwin/x86-64/dcos-1.10/dcos -o dcos && \nsudo mv dcos /usr/local/bin && \nsudo chmod +x /usr/local/bin/dcos && \ndcos cluster setup https://mesos.itp-inc.com && \ndcos

Lets install an application to test if our mesos cluster is up

** Install jenkins using cli**

bikramsingh@dev:~% dcos package install jenkins
By Deploying, you agree to the Terms and Conditions https://mesosphere.com/catalog-terms-conditions/#certified-services
WARNING: If you didn't provide a value for `storage.host-volume` (either using the CLI or via the Advanced Install dialog),
YOUR DATA WILL NOT BE SAVED IN ANY WAY.

Continue installing? [yes/no] yes
Installing Marathon app for package [jenkins] version [3.4.0-2.89.2]
bikramsingh@dev:~%

Lets verify in DCOS Dashboard

How to build HA Mesosphere DC/OS Cluster

Thanks.