How to set up Swarm cluster with Ansible? Based on Docker Engine > 1.12

In a previous post we set up a Swarm cluster and there was not easy, there was a lot of things to set up. Since the version 1.12, Docker Engine has a Swarm built-in orchestration tool. And this is changing everything: no more Consul node, no more TSL certificates to generate.

Clone the git project

Clone my ansible roles from Github:

and go to the project directory:

Configure ansible

Set up the hosts

Create the hosts file and open it:

Add to this file the hosts of your Swarm cluster and create the group swarm with all these hosts:

Ensure that you have root access to the hosts.

Set up the playbook

Create the main.yaml file and open it:

To your swarm group add the roles docker (to install docker engine) and docker-swarm (to configure swarm). You must provide the var docker_swarm_secret shared by the hosts, and the var docker_swarm_manager which is the I.P. address of the host you connect to manage the Swarm cluster:

Set up the host variables

Into the directory host_vars create a file with the name of the swarm manager host, here it is node1:

In this file add the variable docker_swarm_is_main_manager to indicate that our host is the Swarm manager:

Run ansible

Just run the command:

That’s it!

Manage the Swarm

Connect to your Swarm manager (you can connect to your host and manage the Swarm with another user if this user is in the group docker):

Create a service:

Listing the tasks of this service you see that 3 containers are running on different nodes:

Links

How to set up Swarm cluster with Ansible?

Edit 2016/07/05: This tutorial was written for Docker Swarm 1.11 or older. Now Docker Swarm is very easy to install and to deploy.

I wanted to learn more about Docker Swarm and I wanted to learn how to use Ansible, so naturally here I show you to quickly set up a Docker Swarm cluster using Ansible for deployment.

Requirements

Hosts

The hosts I use are all Debian 8.4 Jessie servers, dedicated server and VPS from kimsufi (a OVH brand), so if your hosts are of another linux distribution the Ansible scripts could not work.

Be sure that you have root access to your hosts.

Install Ansible

On your OSX system just run the following brew command on a terminal:

For the other operating systems, read more about how to install Ansible at the install Ansible documentation page.

Install Docker Engine

To install Docker Engine on your desktop computer follow the instructions at this page: install docker engine.

What is Swarm?

Briefly, Docker Swarm lets you manage many hosts with Docker Engine as you only have one host with Docker Engine.

Docker Swarm provides native clustering capabilities to turn a group of Docker engines into a single, virtual Docker Engine.

You can read more about Docker Swarm on the official website.

What is Ansible?

Briefly, Ansible is an orchestration tool that helps you deploy your infrastructure.

Ansible seamlessly unites workflow orchestration with configuration management, provisioning, and application deployment in one easy-to-use and deploy platform.

You can read more about Ansible on the official website.

Architecture of the cluster

The architecture is composed of Consul hosts (one or two hosts), Docker Swarm manager hosts (the number of your choice, three or more allows to set up a high availability cluster) and Docker Swarm node hosts (the number of your choice too).

The Ansible scripts here will set up one or two Consul hosts, but if you can set up a complete high availability Consul cluster this will be better.

In this tutorial we will set up a docker cluster with four different hosts: the first host discovery.example.com will host the Consul services, the host manager.example.com will have the role of the swarm manager (we will connect to it to manage the cluster), and finally the two last hosts node1.example.com and node2.example.com will run the containers of the cluster.

Clone the project

You can find the project here: https://github.com/cygy/ansible-docker-swarm-cluster

I will tell more details about it in this post, for now let’s clone the project and go to the project directory:

Prepare your hosts

Update the kernel version

As Docker Swarm documentation says, the kernel version of the Debian hosts must be at least 3.10. Read this post to upgrade the kernel of your Debian servers.

Docker requires a 64-bit installation regardless of your Debian version. Additionally, your kernel must be 3.10 at minimum. The latest 3.10 minor version or a newer maintained version are also acceptable.

Kernels older than 3.10 lack some of the features required to run Docker containers. These older versions are known to have bugs which cause data loss and frequently panic under certain conditions.

Be careful to not update to a kernel >= 4.0, Docker Swarm does not work with these versions.

 Generate the TLS certificates

To secure the communication between your swarm managers and your swarm nodes, it is recommended to use TLS. To use TLS you need certificates. Later I will check to use Letsencrypt, but for now we will generate self-signed certificates.

The certificates must be generated on a Linux system, so if you are on an OSX system, first copy the keys directory of the project on a linux host:

Then log in your host, and go to the keys directory:

Create your private key by running the script create_CA. You must provide a string passphrase when asked. The key and the certificate will be created under the directory keys/CA.

Now, for each of the hosts you must run the command create_keypair with the hostname of the host. The key and the certificate will be created under the directory hosts/[HOSTNAME].

In order to manage our Docker Swarm cluster later, we need to generate certificate for our home IP address too:

You can log out now:

You must get back the generated certificates and keys from your host to your local machine:

Check in the keys/hosts and keys/CA directories if all is here.

Set up the Ansible variables

Declare your hosts

Ansible needs to know what are your hosts. Open the hosts file and define the IP addresses of your hosts at the beginning of the file, the rest of the file defines the roles of the hosts:

Here seven hosts are declared, the format is:

Then under each role (the lines formatted as [role]) add the hosts which have this role:

  • The role [swarm-discovery-networks] will run Docker Engine and Consul: add one host here. The hosts with Docker Engine will register to this Consul host to manage the docker networking.
  • The role [swarm-discovery-nodes] will run Docker Engine and Consul: add one host here. It can be the same host under the role [swarm-discovery-networks]. The hosts with Docker Engine will register to this Consul host in order to be managed by the hosts with the [swarm-managers] role.
  • The role [swarm-managers] will run Docker Engine and Docker Swarm: add all the hosts you want: one host will be elected as the primary manager the others as secondary managers.
  • The role [swarm-nodes] will run Docker Engine and will be managed by the [swarm-managers] hosts: add all the hosts you want.

Don’t edit the other lines.

Set up the main variables

Open the file main.yaml. Here are defined many variables:

  • Under the part users_groups, you can define some user groups to add to your hosts. Set the name and gid of the groups.
  • Under the users group, you can define some users to add to your hosts. Set the name and uid of the users, aliases can also be defined. If a file named [username].keys containing a SSH key is present at the directory keys/users it will be copied to the host.
  • Set the value of the variable paranoid_allow_ping to 0 if you want that your hosts don’t respond to the ping requests.
  • Under the part paranoid_fail2ban you can define the settings of Fail2ban service.
  • You must update the variable node_discovery_host with the hostname (the same of the TLS certificates) of a node listed under the part swarm-discovery-nodes of the hosts file.
  • You must update the variable network_discovery_host with the hostname (the same of the TLS certificates) of a node listed under the part swarm-discovery-networks of the hosts file.
  • Finally, under the manager_ips part, add all the IP addresses of your hosts listed under the part swarm-managers of the hosts file.

You don’t have to edit the other variables.

Set up the hosts variables

Now, into the host_vars directory, create a file for each of your hosts. Name these files with the name of the variables of the host (i.e. here will be node1, node2discovery or manager).

Fill these files with the following content, setting the hostname of your hosts:

You can add here all the variables, defined at another place, that you want to redefine to this specific host.

Run Ansible: let’s the magic do!

Ensure that you are at the root of the project directory and run:

If there is no error, Ansible sets up your hosts as a Docker Swarm cluster.

Manage the Docker Swarm cluster

In order to manager the cluster we need to connect to a manager host defined in the  hosts file. To do the,  a secure connection is needed.

Remember a TLS certificate for our home IP address has been generated sooner. It is located at the directory keys/hosts/[HOME_IP_ADDRESS].

Copy the certificates at the same place:

Export environment variables to point your Docker Engine to a manager host of the cluster:

Now all your Docker Engine commands will point to your Docker Swarm cluster!

Here the result of the command docker info:

Here the result of the command docker ps -a:

Links

Update the kernel of your Debian server

To upgrade the kernel image of your linux server just follow these steps.

First search for the available images:

The list will be displayed:

Now install the headers and the image:

Once the image is installed grub must be updated, open the configuration file:

Search for the line GRUB_DEFAULT and ensure the value is equal to the index of the new kernel:

Update grub:

And reboot the server:

Its done!

How to know which ports of a server are opened?

As I forget always how to know which ports are opened on a server (on the local machine), I write down here the command to run:

This is the result: