Nomad and Consul Configuration Overview

There are several challenges to configuring Consul/Nomad environments. This page documents some of those issues and configurations I've managed to get working.

The largest challenge is configuring the networking to work correctly. Running services inside Docker containers requires different configuration options than running the client directly.

Additionally, there are specific configurations that are required to get Nomad working Docker for Mac.

This is an overview using Consul 0.7 and Nomad 0.5.

Consul configuration

Below is an overview of the Consul command line options used:

  • -bind - The address that should be bound to for internal cluster communications. This is an IP address that should be reachable by all other nodes in the cluster. By default, this is "0.0.0.0", meaning Consul will bind to all addresses on the local machine and will advertise the first available private IPv4 address to the rest of the cluster.
  • -advertise - The advertise address is used to change the address that we advertise to other nodes in the cluster. By default, the -bind address is advertised.
  • -client - The address to which Consul will bind client interfaces, including the HTTP, DNS, and RPC servers. By default, this is "127.0.0.1", allowing only loopback connections.

Configuration requirements

  • Consul needs to bind to the external adapter so that multi-node clustering works
  • Consul client must be available on localhost to support DNS binding
  • Consul client must be available on docker bridge adapter to support consul access by containers
  • Consul needs privileges to run on port 53 on Linux
  • Host needs to configure primary nameserver as localhost to support .consul resolution

With Binary

Configuring the binary is actually the easiest way to go because you don't have to deal with a layer of abstraction of Docker.

  • Bind to the external address
  • Advertise address uses the bind address automatically, which is what we want since it is bound to the external address
  • Client listens on all addresses
consul agent -server -bind=$host -client=0.0.0.0 -bootstrap_expect=1 -dns-port=53  

Or the equivalent with a config file:

{
  "server": true,
  "bootstrap_expect": 1,
  "data_dir": "/var/consul",
  "ui_dir": "/var/consul-ui",
  "ui": true,
  "ports": {
    "dns": 53
  },
  "client_addr": "0.0.0.0",
  "bind_addr": "$host"
}

With Docker

There are a few challenges to getting Consul running on Docker.

Previously host binding didn't work. This required running the container in bridged mode which creates a few challenges:

  • Container can't directly bind to the external adapter. Instead, you let bind occur on the docker bridge adapter (172.16.0.X)
  • Container client by default runs on loopback, you must allow the client on all network adapters
  • Container is bound to bridge adapter, so clusering doesn't work without advertising the address

The resulting docker command looks like:

docker run -d \  
-p 8300:8300 \
-p 8301:8301 \
-p 8301:8301/udp \
-p 8302:8302 \
-p 8302:8302/udp \
-p 8400:8400 \
-p 8500:8500 \
-p 53:8600 \
-p 53:8600/udp \
consul agent -server -ui -client=0.0.0.0 -advertise=$host -bootstrap-expect=1  

Most of those issues would likely be solved by running Consul in --net=host mode though I haven't gotten that working yet.

Consul issues encountered

These are few issues I've encountered along the way:

  • resolved - bug with previous version for dns request with response larger than udp packet size would renegotiate to tcp
  • resolved - bug with consul docker, could not bind to privileged port when --net=host mode enabled, works when using bridged network since docker is running with appropriate privileges

Nomad configuration

Below is an overview of the Nomad configuration file options used:

  • bind_addr (string: "0.0.0.0") - Specifies which address the Nomad agent should bind to for network services, including the HTTP interface as well as the internal gossip protocol and RPC mechanism.
  • client.network_interface (string: "lo | lo0") - Specifies the name of the interface to force network fingerprinting on. This defaults to the loopback interface.
  • consul.address (string: "127.0.0.1:8500") - Specifies the address to the local Consul agent, given in the format host:port.
  • advertise (Advertise: see below) - Specifies the advertise address for individual network services. This can be used to advertise a different address to the peers of a server or a client node to support more complex network configurations such as NAT. This configuration is optional, and defaults to the bind address of the specific network service if it is not provided. Any values configured in this stanza take precedence over the default bind_addr. If the bind address is 0.0.0.0 then the hostname is advertised.

Configuration requirements

  • Nomad needs to bind to the external adapter so that multi-node clustering works
  • Nomad client must be available on docker bridge adapter to support access by containers
  • Nomad needs to specify the external adapter for binding containers
  • Nomad must specify the external adapter for connection to Consul, this assumes Consul is available on the external adapter.

As a result a Nomad configuration looks like:

datacenter = "dc1"

# bind to $host instead of default (0.0.0.0 prod, 127.0.0.1 dev)
bind_addr = "$host"

# use /tmp to support Docker for Mac
data_dir = "/tmp"

# use external adapter as the network_interface
client {  
  enabled = true
  network_interface = "$adapter"
  options {
    "driver.raw_exec.enable" = "1"
  }
}

# single server that is self-bootstrapped
server {  
  enabled = true
  bootstrap_expect = 1
}

# use the external adapter since consul should be there
consul {  
  address = "$host:8500"
}

Nomad issues

  • resolved: Docker for Mac unsupported until Nomad 5... released last week
  • workaround: Docker for Mac requires configuring the data_dir as /tmp, other paths result in errors when lauching containers
  • nomad consul stanza checksuseadvertise is not being respected, requires running bind_addr as the external adapter
comments powered by Disqus