Saltstack 101 - Webhooks

05/04/2018

Control Saltstack remotely via HTTP(S) webhooks and automate your IT infrastructure…

Prerequisites

If you’ve followed the previous tutorial you will have a working salt master and minion machine which can successfully talk to each other. I highly recommend you review the previous tutorial before continuing.

While following this tutorial you have two choices:

  • Use a domain name with a proper (not self signed) SSL certificate (recommended approach).
  • Setup the system in unsecured HTTP mode (easier, but obviously less secure).

We will be using LetsEncrypt to issue a free SSL certificate and since domain names are cheap, I recommend you get one too.

There are a number of sites offering free domain names available. I’ve never used any of these services so I’m not willing to vouch for them.

Scenario

In this tutorial, we will:

  • Install an Apache server on the minion. This will mimic the customer’s webserver.
  • Create an HTTP endpoint on the salt master. This will act as a webhook to receive POST values.
  • Control the minion remotely via webhooks.
  • Demonstrate the webhook functionality by remotely starting and stopping the apache server.

Point Domain to Master

If using your own domain, make sure it’s pointing at your master now.

Install Customer Web Server

INSTALL CUSTOMER WEB SERVER

Install apache2 on the minion.

Remember this apache2 will not be served your domain, but will be accessible via the minion’s IP address.

From the master, run: salt 'saltstack-minion' pkg.install apache2

Check that it’s up and running. You should see the default Apache page.

Install Salt API on Master

The salt-api package allows remote connections to the salt master.

On the master run: sudo apt-get install salt-api -y

Ensure outputs of both salt-api --version and salt --version match (on the master):

root@saltstack-master:~# salt-api --version
  salt-api 2018.3.0 (Oxygen)
root@saltstack-master:~# salt --version
  salt 2018.3.0 (Oxygen)

Create REST User

This user will be used to authenticate while using the REST interface. On the master:

sudo useradd demouser
sudo passwd demouser
GRANT REST USER PERMISSIONS

Now give your demouser permissions to access all saltstack functions.

On the master: nano /etc/salt/master

Search for external_auth and uncomment the template to give your demouser permissions.

Your code should look like this:

external_auth:
  pam:
    demouser:
      - .*

Install CherryPy

This is a python webserver which will give us our webhook endpoints First, we need the python3-setuptools package as it’s used by cherrypy.

# Install pip and the setuptools package as it's used by cherrypy
sudo apt-get install python-pip python3-setuptools -y
# Now install cherrypy
sudo pip install cherrypy
# Make sure cherrypy is up to date
sudo pip install --upgrade cherrypy

Create SSL Certificate

If you’re using HTTP, skip this step. Install the LetsEncrypt certbot and generate your SSL certificates. Replace example.com and www.example.com with your domain names.

Run the following on the master:

sudo add-apt-repository ppa:certbot/certbot -y
sudo apt-get update
sudo apt-get install certbot -y
sudo certbot certonly --standalone --agree-tos --email me@example.com -n -d example.com -d www.example.com

Certs will be stored in:

/etc/letsencrypt/live/example.com/fullchain.pem
/etc/letsencrypt/live/example.com/privkey.pem

Add CherryPy Configuration to Master

Add the following configuration (altering as appropriate) to the end of your /etc/salt/master config file.

nano /etc/salt/master

Then add this:

rest_cherrypy:
  port: 8000
  # Uncomment following line only if you're running without a domain name and in HTTP mode.
  #disable_ssl: True
  # Uncomment following lines if using custom domain with LetsEncrypt
  #ssl_crt: /etc/letsencrypt/live/example.com/fullchain.pem
  #ssl_key: /etc/letsencrypt/live/example.com/privkey.pem

Restart Salt Master and Salt API

sudo service salt-master stop && sudo service salt-api stop
sudo service salt-master start && sudo service salt-api start

You’ve Come A Long Way Baby…

If you’re here you should be able to access the master on port 8000 (either via HTTPS with your domain or HTTP with an IP address). Either way, you should have no SSL certificate warnings.

You’ll get a 200 OK response and this text:

{"clients": ["local", "local_async", "local_batch", "local_subset", "runner", "runner_async", "ssh", "wheel", "wheel_async"], "return": "Welcome"}

Retrieve Your Auth Token

In order to use the REST endpoints, we need to authenticate. The portable authentication module (pam) makes this easy.

Login once with your username and password and you’ll receive a token

POST https://DOMAIN-OR-MASTER-IP:8000/login
Headers:
  Content-Type: application/json

Body:
{
  "username": "demouser",
  "password": "password",
  "eauth": "pam"
}

Copy the token value from the returned content and include it in the X-Auth-Token header.

Ping Minions Via REST

POST https://DOMAIN-OR-MASTER-IP:8000
Headers:
  X-Auth-Token: token value from above...

Request Body:
{
  "client": "local",
  "tgt": "*",
  "fun": "test.ping"
}

fun meaning function, not fun and frivolity :)

If successful, you should see this content returned:

{
"return": [
    {
        "saltstack-minion": true
    }
  ]
}

Congratulations. Your ping test was successful and you can now successfully control your infrastructure via webhooks and REST calls. How very DevOps of you!

YouTube Tutorial?

Would you prefer this post as a live walkthrough on Youtube?

Let me know by giving a thumbs-up (no login / account required).

Thanks. Your vote has been recorded.