Slipway is some automation to provision Spinnaker on a Kubernetes cluster with Drone CI.

Motivation

Our infrastructure requires that we have a production Kubernetes cluster with Spinnaker as our continuous delivery tooling and Drone as our continuous integration tooling. To set up this infrastructure from scratch is quite a slow and laborious process and one prone to mis-configuration. The cluster that we are running on will also need to be updated however performing in-place updates could very well leave our production system in a non-functional state. As such our upgrade process is to provision an entirely new stack of mirror infrastructure and when that is proved to be stable we cut over to that using DNS then decommission the old infrastructure.

Slipway: Automated Halyard and Spinnaker provisioning

Slipway automates one use-case only: provisioning Spinnaker on a Kubernetes cluster, controlled by an Ubuntu host VM running Halyard on Google Cloud Platform based on Halyard on GKE Quickstart guide.

Slipway also sets up Spinnaker to be publicly accessible with Oauth2 authentication and sets up some container registry repositories and a Slackbot. If this exact configuration is not what you want then the Spinnaker Setup docs might be a good starting point.

These scripts make a lot of decisions which might not fit your use-case. YMMV.

0. Preamble

Install the gcloud SDK then configure:

gcloud auth login

gcloud projects list

gcloud config set project $PROJECT_NAME

…where $PROJECT_NAME is the project you wish to provision infrastructure in.

1. Provision Kubernetes cluster

Get the Kubernetes provisioning script

curl -O  https://raw.githubusercontent.com/betandr/slipway/master/01_kube_up.sh

Run Kubernetes provisioning script

sh 01_kube_up.sh $INDEX $CLUSTER_NAME $IP_REGION $CLUSTER_ZONE

…where $INDEX is your index, such as 001, $CLUSTER_NAME is your cluster name such as foobarbaz, $IP_REGION is the region in which the external IPs for the Spinnaker external endpoints will be provisioned such as europe-west2, and $CLUSTER_ZONE is the zone you wish to provision in, such as europe-west2-c.

For these example values the following cluster will be provisioned:

• foobarbaz-cluster-001: 6 Nodes of 2 vCPUs in Zone europe-west2-c

The script will also reserve two IP addresses spinnaker-$INDEX and spinnaker-api-$INDEX which you should note for the Run Spinnaker provisioning script step later.

2. Provision Halyard Host VM

Get the Halyard host provisioning script

curl -O  https://raw.githubusercontent.com/betandr/slipway/master/02_hal_up.sh

Run Halyard host provisioning script

sh 02_hal_up.sh $INDEX

This script will ask you if you want to connect to the provisioned instance but also creates a shell script which can be used to connect to that instance later on.

To connect, run:

sh connect-to-halyard-host-$INDEX.sh

…where $INDEX matches your instance.

IMPORTANT: ALL FURTHER SCRIPTS ARE RUN FROM THE HALYARD HOST VM!

3. Provision Kubernetes Spinnaker and production clusters (from Halyard host VM):

Set up gcloud on Halyard VM

gcloud auth login

gcloud projects list

gcloud config set project $PROJECT_NAME

…where $PROJECT_NAME is the project you wish to provision infrastructure in.

Set up a DNS record for Spinnaker UI and Spinnaker API

Follow the Public Spinnaker on GKE) instructions to reserve two IP addresses then create DNS records for the Spinnaker API and UI load-balancers. The DNS records become OVERRIDE_API_URL and OVERRIDE_UI_URL and the IPs are used to configure Kubernetes in the Run Spinnaker provisioning script step.

Get the Spinnaker provisioning script

curl -O  https://raw.githubusercontent.com/betandr/slipway/master/03_spin_up.sh

Set the environment variables:

One method to do this is to have a local secrets file, containing something like:

export INDEX=000
export CLUSTER_NAME=hello-world
export IP_REGION=europe-west2
export CLUSTER_ZONE=europe-west2-c
export OAUTH2_DOMAIN=example.com
...etc...

…then load these using:

source secrets

Add your repos:

To ensure Spinnaker is set up with all the repos it needs, you should create a file called repos.txt and inside have repositories such as:

project1/container-a
project2/container-b

Run Spinnaker provisioning script

sh 03_spin_up.sh $INDEX $CLUSTER_NAME

…where $INDEX and $CLUSTER_NAME match the values specified in Run Kubernetes provisioning script in the 1. Provision Kubernetes clusters section.

Enter reserved external addresses

To enable external access to Spinnaker the Kubernetes service config will be updated. You will be prompted for the spinnaker-$INDEX and the spinnaker-api-$INDEX addresses from the 1. Provision Kubernetes clusters step.

04. Drone.io

Run Drone provisioning script

This script is based on Bo-Yi Wu’s drone-on-kubernetes scripts, used with thanks in accordance with the MIT licence.

The Drone script can provision a persistent disk instance which is used for the server database. You’ll probably only really want to do this once as you’ll probably want to keep the same server database between installs, otherwise you’ll need to set up the build tasks each time.

sh 04_drone_up.sh $ZONE

Decommissioning

This is not yet automated as it could potentially be dangerous, but things to clear up for the clusters and VM host are:

• Compute Engine: Halyard Host VM: halyard-host-$INDEX + Persistent Disk
• Kubernetes Engine: $CLUSTER_NAME-cluster-$INDEX
• Service Account: halyard-service-account-$INDEX@$PROJECT.iam.gserviceaccount.com
• Service Account: gcs-spin-service-account-$INDEX@$PROJECT.iam.gserviceaccount.com
• IAM Permissions: gcs-spin-service-account-$INDEX@$PROJECT.iam.gserviceaccount.com
• IAM Permissions: halyard-service-account-$INDEX@$PROJECT.iam.gserviceaccount.com
• VPC Network: spinnaker-$INDEX reserved external IP
• VPC Network: spinnaker-api-$INDEX reserved external IP
• VPC Network: target pools, forwarding rules

TODO

• Check quota before provisioning
• Check permissions before provisioning
• Get the reserved external IPs automatically via gcloud compute addresses list.