Concourse CI on Kubernetes (GKE), Part 4: Concourse

In our previous post, we configured our GKE (Google Kubernetes Engine) to use Let’s Encrypt TLS certificates. In this post, the capstone of our series, we install Concourse CI.

Installation

These instructions are a more-opinionated version of the canonical instructions for the Concourse CI Helm chart found here: https://github.com/concourse/concourse-chart.

First Install: with Helm

We use helm to install Concourse. We first add the Helm repo, and then install it. We take the opportunity to bump the default login time from 24 hours to ten days (duration=240h) because we hate re-authenticating to our Concourse every morning. Replace gke.nono.io with your DNS record:

kubectl delete ingress kuard # to free up https://gke.nono.io
helm repo add concourse https://concourse-charts.storage.googleapis.com/
helm install gke-nono-io concourse/concourse \
  --set concourse.web.externalUrl=https://gke.nono.io \
  --set concourse.web.auth.duration=240h \
  --set 'web.ingress.enabled=true' \
  --set 'web.ingress.annotations.cert-manager\.io/issuer=letsencrypt-prod' \
  --set 'web.ingress.annotations.kubernetes\.io/ingress.class=nginx' \
  --set 'web.ingress.hosts={gke.nono.io}' \
  --set 'web.ingress.tls[0].hosts[0]=gke.nono.io' \
  --set 'web.ingress.tls[0].secretName=gke.nono.io' \
  \
  --wait

Browse to our site https://gke.nono.io. You’ll see a secure connection icon & the initial Concourse CI login page.

First Upgrade: Locking Down Concourse

Our Concourse is insecure: we haven’t changed the default private keys. Our Concourse is public-facing, and we must change the keys lest evildoers compromise us. The Concourse README warns:

For your convenience, this chart provides some default values for secrets, but it is recommended that you generate and manage these secrets outside the Helm chart.

Let’s make our keys. The Concourse documentation provides two excellent ways to do it, and we’ve modified one of the ways to suit our setup. Replace gke.nono.io with your DNS record:

mkdir -p secrets/
for KEY in session_signing_key tsa_host_key worker_key; do
  ssh-keygen -t rsa -b 4096 -m PEM -f secrets/$KEY -C $KEY < /dev/null
done
rm secrets/session_signing_key.pub # "You can remove the session_signing_key.pub file if you have one, it is not needed by any process in Concourse"

Let’s re-deploy our Concourse with our newly-generated secrets. Replace gke.nono.io with your DNS record:

helm upgrade gke-nono-io concourse/concourse \
  --set concourse.web.externalUrl=https://gke.nono.io \
  --set concourse.web.auth.duration=240h \
  --set 'web.ingress.enabled=true' \
  --set 'web.ingress.annotations.cert-manager\.io/issuer=letsencrypt-prod' \
  --set 'web.ingress.annotations.kubernetes\.io/ingress.class=nginx' \
  --set 'web.ingress.hosts={gke.nono.io}' \
  --set 'web.ingress.tls[0].hosts[0]=gke.nono.io' \
  --set 'web.ingress.tls[0].secretName=gke.nono.io' \
  \
  --set-file secrets.sessionSigningKey=secrets/session_signing_key \
  --set-file secrets.hostKey=secrets/tsa_host_key \
  --set-file secrets.hostKeyPub=secrets/tsa_host_key.pub \
  --set-file secrets.workerKey=secrets/worker_key \
  --set-file secrets.workerKeyPub=secrets/worker_key.pub \
  \
  --wait

Third Upgrade: now with GitHub OAuth

We have a Concourse CI server, and we’ve generated our own keys, but we’re still not secure: people can log in with the user “test” using the password “test”. Yes, really.

We don’t want to have any hard-coded users; we want to authenticate against our GitHub organization, “blabbertabber”, so we browse to our organization (https://github.com/blabbertabber) → Settings → Developer Settings → OAuth Apps → New OAuth App.

Note: “Note that the client must be created under an organization if you want to authorize users based on organization/team membership.”

Here’s how we filled out ours. Replace gke.nono.io with your URL. The authorization callback URL is particularly important; don’t mess it up:

We click “Register Application”, which brings us to the next screen, where we get the Client ID (5e4ffee9dfdced62ebe3) and then click “Generate a new client secret” to get the Client secret (549e10b1680ead9cafa30d4c9a715681cec9b074). Don’t forget to click “Update Application”!

Now we can add the five GitHub OAuth-related lines to our helm upgrade command. Replace the GitHub org blabbertabber, the GitHub Client ID and Client Secret with the ones you’ve created, gke.nono.io with your DNS record:

While we’re locking things down, we also remove the local user “test” (along with the easy-to-guess password, “test”). We do this by setting secrets.localUsers to “”. To be safe, we also disable local auth (we set concourse.web.localAuth.enabled to false).

helm upgrade gke-nono-io concourse/concourse \
  --set concourse.web.externalUrl=https://gke.nono.io \
  --set concourse.web.auth.duration=240h \
  --set 'web.ingress.enabled=true' \
  --set 'web.ingress.annotations.cert-manager\.io/issuer=letsencrypt-prod' \
  --set 'web.ingress.annotations.kubernetes\.io/ingress.class=nginx' \
  --set 'web.ingress.hosts={gke.nono.io}' \
  --set 'web.ingress.tls[0].hosts[0]=gke.nono.io' \
  --set 'web.ingress.tls[0].secretName=gke.nono.io' \
  \
  --set-file secrets.sessionSigningKey=secrets/session_signing_key \
  --set-file secrets.hostKey=secrets/tsa_host_key \
  --set-file secrets.hostKeyPub=secrets/tsa_host_key.pub \
  --set-file secrets.workerKey=secrets/worker_key \
  --set-file secrets.workerKeyPub=secrets/worker_key.pub \
  \
  --set secrets.localUsers="" \
  --set concourse.web.localAuth.enabled=false \
  --set concourse.web.auth.mainTeam.github.org=blabbertabber \
  --set concourse.web.auth.github.enabled=true \
  --set secrets.githubClientId=5e4ffee9dfdced62ebe3 \
  --set secrets.githubClientSecret=549e10b1680ead9cafa30d4c9a715681cec9b074 \
  \
  --wait

Browse to our URL: https://gke.nono.io. We log in with GitHub Auth. We authorize our app. We download & install our fly CLI. Then we log in. Replace gke.nono.io with your DNS record:

fly -t gke login -c https://gke.nono.io
 # click the link
 # click "Authorize blabbertabber"
 # see "login successful!"

We create the following simple pipeline file, simple.yml:

jobs:
- name: simple
  plan:
  - task: simple
    config:
      platform: linux
      image_resource:
        type: docker-image
        source:
          repository: fedora
      run:
        path: "true"

Let’s fly our new pipeline:

fly -t gke set-pipeline -p simple -c simple.yml
fly -t gke expose-pipeline -p simple
fly -t gke unpause-pipeline -p simple

We browse to our Concourse and see the sweet green of success (it’ll take a minute or two to run):

Yay! We’re done.

Pro-tip

Rather than having an onerous number of --set arguments to our helm upgrade command, we find it easier to modify the corresponding settings in the values.yml file and pass it to our invocation of helm, i.e. helm upgrade -f values.yml .... Here’s our file of overrides.

Addendum: Keeping Concourse Up-to-date

Blindly upgrading Concourse without reading the release notes is a recipe for disaster; however, that’s what we’re going to show you. Let’s update the Helm repos first.

helm repo update

Now let’s upgrade our install. Replace gke.nono.io with your DNS record:

helm upgrade gke-nono-io concourse/concourse \
  --set concourse.web.externalUrl=https://gke.nono.io \
  --set concourse.web.auth.duration=240h \
  --set 'web.ingress.enabled=true' \
  --set 'web.ingress.annotations.cert-manager\.io/issuer=letsencrypt-prod' \
  --set 'web.ingress.annotations.kubernetes\.io/ingress.class=nginx' \
  --set 'web.ingress.hosts={gke.nono.io}' \
  --set 'web.ingress.tls[0].hosts[0]=gke.nono.io' \
  --set 'web.ingress.tls[0].secretName=gke.nono.io' \
  \
  --set-file secrets.sessionSigningKey=secrets/session_signing_key \
  --set-file secrets.hostKey=secrets/tsa_host_key \
  --set-file secrets.hostKeyPub=secrets/tsa_host_key.pub \
  --set-file secrets.workerKey=secrets/worker_key \
  --set-file secrets.workerKeyPub=secrets/worker_key.pub \
  \
  --set secrets.localUsers="" \
  --set concourse.web.localAuth.enabled=false \
  --set concourse.web.auth.mainTeam.github.org=blabbertabber \
  --set concourse.web.auth.github.enabled=true \
  --set secrets.githubClientId=5e4ffee9dfdced62ebe3 \
  --set secrets.githubClientSecret=549e10b1680ead9cafa30d4c9a715681cec9b074 \
  \
  --wait

Browse to your Concourse server, and check that it has the updated version number.

References

Concourse CI Helm chart: https://github.com/concourse/concourse-chart
Helm Chart Install: Advanced Usage of the “Set” Argument: https://itnext.io/helm-chart-install-advanced-usage-of-the-set-argument-3e214b69c87a
Creating an OAuth App on GitHub: https://docs.github.com/en/developers/apps/building-oauth-apps/creating-an-oauth-app

Updates/Errata

2021-11-13 Added section on keeping Concourse up-to-date.

2021-11-14 Added section on locking down Concourse.

Installation#

First Install: with Helm#

First Upgrade: Locking Down Concourse#

Third Upgrade: now with GitHub OAuth#

Pro-tip#

Addendum: Keeping Concourse Up-to-date#

References#

Updates/Errata#