Securely using secrets in a pipeline — HashiCorp Vault + JWT Auth

The typical way of communicating with the Vault service is adding the VAULT_TOKEN value as a constant in the environment, But is it a safe solution? Of course not! There is another way which is more reliable and secure. In this article, I’m just trying to explain the main concept and not dive into the details on each step because If you get the concept, after this, it’s up to you how to configure the auth method, put these pieces together, or how to make up your environment to use it.

But before getting involved with all aspects, Let’s see the high-level design of the flow:

Prerequisites:

• HashiCorp Vault (If you want to try and looking for a way other than installing on a local machine, I recommend registering in the HashiCorp cloud and receive 50$ free credits, And after that, creating your own Vault cluster EASY PEASY! 😋 )
• 🤔 Hmm… That’s that! xD

Let’s do this!

First of all, we need to set the root (!) token to theVAULT_TOKEN env value because we will use some root-level commands. As a best practice, use tokens with the appropriate set of policies based on your role in the organization.

Enable key/value v1-v2 secrets engine at secrets/if it’s not enabled already.

> vault secrets enable -version=2 -path=secrets kv
#Or > vault secrets enable -version=1 -path=secrets kv

We need to enable the jwt auth method in Vault. The jwt auth method can be used to authenticate with Vault using OIDC or by providing a JWT. JWT signatures will be verified against public keys from the issuer. This process can be done in three different ways, though only one method may be configured for a single backend:

1. Static Keys
2. JWKS
3. OIDC Discovery

> vault auth enable jwt

If you received Permission Denied, probably need to set the VAULT_NAMESPACE value.

Let’s add some test K/Vs to the Vault.

> vault kv put secrets/services/payment/mysecret value=1t3s3cr3t
> vault kv put secrets/providers/slack token=1t3s3cr3t1t3s3cr3t
> vault kv put secrets/providers/github token=1t3sxxx3t1xxx3cr3t
> vault kv put secrets/gcp/gke/production/payment/deploy_sa token='{}'
> vault kv put secrets/services/user/his value=1t3s3cr3t

If you are curious about that how to protect the GCP service account, I recommend reading “How to generate short-lived GCP Service Account Keys, OAuth2 tokens with Vault” post.

Now, If you run this command, you should receive the same result. For sure, If your Vault service was freshly installed like me, otherwise It doesn’t matter; leave this step.

> vault kv list secret/services
Keys
----
payment/
user/

Policy! Everything in Vault is path-based, and policies are no exception. Policies provide a declarative way to grant or forbid access to certain paths and operations in Vault. Now we need to create a policy with the specific conditions for the payment service.

All jobs in our case require the Slack and GitHub secret; therefore, it would be better to create a single policy and use it when we would define a role.

Put the following policies into the file.

cicd-policy.hcl :

path "secrets/providers/slack" {
  capabilities = [ "read" ]
}
path "secrets/providers/github" {
  capabilities = [ "read" ]
}

payment-deploy-policy.hcl :

# This allows the user to read "secrets/services/payment/stripe"
# with a parameter named "token" and can contain any value.
path "secrets/services/payment/mysecret" {
  capabilities = [ "read" ]
}
path "secrets/gcp/gke/production/payment/deploy_sa" {
  capabilities = [ "read" ]
}

Apply the policies with the following command:

> vault policy write payment-deploy-policy payment-deploy-policy.hcl
> vault policy write cicd-policy cicd-policy.hcl

Create another file to write a role for JWT auth. payment-deploy-role.json

{
  "role_type": "jwt",
  "policies": ["cicd-policy", "payment-deploy-policy"],
  "token_explicit_max_ttl": 30,
  "user_claim": "sub",
  "bound_claims": {
	"service_name": "payment",
	"branch": "main"
  },
  "claim_mappings": {}
}

We set the max TTL to 30 seconds, and ONLY a valid JWT that contains the same service_name and branch has access to this role. Depends on your use case, you can modify these values. The only claim configuration a role requires is user_claim. After authentication is known to work, you can add additional claims bindings and metadata copying. A role may also be configured to check arbitrary claims through the bound_claims map. The map contains a set of claims and their required values. For the advanced technique, I suggest you work with claim_mappings(Link)

Through this command, The JWT role will be created:

> cat payment-deploy-role.json | vault write auth/jwt/role/payment-deploy-role -

Alright, now it’s time to add the Static Key(s). I mentioned the available methods above, but the easiest way is using static keys. If you can provide JWKS URL, I suggest using it, or the third option is providing the OIDC Discovery method.

Use these command to generate the RS256 key (Don’t enter the passphrase):

> ssh-keygen -t rsa -b 4096 -m PEM -f jwtRS256.key
...
+---[RSA 4096]----+
...
> openssl rsa -in jwtRS256.key -pubout -outform PEM -out 
jwtRS256.key.pub
writing RSA key

After this, You must have public (jwtRS256.key.pub) and private (jwtRS256.key) keys.

So, we should introduce the public (jwtRS256.key.pub) key to the Vault JWT config:

> vault write auth/jwt/config jwt_supported_algs=RS256 jwt_validation_pubkeys=@jwtRS256.key.pub

Now it’s time to create a specific JWT for the payment service. (For test purposes, you can use this website: Click!). After that, we need to put the generated JWT as a key value in the project's environment or secure context in your provider (CircleCI, Gitlab, etc.).

Services like Gitlab each job has JSON Web Token (JWT) provided as CI/CD variable named CI_JOB_JWT and also provide a JWKS URL (A JSON Web Key Set (JWKS) URL is configured. Keys will be fetched from this endpoint during authentication). Therefore, in that case, no need to set static key(s) or define a JWT by ourselves per project or service.

Decoded payload:

{
  "iss": "myprovider.com",
  "sub": "amirso",
  "aud": "vault",
  "iat": 1620515613,
  "exp": 1652044413,
  "service_name": "payment",
  "branch": "main"
}

🔥 We are almost settled! Now we can, through the below command, get a short-lived token from the Vault.

export VAULT_TOKEN=$(vault write -field=token auth/jwt/login role=payment-deploy-role jwt=...GENERATED_TOKEN...)

What about the deployment? One way is using the above command in your deployment to GET and SET the VAULT_TOKEN.

When you run this command, VAULT_TOKEN will be updated with a new token (TTL=30s). Let's try to read some secrets! 🕵️‍♂️

> vault kv get -field=value secrets/services/payment/mysecret
1t3s3cr3t
> vault kv get -field=token secrets/providers/slack
1t3s3cr3t1t3s3cr3t
> vault kv get -field=token secrets/providers/github
1t3sxxx3t1xxx3cr3t
> vault kv get -field=token secrets/gcp/gke/production/payment/deploy_sa
{}
> vault kv get secrets/services/user/his
...
* permission denied

And after 30 seconds, none of the above requests will work 🐞.

It was easy. Right? Certainly, modifying many other configurations can help you achieve more security, but this one better than nothing 😉 . With this information, you can define an architecture and steps for your CI/CD flows (Access Level). It certainly depends on your environment, project, and resources. Vault is mighty, and you have access to the variant features, which included low-level configuration, auth, secret plugins, etc. While powerful, it can also be very complex.

//Thanks for reading 🙏