Skip to main content

Continuous deployment to a local k3d cluster

The repository deploys a suite the same way a released product would: an operator converges a BlongDeployment, a release workflow builds the artifact, and the operator's gateway is the only door the workflow opens. realm/blong-sport is the worked example; the mechanism is generic.

The whole path, from the merged commit to a pod that mounts the suite's version volume:

From a merged commit to a running suiteOpen full screen

The three halves​

  • Operator — one per cluster, installed once, reconciling every tenant's CR; in realm/blong-kustomize.
  • Artifact — the rush deploy scenario, zipped and fetched by the fill Job from a URL; the release asset.
  • Trigger — the token-gated kustomize.release.approve route on the operator's gateway, at the address the installer names in kustomize.deploy.operator.ingress.host.

The suite's tree never carries the operator: it carries the CR, and the bindings the operator needs in the suite's namespaces. The operator's own install tree carries the CRD, its RBAC, the BLONG_RELEASE_TOKEN reference and the ingress. The two are applied from system/operator/local and the suite's system/kustomize/local respectively.

.github/deployments.json​

The deployment map is the only place a suite is named. A key is the alias a human dispatches, and its fields are what the workflow reads:

{
"sport": {
"release": "blong-sport",
"package": "@feasibleone/blong-sport",
"packagePath": "realm/blong-sport",
"scenario": "sport",
"artifact": "sport.zip",
"namespace": "blong-sport",
"suite": "blong-sport",
"entry": "./index.ts",
"profile": "monolith",
"follows": [
"blong",
"blong-access",
"blong-browser",
"blong-core",
"blong-gateway",
"blong-gogo",
"blong-kustomize",
"blong-lib",
"blong-login",
"blong-mock",
"blong-party",
"blong-server"
]
}
}

release is the release-please component, so the release tag is <release>-v<version>. scenario names both common/config/rush/deploy-<scenario>.json and the kustomize deploy scenario; artifact is the name the zip is published under; suite and namespace are the CR's name and tenant. A second suite is a second key plus its scenario file — the workflow does not change.

follows is the list of release-please components whose release re-deploys this deployment, and it is what makes a batch reach the suites it changed: a release of a realm inside a suite publishes no release of the suite, so without the list the running suite keeps the version it already had. Each list is the deployment's production dependency closure — what the artifact carries, restricted to the components release-please publishes — plus core/blong-gogo, whose version keys the framework image, and core/blong-browser, which the portal bundle is built from. The deployment's own release is always a trigger and is not repeated in the list, and a name that no component answers fails the run instead of waiting for a release that will never come.

The two tenant deployments are sport (one process, the monolith profile) and suite — the framework's own full-stack suite, deployed with the default realm profile, so each realm that serves something is a process of its own:

{
"suite": {
"release": "blong-suite",
"package": "@feasibleone/blong-suite",
"packagePath": "suite/blong-suite",
"scenario": "suite",
"artifact": "suite.zip",
"namespace": "blong-suite",
"suite": "blong-suite",
"entry": "./index.ts",
"profile": "realm",
"follows": [
"blong",
"blong-access",
"blong-browser",
"blong-chain",
"blong-core",
"blong-gogo",
"blong-kustomize",
"blong-lib",
"blong-login",
"blong-marine",
"blong-mock",
"blong-server",
"marine-data"
]
}
}

profile is documentation, not input: the split is the CR's spec.profile, which the operator re-derives the plan from, and realm is what plan.ts falls back to when a CR names none.

One entry is not a tenant's suite at all: the deployment realm's own install, marked "installer": true, which resolve skips when it looks for the alias a tag names. It is there for the other readers of the map — the workflow job that publishes the operator's artifact, and realm/blong-kustomize/deploy.sh, which resolves a released artifact's URL from the same record rather than hardcoding a prefix in a shell script:

{
"operator": {
"installer": true,
"release": "blong-kustomize",
"package": "@feasibleone/blong-kustomize",
"packagePath": "realm/blong-kustomize",
"scenario": "operator",
"artifact": "blong-kustomize.zip"
}
}

No address is in the map: a host belongs to the machine that deploys, so the operator's comes from .blong_k8src and the suite's portal host from the same file — see where an address comes from.

.github/workflows/deploy.yaml​

The workflow has one trigger that matters for releases and one for people:

  • workflow_run of the Release workflow, completed, on main — a release-please batch. The releases that run published are the batch: release-please creates them inside it, so the run's own window bounds them, and every release of one batch is tagged at the same commit.
  • workflow_dispatch with deployment and tag — a re-deploy, or a deployment whose release was published earlier.

A batch that names nothing a deployment follows is not a failure: resolve plans an empty matrix and the jobs are skipped. Because a batch is one commit, the newest tag of it builds every artifact — a blong-gogo release and a blong-marine release of the same batch carry the same tree.

resolve turns the batch into a matrix: one entry per deployment whose follows list names a component in the batch, plus the operator when blong-kustomize is one of them. The framework version is read from .release-please-manifest.json at the batch's tag (core/blong-gogo), and the job waits for ghcr.io/feasibleone/blong-gogo:v<gogoVersion> to exist — a release is not a promise that its image is pullable yet.

artifact is one job per matrix entry, named and serialised by its alias: it checks out the batch's tag, restores the pnpm store CI cached, installs, builds what the deploy scenario carries, runs rush deploy --scenario <scenario>, zips the target folder, hashes it, attaches it to the release with gh release upload, and asks the operator to converge — in the same leg, so the digest it reports is the bytes it just uploaded. The URL the operator receives is the release asset URL of the batch's tag.

The deployment's own version travels with that call, read from the manifest at the tag rather than from the tag's suffix: a batch triggered by blong-gogo deploys suite 1.19.0 with framework v1.51.0, and the operator shows both, so a reader can tell what actually moved.

operator-artifact is the same work for the installer rather than for a tenant: it answers a batch that contains a blong-kustomize-v<version> release and attaches three assets — blong-kustomize-<version>.zip, its .sha256, and blong-kustomize.zip as a stable alias for releases/latest/download/, which is the name the install tree falls back to. The versioned name is what a version installs and the next release cannot overwrite; the digest is an asset rather than a sentence in the release notes because it is a script that reads it, and what it names is the directory the artifact lands in.

The same job then deploys what it published, and the caller is the operator's own release route: suite: blong-operator tells the operator the announcement is about its own install tree, so instead of patching a BlongDeployment — which it does not have — it runs the registry pass that plans that tree, with the release's version and artifact laid over its own configuration. The pass applies the fill Jobs for the new artifact identity first and waits for them, then the Deployment that runs the operator, so an artifact nothing can fetch leaves the running process where it was. The install-owned kinds — the namespace, the CRD, the ServiceAccount and the RBAC objects — are skipped by name, so a release can never widen the rights the operator runs under; a tag that changes one of them still needs npm run deploy from a checkout (or kubectl apply -k system/operator/local).

The call builds the JSON-RPC body with jq -n --arg — never shell interpolation, because the token and the URL are data — and posts it to secrets.DEPLOY_ENDPOINT with secrets.DEPLOY_TOKEN. The route is plain JSON-RPC over HTTPS: the gate is the token, not a bearer or an MLE handshake, and the ingress certificate protects it in transit.

# The route the operator's ingress serves, and the token the operator reads from its Secret.
DEPLOY_ENDPOINT=https://<operator host>/rpc/kustomize/release/approve
DEPLOY_TOKEN=<the same value as the operator's blong-operator-release secret>

A wrong or missing token is answered 401 with kustomize.release.tokenMismatch or kustomize.release.tokenMissing; the CR is untouched, because the check happens before the notify handler. A correct token patches the CR's version, suiteVolume.artifact (URL and digest) and, when the release moved it, minFrameworkVersion, and the operator converges.

The answer is a report as well as a verdict: report carries the suite, the version it moved to — and from, when the operator could read the previous value — the fields the seam applied and, for the operator's own release, the pass counts; markdown is those facts as one document. The workflow appends that document to the job summary, and a chat notification can post it unchanged, so the two never describe one release differently.

Where an address comes from​

A host is a DNS record and a certificate somebody owns, so it is configuration rather than a value in a source tree: no realm, workflow or committed manifest names one. The framework reads the file of the k8s intent — .blong_k8src, found from the working directory upwards — and its kustomize.deploy block supplies the addresses of both halves:

kustomize:
deploy:
operator:
ingress:
host: <the operator's host>
tls: {secretName: blong-operator-tls}
annotations:
cert-manager.io/cluster-issuer: letsencrypt-prod
portal:
web:
host: <the suite's portal host>
annotations:
cert-manager.io/cluster-issuer: letsencrypt-prod

The file is ignored (.blong_*) and local to the machine. The repository's copy names the operator's address, which every k8s run reads; a suite's portal address lives beside the suite that declares the portal — realm/blong-sport/.blong_k8src — because the alias is the suite's own and a shared file would hand one suite's host to the next. rc takes the closest file, so a package's copy shadows the repository's for runs started in it.

A host a realm contributes​

A webhook is the other address a suite publishes, and it is not in that block: the realm contributes the path and the port, and the deployment names the host, its certificate and the issuer. The container is ingresses, keyed by the name the contribution carries — the alias in the adapter's k8sIngresses:

chat:
slack:
k8sIngresses:
chatSlack:
host: <the webhook host>
kustomize:
deploy:
# Every path this host serves, by the name each contribution uses.
ingresses:
chatSlack:
host: <the webhook host>
tls: {secretName: chat-hook-tls}
annotations:
cert-manager.io/cluster-issuer: letsencrypt-prod

This is the one field that has to exist in both places. A local run reads it from .blong_k8src; a deployment reads it from the CR's spec.ingresses, which is where the operator takes its plan from — and the tree writes the addresses it planned with into the CR it ships, so the two do not drift. The CRD declares the field as a map, tls and annotations are merged over what the contribution said (annotations by key), and the sample beside the committed base writes PLACEHOLDER where the host goes.

A suite whose webhook host lived in its own index.ts could not be deployed by a tenant who wanted a different host — and the operator, regenerating the tree from a CR that could not state one, dropped the Ingress it had on the next pass.

cert-manager.io/cluster-issuer is the switch between the two ClusterIssuers: letsencrypt-staging issues quickly and untrusted — a browser warns and curl needs -k — while letsencrypt-prod issues a publicly trusted certificate and counts against Let's Encrypt's rate limit. Changing the annotation and re-running the cycle is the whole switch; cert-manager notices that the Secret's certificate came from another issuer and re-issues.

What a deployment publishes stays out of the repository's generated half. A split tree keeps the Ingress in local/, which is not committed, for the same reason it keeps the artifact's digest and the node names there; the base every deployment of the suite shares carries no address at all. The declaration follows the same rule: local/declaration/blongdeployment.yaml is the CR a run applies, with the real host, and example/blongdeployment.yaml beside the base is the sample a tenant copies, with the host written as PLACEHOLDER.

The cluster prerequisites​

  • Traefik and cert-manager. The operator and portal ingresses carry a tls.secretName and a cert-manager.io/cluster-issuer annotation; cert-manager turns them into certificates. Without an ingress controller the hostnames answer nothing, and without cert-manager the secrets never exist. cert-manager itself is a chart, not a vendored manifest: test/integration/cert-manager declares helmCharts: (name, repo and a pinned version) and the Namespace the chart does not create, so the install is applied as kubectl kustomize --enable-helm test/integration/cert-manager | kubectl apply -f -. apply -k cannot read helmCharts — the inflator is a flag on kustomize build, it needs a helm binary on PATH, and it pulls the chart at build time.
  • Host ports. Rootless Podman cannot bind privileged ports, so the k3d load balancer publishes host 8080/8443 and forwards them to Traefik's 80/443; the router does the public half.
  • ServiceLB. Traefik is a LoadBalancer, so k3d's ServiceLB has to stay enabled; disabling it leaves the address pending and the load balancer with no backend.
  • The release Secret. blong-operator-release holds token, referenced optional: true by the operator Deployment. Absent, the operator starts and the route refuses rather than the pod waiting in CreateContainerConfigError.

Proving it locally​

test/blong-int-kustomize is the same cycle without GitHub: its machine prepare builds and publishes the operator and suite zips to a file server in the cluster, and its suite deploy applies the generated trees, waits for the migration Job, checks the portal and logs in. realm/blong-sport is deployed with SUITE_SCENARIO=sport ARTIFACT_NAME=sport.zip; see its README for the command. The runbook and the workflow differ in where the artifact comes from, not in what a correct deployment is.