Continuous deployment to a local k3d cluster
The repository deploys a suite the same way a released product would: an operator converges a
BlongDeployment, a release workflow builds the artifact, and the operator's gateway is the only
door the workflow opens. realm/blong-sport is the worked example; the mechanism is generic.
The whole path, from the merged commit to a pod that mounts the suite's version volume:
The three halves
- Operator — one per cluster, installed once, reconciling every tenant's CR; in
realm/blong-kustomize. - Artifact — the
rush deployscenario, zipped and fetched by the fill Job from a URL; the release asset. - Trigger — the token-gated
kustomize.release.approveroute on the operator's gateway, at the address the installer names inkustomize.deploy.operator.ingress.host.
The suite's tree never carries the operator: it carries the CR, and the bindings the operator needs
in the suite's namespaces. The operator's own install tree carries the CRD, its RBAC, the
BLONG_RELEASE_TOKEN reference and the ingress. The two are applied from system/operator/local
and the suite's system/kustomize/local respectively.
.github/deployments.json
The deployment map is the only place a suite is named. A key is the alias a human dispatches, and its fields are what the workflow reads:
{
"sport": {
"release": "blong-sport",
"package": "@feasibleone/blong-sport",
"packagePath": "realm/blong-sport",
"scenario": "sport",
"artifact": "sport.zip",
"namespace": "blong-sport",
"suite": "blong-sport",
"entry": "./index.ts",
"profile": "monolith",
"follows": [
"blong",
"blong-access",
"blong-browser",
"blong-core",
"blong-gateway",
"blong-gogo",
"blong-kustomize",
"blong-lib",
"blong-login",
"blong-mock",
"blong-party",
"blong-server"
]
}
}
release is the release-please component, so the release tag is <release>-v<version>. scenario
names both common/config/rush/deploy-<scenario>.json and the kustomize deploy scenario; artifact
is the name the zip is published under; suite and namespace are the CR's name and tenant. A
second suite is a second key plus its scenario file — the workflow does not change.
follows is the list of release-please components whose release re-deploys this deployment, and it
is what makes a batch reach the suites it changed: a release of a realm inside a suite publishes no
release of the suite, so without the list the running suite keeps the version it already had. Each
list is the deployment's production dependency closure — what the artifact carries, restricted to
the components release-please publishes — plus core/blong-gogo, whose version keys the framework
image, and core/blong-browser, which the portal bundle is built from. The deployment's own
release is always a trigger and is not repeated in the list, and a name that no component answers
fails the run instead of waiting for a release that will never come.
The two tenant deployments are sport (one process, the monolith profile) and suite — the
framework's own full-stack suite, deployed with the default realm profile, so each realm that
serves something is a process of its own:
{
"suite": {
"release": "blong-suite",
"package": "@feasibleone/blong-suite",
"packagePath": "suite/blong-suite",
"scenario": "suite",
"artifact": "suite.zip",
"namespace": "blong-suite",
"suite": "blong-suite",
"entry": "./index.ts",
"profile": "realm",
"follows": [
"blong",
"blong-access",
"blong-browser",
"blong-chain",
"blong-core",
"blong-gogo",
"blong-kustomize",
"blong-lib",
"blong-login",
"blong-marine",
"blong-mock",
"blong-server",
"marine-data"
]
}
}
profile is documentation, not input: the split is the CR's spec.profile, which the operator
re-derives the plan from, and realm is what plan.ts falls back to when a CR names none.
One entry is not a tenant's suite at all: the deployment realm's own install, marked
"installer": true, which resolve skips when it looks for the alias a tag names. It is there for
the other readers of the map — the workflow job that publishes the operator's artifact, and
realm/blong-kustomize/deploy.sh, which resolves a released artifact's URL from the same record
rather than hardcoding a prefix in a shell script:
{
"operator": {
"installer": true,
"release": "blong-kustomize",
"package": "@feasibleone/blong-kustomize",
"packagePath": "realm/blong-kustomize",
"scenario": "operator",
"artifact": "blong-kustomize.zip"
}
}
No address is in the map: a host belongs to the machine that deploys, so the operator's comes from
.blong_k8src and the suite's portal host from the same file — see
where an address comes from.
.github/workflows/deploy.yaml
The workflow has one trigger that matters for releases and one for people:
workflow_runof theReleaseworkflow,completed, onmain— a release-please batch. The releases that run published are the batch: release-please creates them inside it, so the run's own window bounds them, and every release of one batch is tagged at the same commit.workflow_dispatchwithdeploymentandtag— a re-deploy, or a deployment whose release was published earlier.
A batch that names nothing a deployment follows is not a failure: resolve plans an empty matrix
and the jobs are skipped. Because a batch is one commit, the newest tag of it builds every artifact
— a blong-gogo release and a blong-marine release of the same batch carry the same tree.
resolve turns the batch into a matrix: one entry per deployment whose follows list names a
component in the batch, plus the operator when blong-kustomize is one of them. The framework
version is read from .release-please-manifest.json at the batch's tag (core/blong-gogo), and the
job waits for ghcr.io/feasibleone/blong-gogo:v<gogoVersion> to exist — a release is not a promise
that its image is pullable yet.
artifact is one job per matrix entry, named and serialised by its alias: it checks out the batch's
tag, restores the pnpm store CI cached, installs, builds what the deploy scenario carries, runs
rush deploy --scenario <scenario>, zips the target folder, hashes it, attaches it to the release
with gh release upload, and asks the operator to converge — in the same leg, so the digest it
reports is the bytes it just uploaded. The URL the operator receives is the release asset URL of the
batch's tag.
The deployment's own version travels with that call, read from the manifest at the tag rather than
from the tag's suffix: a batch triggered by blong-gogo deploys suite 1.19.0 with framework
v1.51.0, and the operator shows both, so a reader can tell what actually moved.
operator-artifact is the same work for the installer rather than for a tenant: it answers a batch
that contains a blong-kustomize-v<version> release and attaches three assets —
blong-kustomize-<version>.zip, its .sha256, and blong-kustomize.zip as a stable alias for
releases/latest/download/, which is the name the install tree falls back to. The versioned name is
what a version installs and the next release cannot overwrite; the digest is an asset rather than a
sentence in the release notes because it is a script that reads it, and what it names is the
directory the artifact lands in.
The same job then deploys what it published, and the caller is the operator's own release route:
suite: blong-operator tells the operator the announcement is about its own install tree, so
instead of patching a BlongDeployment — which it does not have — it runs the registry pass that
plans that tree, with the release's version and artifact laid over its own configuration. The pass
applies the fill Jobs for the new artifact identity first and waits for them, then the Deployment
that runs the operator, so an artifact nothing can fetch leaves the running process where it was.
The install-owned kinds — the namespace, the CRD, the ServiceAccount and the RBAC objects — are
skipped by name, so a release can never widen the rights the operator runs under; a tag that changes
one of them still needs npm run deploy from a checkout (or
kubectl apply -k system/operator/local).
The call builds the JSON-RPC body with jq -n --arg — never shell interpolation, because the token
and the URL are data — and posts it to secrets.DEPLOY_ENDPOINT with secrets.DEPLOY_TOKEN. The
route is plain JSON-RPC over HTTPS: the gate is the token, not a bearer or an MLE handshake, and the
ingress certificate protects it in transit.
# The route the operator's ingress serves, and the token the operator reads from its Secret.
DEPLOY_ENDPOINT=https://<operator host>/rpc/kustomize/release/approve
DEPLOY_TOKEN=<the same value as the operator's blong-operator-release secret>
A wrong or missing token is answered 401 with kustomize.release.tokenMismatch or
kustomize.release.tokenMissing; the CR is untouched, because the check happens before the notify
handler. A correct token patches the CR's version, suiteVolume.artifact (URL and digest) and,
when the release moved it, minFrameworkVersion, and the operator converges.
The answer is a report as well as a verdict: report carries the suite, the version it moved to —
and from, when the operator could read the previous value — the fields the seam applied and, for the
operator's own release, the pass counts; markdown is those facts as one document. The workflow
appends that document to the job summary, and a chat notification can post it unchanged, so the two
never describe one release differently.
Where an address comes from
A host is a DNS record and a certificate somebody owns, so it is configuration rather than a value
in a source tree: no realm, workflow or committed manifest names one. The framework reads the file
of the k8s intent — .blong_k8src, found from the working directory upwards — and its
kustomize.deploy block supplies the addresses of both halves:
kustomize:
deploy:
operator:
ingress:
host: <the operator's host>
tls: {secretName: blong-operator-tls}
annotations:
cert-manager.io/cluster-issuer: letsencrypt-prod
portal:
web:
host: <the suite's portal host>
annotations:
cert-manager.io/cluster-issuer: letsencrypt-prod
The file is ignored (.blong_*) and local to the machine. The repository's copy names the
operator's address, which every k8s run reads; a suite's portal address lives beside the suite
that declares the portal — realm/blong-sport/.blong_k8src — because the alias is the suite's own
and a shared file would hand one suite's host to the next. rc takes the closest file, so a
package's copy shadows the repository's for runs started in it.
A host a realm contributes
A webhook is the other address a suite publishes, and it is not in that block: the realm contributes
the path and the port, and the deployment names the host, its certificate and the issuer. The
container is ingresses, keyed by the name the contribution carries — the alias in the adapter's
k8sIngresses:
chat:
slack:
k8sIngresses:
chatSlack:
host: <the webhook host>
kustomize:
deploy:
# Every path this host serves, by the name each contribution uses.
ingresses:
chatSlack:
host: <the webhook host>
tls: {secretName: chat-hook-tls}
annotations:
cert-manager.io/cluster-issuer: letsencrypt-prod
This is the one field that has to exist in both places. A local run reads it from .blong_k8src; a
deployment reads it from the CR's spec.ingresses, which is where the operator takes its plan from
— and the tree writes the addresses it planned with into the CR it ships, so the two do not drift.
The CRD declares the field as a map, tls and annotations are merged over what the contribution
said (annotations by key), and the sample beside the committed base writes PLACEHOLDER where the
host goes.
A suite whose webhook host lived in its own index.ts could not be deployed by a tenant who wanted
a different host — and the operator, regenerating the tree from a CR that could not state one,
dropped the Ingress it had on the next pass.
cert-manager.io/cluster-issuer is the switch between the two ClusterIssuers: letsencrypt-staging
issues quickly and untrusted — a browser warns and curl needs -k — while letsencrypt-prod
issues a publicly trusted certificate and counts against Let's Encrypt's rate limit. Changing the
annotation and re-running the cycle is the whole switch; cert-manager notices that the Secret's
certificate came from another issuer and re-issues.
What a deployment publishes stays out of the repository's generated half. A split tree keeps the
Ingress in local/, which is not committed, for the same reason it keeps the artifact's digest and
the node names there; the base every deployment of the suite shares carries no address at all. The
declaration follows the same rule: local/declaration/blongdeployment.yaml is the CR a run applies,
with the real host, and example/blongdeployment.yaml beside the base is the sample a tenant
copies, with the host written as PLACEHOLDER.
The cluster prerequisites
- Traefik and cert-manager. The operator and portal ingresses carry a
tls.secretNameand acert-manager.io/cluster-issuerannotation; cert-manager turns them into certificates. Without an ingress controller the hostnames answer nothing, and without cert-manager the secrets never exist. cert-manager itself is a chart, not a vendored manifest:test/integration/cert-managerdeclareshelmCharts:(name, repo and a pinnedversion) and the Namespace the chart does not create, so the install is applied askubectl kustomize --enable-helm test/integration/cert-manager | kubectl apply -f -.apply -kcannot readhelmCharts— the inflator is a flag onkustomize build, it needs ahelmbinary on PATH, and it pulls the chart at build time. - Host ports. Rootless Podman cannot bind privileged ports, so the k3d load balancer publishes
host
8080/8443and forwards them to Traefik's80/443; the router does the public half. - ServiceLB. Traefik is a
LoadBalancer, so k3d's ServiceLB has to stay enabled; disabling it leaves the address pending and the load balancer with no backend. - The release Secret.
blong-operator-releaseholdstoken, referencedoptional: trueby the operator Deployment. Absent, the operator starts and the route refuses rather than the pod waiting inCreateContainerConfigError.
Proving it locally
test/blong-int-kustomize is the same cycle without GitHub: its machine prepare builds and
publishes the operator and suite zips to a file server in the cluster, and its suite deploy
applies the generated trees, waits for the migration Job, checks the portal and logs in.
realm/blong-sport is deployed with SUITE_SCENARIO=sport ARTIFACT_NAME=sport.zip; see its README
for the command. The runbook and the workflow differ in where the artifact comes from, not in what a
correct deployment is.