Skip to content
This documentation covers the kagent 1.0 alpha. For the latest 0.x release, see the 0.x docs.

For the complete documentation index, see llms.txt. Markdown versions of all docs pages are available by appending .md to any docs URL.

Operational considerations

Page as Markdown
Tune the database, controller replicas, and the node pools that host Substrate Workers before you run kagent in production.
A default kagent installation runs one controller replica against the development database that kagent install creates, and schedules agents onto any node in the cluster. Each default works for a trial installation, and each needs a decision before you run kagent in production: which database to run, how many controller replicas to keep, and how to treat the nodes that host Workers.
Choose a database

kagent stores SessionsSessionA running conversation with one Agent. Unlike the resources it is built from, a Session is not a Kubernetes resource: kagent's gRPC API creates it and its PostgreSQL database tracks it.Learn more and compiled revisionsRevisionThe compiled, immutable output of one Agent, identified by a content digest. A Session runs the revision it was created from for its whole life, so editing the Agent affects only sessions created afterward. in PostgreSQL. The kagent chart deploys no database of its own. It reads a connection string from a Kubernetes Secret, and database.postgres.connectionStringSecretRef names that Secret and the key inside it.
database:
  postgres:
    connectionStringSecretRef:
      name: kagent-postgres
      key: connectionString
kagent install creates a development database in the cluster, so an installation needs no external database to start. Run your own PostgreSQL instance for production, so that the data outlives the cluster that kagent runs on.
The development database

kagent install creates a database for you unless you pass --skip-database-setup. It deploys a single-replica kagent-postgresql Deployment backed by a 500Mi PersistentVolumeClaim, creates the roles that kagent and Agent Substrate connect as, and writes the connection Secrets for both: kagent-postgres in the kagent namespace, and substrate-postgres-readwrite and substrate-postgres-owner in ate-system.
Connections to this database carry no password. When database.postgres.clientCertificate.enabled is set to true, which kagent install does, each client authenticates with a Pod Certificate projected into its pod under /run/postgres.podcert.ate.dev. The connection string points its sslcert and sslkey parameters at those files.
Important
The development database is a single pod backed by one PersistentVolumeClaim, and kagent install sizes that claim at 500Mi. Losing the pod’s volume loses every Session and every compiled revision, so move a production installation to a PostgreSQL instance that you manage.
Use an external PostgreSQL instance

For production, run PostgreSQL outside the cluster’s lifecycle so that a kagent uninstall cannot delete the data.
  1. Create a Secret that holds the connection string.
    kubectl create secret generic my-postgres \
  --namespace kagent \
  --from-literal=connectionString="postgresql://user:password@postgres.example.com:5432/kagent?sslmode=verify-full"
  2. Point the chart at your Secret in your Helm values file.
    database:
  postgres:
    connectionStringSecretRef:
      name: my-postgres
      key: connectionString
    clientCertificate:
      enabled: false
    role: kagent_owner
    schema: kagent
    vectorEnabled: true
    skipMigrations: false
    pool:
      maxConns: 20
      minConns: 2
controller:
  replicas: 3
    SettingDescription
    database.postgres.connectionStringSecretRefThe Secret and key that hold the connection string. Keeping the string in a Secret keeps the credentials out of Helm values. The controller restarts to pick up a changed Secret.
    database.postgres.clientCertificateSet enabled to true only for a database that authenticates clients with Pod Certificates, such as the one that kagent install creates. The connection string must then point its sslcert and sslkey parameters at the files under /run/postgres.podcert.ate.dev. An external instance that authenticates with a password, such as the one in the preceding example, leaves enabled at false.
    database.postgres.roleThe role that kagent assumes on each connection. Give each installation that shares one database a distinct role.
    database.postgres.schemaThe schema that holds kagent’s tables. database.postgres.vectorSchema names the schema that holds the shared pgvector extension, and kagent needs USAGE on it.
    database.postgres.vectorEnabledSet to true only when the instance has the pgvector extension installed. The setting enables the vector migration, and features that depend on it, such as long-term memory, fail without the extension. The development database does not include pgvector.
    database.postgres.poolConnection pool sizing, through maxConns, minConns, maxConnIdleTime, and maxConnLifetime. Omit the fields to keep the pgx library defaults.
    database.postgres.skipMigrationsSet to true to stop the controller from running migrations at startup. The controller then verifies that the database is already migrated and fails if it is not. Apply the migrations from a pipeline before you install or upgrade.
  3. Apply the values to the kagent Helm release.
    helm upgrade --install kagent \
  oci://ghcr.io/kagent-dev/kagent/helm/kagent \
  --version 1.0.0-alpha9 \
  --namespace kagent \
  --values kagent.yaml
Run multiple controller replicas

So that a controller failure does not stop reconciliation, set controller.replicas to a number higher than 1.
controller:
  replicas: 3
Leader election keeps the replicas from conflicting. One replica takes a Kubernetes lease and performs reconciliation, garbage collection, and scheduled runs. The other replicas stay ready and take over when the leader’s lease expires.
Important
If an installation runs on the development database, extra controller replicas do not make the installation highly available. That database is a single pod backed by one PersistentVolumeClaim, so losing it stops every replica at once. Run an external PostgreSQL instance before counting controller replicas toward availability.
Node operations

A WorkerWorkerA pre-started, sandboxed pod that hosts at most one Actor at a time. Actors are multiplexed across a pool of Workers over time.Learn more is an ordinary Kubernetes pod, so every node event reaches the ActorsActorThe sandboxed unit of compute, provided by Agent Substrate, that runs a Session's conversation loop. Every Session is backed by one.Learn more that this Worker hosts. What the event costs depends on what each Actor was doing when the node went away.
  • A suspended Actor costs nothing. Its state is already written to snapshot storage, and it occupies no Worker while it waits, so the next message resumes it onto whichever Worker in the pool has availability.
  • An awake Actor is lost. The Session’s in-flight state lives in the memory of its sandbox on the Worker, and Agent Substrate writes nothing to snapshot storage until this Actor suspends.
A lost Actor moves to ACTOR_STATE_CRASHED, and Agent Substrate refuses resume and suspend on an Actor in this state. Through kagent, deleting the Actor and starting a new SessionSessionA running conversation with one Agent. Unlike the resources it is built from, a Session is not a Kubernetes resource: kagent's gRPC API creates it and its PostgreSQL database tracks it.Learn more is the only available option, so the Session ends with the node.
Many Actors on a node are suspended at any moment, so losing the node crashes only the ones that are mid-turn. Those Actors still lose their Sessions, and a node pool that runs Workers therefore needs stricter rules than the rest of the cluster. A Worker pod that Kubernetes deletes gives its Actors time to suspend. Node auto-upgrade, a spot reclaim, and a pool scale-down do not.
Graceful pod deletion

Deleting a Worker pod is safe. Kubernetes forwards SIGTERM into the Actor’s containers, the Worker drains for up to 30 minutes inside a pod termination grace period of 3600 seconds, and an Actor that suspends inside this window keeps its state and stays resumable. Rolling a workerImage change through a pool therefore finishes any in-flight turns rather than cutting them off.
Node auto-upgrade

A reclaimed node is not a graceful deletion, and node auto-upgrade is the most likely way to hit one. Google Kubernetes Engine (GKE) enables auto-upgrade by default and runs it on Google’s maintenance schedule, not yours.
Important
Turn node auto-upgrade off on every node pool that runs Workers.
gcloud container node-pools update "${NODE_POOL}" \
  --cluster "${CLUSTER_NAME}" --location "${CLUSTER_LOCATION}" \
  --no-enable-autoupgrade
Spot and preemptible nodes

A provider reclaims spot and preemptible nodes on its own schedule, and each reclaim crashes every Actor that was awake on the node. A pool on reclaimable capacity therefore trades a lower bill for Sessions that end without warning. Use these nodes only where the agents on this pool can afford to lose a turn.
WorkerPool scale-down

When you scale a serving WorkerPool down, it removes pods without suspending the Actors on them. This strands Sessions in the same way that a reclaimed node does. For pool sizing and the rest of the Substrate runtime settings, see Tune Agent Substrate.
Expire idle Sessions

Every SessionSessionA running conversation with one Agent. Unlike the resources it is built from, a Session is not a Kubernetes resource: kagent's gRPC API creates it and its PostgreSQL database tracks it.Learn more holds database records and pins a runtime revision for as long as it exists, including a Session that is never opened again. The kagent controller deletes idle Sessions on a timer. Deleting idle sessions bounds the records that a kagent installation stores as well as the revisions that it holds open.
A leader-only worker sweeps Sessions once a minute and deletes every Session that has been idle past the limit, using the same deletion workflow that a client delete uses.
To choose how long a Session can be idle before the worker deletes it, set controller.sessionIdleTTL in your kagent Helm values. The value is a Go duration, and it defaults to 168h, or seven days. Setting it to 0 turns the worker off, retries included, and a negative value is rejected.
controller:
  sessionIdleTTL: "720h"
No per-agent override exists, so this one value governs every Session in the installation. No maximum exists either, so a longer value is accepted.
Idle time runs from whichever is later: the Session’s creation, or its most recently stored A2A event. Reads, renames, lifecycle calls, and retried writes do not reset it, so a Session that is only ever listed still expires.
Work in progress is never deleted. A Session with a running task survives past the interval, and so does one with a turn waiting at INPUT_REQUIRED or AUTH_REQUIRED. A pending lifecycle operation or a checkpoint being captured also holds a Session open.
Important
Expiry deletes the Session, not its history. Deleting an idle Session removes the Session, its shares, its runtime row, and its creation receipt, so GetSession returns not-found afterward. The A2A context, its tasks, its event history, and any explicit checkpointsCheckpointA durable pin on the snapshot that a Session most recently suspended to, and a record of how far its transcript had advanced. Not a new state: tagging copies the snapshot so that Agent Substrate does not collect it, and a second Session can be forked from it.Learn more stay in PostgreSQL, and a retained checkpoint outlives the Session it was taken on. kagent has no setting that bounds this retained history, so size a database for a long-running installation with the audit history planned separately.
To watch the sweep, scrape kagent_session_expired_total, which counts the Sessions that it removed. Each removal also logs expired idle session at debug level with the Session’s ID and how long it had been idle.
How configuration changes reach agents

kagent watches the Secrets and ConfigMaps that a HarnessHarnessA Kubernetes custom resource defining how an agent is allowed to run: its runtime, workload image, and WorkerPool and snapshot storage. An Agent pairs it with the AgentTemplate it runs.Learn more and AgentTemplateAgentTemplateA Kubernetes custom resource defining what an agent does: its model, system prompt, tools, skills, and plugins. It runs only once an Agent pairs it with a Harness.Learn more reference, such as the API keys and TLS certificates in a ModelConfigModelConfigA Kubernetes custom resource naming one model at one provider, along with the credentials to reach it. An AgentTemplate references one by name, and every agent compiled from that template calls the model that it names.Learn more. An edit to one of them recompiles the pair into a new revision.
A new revision does not reach the Sessions that are already running. A Session is pinned to the revision that it was created from and keeps that revision for life. To move an existing conversation onto new configuration, create a new Session.
The model provider API key is the exception. Agents do not hold the key. Instead, the Agent Substrate egress gateway reads the key from the Secret and adds it to each model request, so a rotated key reaches running Sessions after you restart the egress gateway. See Rotate the model provider API key.
This behavior differs from kagent 0.x, where an agent ran as a Deployment and a secret change restarted its pods.
Rotate the model provider API key

To rotate the model provider API key, update the Secret that holds the key. Then you can restart the Agent Substrate egress gateway so that running Sessions use the new value.
Manage the Secret yourself in a production installation. You then rotate the key at its source, and kagent reads the current value.
  1. Update the key.
    When providers.<provider>.apiKey is empty and apiKeySecretRef names a Secret that you create, such as one that an ExternalSecret syncs, the chart does not render the Secret. Update the Secret wherever you manage it.
  2. Restart the Agent Substrate egress gateway. The egress gateway can keep sending the previous key after the Secret changes, and a restart makes it use the new key right away.
    kubectl rollout restart deployment/atenet-egress -n ate-system
Route agent traffic through a proxy

A proxy puts agent-to-agent and agent-to-MCP calls on the same egress path as the rest of your cluster. Your existing inspection, logging, and policy then cover them too. When agents and MCP servers sit behind an API gateway or proxy, point kagent at the proxy endpoint.
proxy:
  url: "http://proxy.kagent.svc.cluster.local:8080"
The controller rewrites internally built Kubernetes URLs to the proxy and sets the x-kagent-host header so that the proxy routes each request to the correct backend. The rewrite covers one agent invoking another as a tool, and an agent calling a RemoteMCPServer at an internal URL. An external URL, such as a RemoteMCPServer at https://external.example.com, is left alone.
Scrape controller metrics

The controller serves a Prometheus /metrics endpoint, turned off by default. To enable the endpoint, grant Prometheus access to it, and review the metrics that the controller reports, see Metrics.
Next steps

Tune Agent SubstrateSize the WorkerPool and configure snapshot storage.
DebugWork back from a symptom to the resource that caused it.
Tune Agent Substrate