Update Kubernetes Secrets in Go Without Restarting Pods
The operational cost of a small configuration change
Restarting a Go service to apply a Kubernetes Secret or ConfigMap change can interrupt requests and leave connection pools cold. For a rotated database password, that’s a lot of disruption to change one value. The configuration edit may be small, but applying it through a restart touches the whole running process: the old instance stops accepting work, a replacement starts, and the application has to rebuild any state it held in memory.
Kubernetes can manage the transition between old and new pods, but that doesn’t make a restart operationally invisible. Readiness checks and replica counts can help keep a service available while pods turn over. They can’t guarantee that every request avoids delay, that every connection is immediately ready, or that application behavior won’t briefly vary between instances while the rollout proceeds. The amount of disruption depends on the service and its deployment, but the underlying tradeoff is familiar: configuration changes are easy to express, while applying them safely can involve several systems.
Three common ways to apply an update
Teams commonly trigger a rollout with kubectl rollout restart deployment/my-app, use Reloader to restart pods when a Secret changes, or mount the secret as a volume and rely on the application to reread it. Many services load configuration only at startup, so that last approach may not apply updates.
A manual rollout is straightforward and explicit. Someone or some automation tells the deployment to replace its pods, and the application reads configuration again as each new process starts. That approach fits systems where restart behavior is already well understood. Its cost is that even a change to one setting can repeat the work of starting the entire service.
Automation can remove the need for an operator to notice each change and initiate a rollout. A controller that watches configuration changes can prompt the same pod replacement process. That reduces a manual step, but it doesn’t change what a restart means for the application: new processes still need to start, establish dependencies, and become ready to serve traffic.
Volume mounts offer a different mechanism. Kubernetes can update files mounted from a Secret or ConfigMap, but an application must detect and interpret those updates before they affect its behavior. A process that reads settings once during startup won’t automatically begin using a new file just because the file changed. The file can be current while the service continues using values it loaded earlier.
What hot reload changes
Mamori offers another path for Go services: hot reload configuration without restarting pods. The practical gain is applying a changed value while keeping the running process—and its in-flight work—intact.
That distinction matters because configuration is consumed by application code, not by Kubernetes alone. A successful update requires a path from the changed value to the parts of the service that depend on it. In a Go process, those parts might include components that manage database access or other connections. The source doesn’t describe Mamori’s specific implementation, so the relevant point is the stated outcome: configuration can change without replacing the pod.
Keeping the process alive can avoid startup work associated with a restart. It may also avoid dropping work simply because a process was replaced. But hot reload doesn’t make every setting safe to change at runtime. Some values can be swapped while a service is running; others may be tied to initialization or to objects that don’t support reconfiguration. Each application still needs a clear account of which settings can change and how the new value takes effect.
Secrets, credentials, and connection behavior
A rotated database password makes the operational question concrete. If a service reads credentials only at startup, updating the Secret won’t necessarily update the credentials held by the live process. A restart can force the application to read the new value, but it also replaces the process. A hot reload path aims to update configuration without that replacement.
There’s another layer: changing a password in configuration and changing the credentials used by existing database connections aren’t always the same operation. Connection pools manage reusable connections, and established connections may have their own lifecycle. An application has to account for how updated credentials reach new connections and what happens to existing ones. The source doesn’t specify how Mamori handles those details, so readers should distinguish the general promise of configuration reload from the behavior of any particular database integration.
This is more significant than it looks when a service has work in flight or a costly startup sequence. And yet, hot reload isn’t automatically a cure for every kind of interruption. A dependency may still reject connections, a setting may require rebuilding internal state, or application code may not apply a new value consistently. The operational benefit depends on the update path being correct as well as fast.
What teams should evaluate
If you're working in this space, the useful question isn’t just whether a process can read a new value. It’s whether the service can move from the old configuration to the new one in a controlled way, and whether that change can be observed when something goes wrong. Teams typically need to know which configuration sources are supported, which settings are reloadable, and how the application behaves if a new value is invalid or unavailable.
(And this is the part most people overlook.) A reload mechanism shifts some responsibility into the running application. With restart-based updates, process initialization provides a natural point at which configuration is read and checked. With hot reload, validation and error handling have to work while the service is already handling requests. That doesn’t make the approach worse; it makes the application’s update behavior part of the operational design.
There’s also a consistency question for services running more than one pod. If configuration changes propagate at different times, instances may briefly operate with different values. A restart-based rollout can have that property during a rolling update too, so neither method should be assumed to produce an instantaneous fleet-wide switch. Teams need to understand how the chosen mechanism distributes updates and how their service behaves during any transition.
The key tradeoff is plain. Restarts use a familiar lifecycle boundary and can be easier to reason about, but they repeat startup work. Hot reload can preserve a running process and its in-flight work, but it depends on application support and careful handling of changed values. For a service where a configuration edit is frequent or disruptive, that difference deserves attention. A smaller operational change, when it works as intended.
