Cloud & Infrastructure
When Kubernetes is actually necessary
Kubernetes is a platform for building platforms, and its operational cost is real even when the control plane is managed. For many workloads a container service such as ECS on Fargate, Cloud Run or Azure Container Apps is the better engineering decision.
· 4 min read · Starone Engineering
Kubernetes solves a hard problem well: scheduling many containerised workloads across a pool of machines, keeping them running and exposing a uniform API for everything around them. It has become the default answer to the question of where containers run, including for systems that do not have the problem it solves.
Our position is that Kubernetes should be a decision with stated reasons. For a small number of stateless services, a managed container service delivers the same outcome with a fraction of the operational surface. Kubernetes becomes the right choice when the workload or the organisation needs what only it provides.
What you take on with a cluster
A managed offering such as EKS, GKE or AKS runs the control plane. It does not remove the rest of the work. A production cluster is an assembly of components that someone has to choose, configure, upgrade and debug.
- Version upgrades several times a year, each with API deprecations and a version-skew policy to respect across control plane, nodes and add-ons.
- Node lifecycle: operating system patching, node pool sizing, autoscaling with Cluster Autoscaler or Karpenter, and draining without dropping traffic.
- Networking: the CNI plugin, IP address planning, ingress or Gateway API controllers, DNS and network policies.
- Security: RBAC, pod security standards, workload identity for cloud access, admission policies, secrets integration and image provenance.
- The add-on estate: certificate management, external DNS, metrics and log collection, each with its own release cycle.
The cost is mostly people
The cluster fee is rarely the significant line. The significant cost is the engineering attention needed to keep the platform current and to diagnose failures that cross layers, such as a pod that cannot resolve DNS because of a conntrack limit on the node, or a deployment stuck because a pod disruption budget blocks a drain.
Application teams pay as well. They need to understand requests and limits, probes, rollout strategies and how a manifest becomes running pods. A misjudged liveness probe restarts a healthy service under load. Missing resource requests let one workload starve its neighbours. Without a platform team to provide sensible defaults, that knowledge has to live in every team.
Where simpler services are enough
If the system is a handful of stateless HTTP services and some background workers, managed container platforms cover the requirement. Amazon ECS on Fargate runs containers without nodes to manage and integrates directly with IAM, load balancers and CloudWatch. Google Cloud Run scales request-driven containers to zero and back. Azure Container Apps and App Service fill the same role on Azure.
These services provide rolling deployments, autoscaling, health checks, secrets injection and private networking. What they withhold is extensibility: no custom controllers, limited control over scheduling and a narrower networking model. For most business applications, that limitation never becomes a constraint. Because the unit of deployment is still a container image, moving to Kubernetes later is a change of deployment descriptors and pipeline, not a rewrite.
Signals that Kubernetes is the right tool
The case for Kubernetes is strong when several of the following hold.
- Dozens of services owned by multiple teams, where a consistent deployment and policy model is worth a dedicated platform.
- Workloads that need fine scheduling control: GPU sharing for inference, node affinity, topology spread or bin-packing of batch jobs.
- Reliance on operators and the wider ecosystem, such as Argo CD for GitOps, KEDA for event-driven scaling or a service mesh for mutual TLS.
- A requirement to run the same platform across cloud providers, on premises or at the edge.
- Software that is itself distributed for customers to run in their own clusters.
Stateful workloads deserve a separate decision
Running databases and brokers on Kubernetes is possible with StatefulSets and mature operators, and it is a reasonable choice for teams with the expertise or with portability requirements. It also places backup, failover, storage performance and upgrade risk on the same team that runs the cluster. Where a managed database or managed Kafka service meets the requirement, we prefer it and keep the cluster for stateless compute.
The middle ground
The choice is not binary. GKE Autopilot and EKS Auto Mode remove node management while keeping the Kubernetes API, which suits teams that want the ecosystem without owning the data plane. They cost more per unit of compute and restrict some low-level configuration, which is usually a fair exchange.
Portability is often cited as a reason to adopt Kubernetes early. It is partly true. Manifests are portable, but identity, load balancing, storage classes and managed data services are provider-specific on every platform. A containerised application with configuration in the environment and no provider SDK calls in its business logic is portable enough to defer the decision.
How we decide
We start from the workload and the team. How many services exist now and realistically within two years, whether any need scheduling features a simpler platform lacks, and who will own upgrades and incidents at three in the morning. If there is no credible answer to the last question, the simpler platform is the responsible choice regardless of the technical appeal.
The opposite mistake also exists. A team that outgrows a simple platform and responds by building its own scheduler, service discovery and deployment tooling around it has recreated Kubernetes badly. When the workarounds start to accumulate, adopting the standard platform is cheaper than maintaining a bespoke one.