Kubernetes Deployment Best Practices for Stateful IoT Workloads

The Challenge with Stateful Workloads

IoT data collection services maintain persistent TCP connections, local state, and time-sensitive processing. Standard Kubernetes Deployments treat pods as disposable – which conflicts with long-lived IoT connections.

Use StatefulSets for Connection-Holding Services

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: meter-collector
spec:
  replicas: 3
  serviceName: meter-collector
  podManagementPolicy: Parallel
  template:
    spec:
      containers:
      - name: collector
        resources:
          requests:
            cpu: "500m"
            memory: "512Mi"
          limits:
            cpu: "2"
            memory: "2Gi"

Graceful Shutdown is Non-Negotiable

Always implement SIGTERM handlers that flush in-flight data before the pod terminates:

func gracefulShutdown(server *CollectorServer) {
    sigCh := make(chan os.Signal, 1)
    signal.Notify(sigCh, syscall.SIGTERM, syscall.SIGINT)
    <-sigCh
    server.DrainConnections(30 * time.Second)
    server.Shutdown()
}

Node Affinity for Predictable Performance

Use node affinity to pin data-intensive pods to nodes with local NVMe storage, avoiding network storage latency spikes.

Conclusion

Kubernetes works extremely well for IoT workloads when you respect the stateful nature of the services. StatefulSets + graceful shutdown + node affinity covers 90% of the edge cases.