Kubernetes 80/20 Rule
Pareto principle
The Pareto principle (also known as the 80/20 rule, the law of the vital few and the principle of factor sparsity) states that for many outcomes, roughly 80% of consequences come from 20% of causes.
In our case of Kubernetes, that means that by focusing also upfront to a potential issues that we might face, we can save a lot of time by being aware of these very issues, and how to best debug them.
Universal Kubernetes Annoyances
A huge amount of time is wasted on a few common pod errors. This interactive troubleshooter helps you diagnose the root cause quickly by treating the symptom to find the disease.
Diagnosing `CrashLoopBackOff`
The container starts, but exits with an error almost immediately. Kubernetes tries to restart it, creating a "crash loop." The problem is almost always inside your application or its configuration.
This is the most critical step. The container logs almost always contain the application error that caused the crash.
kubectl logs <pod_name>
The Bare-Metal Gauntlet
In the cloud, networking and storage are managed services. On bare metal, they are your direct responsibility and the source of the most difficult challenges. This section provides a roadmap to tame them.
Networking on Hard Mode
Bare-metal networking is a three-layer problem. You must solve each in order: Host Prep, CNI, and Load Balancing. Failure at any layer leads to a non-functional cluster.
Prepare the Host
Disable swap, load kernel modules, and configure host firewalls. Most `NotReady` nodes are caused by skipping these steps.
Choose a CNI
K3s defaults to Flannel (no Network Policies). For production security, disable it and install Calico to enable network segmentation.
Expose Services
`LoadBalancer` services will be `Pending` forever without a controller. Install MetalLB to assign external IPs from your local network.
The Persistent Storage Quagmire
Choosing your storage solution is a critical architectural decision. There's no single best answer, only trade-offs. Compare the most common options below.
| Solution | Best For | Ease of Setup | Performance | Resilience | Key "Gotcha" |
|---|---|---|---|---|---|
| NFS | Homelab, simple sharing | Very Easy | Low | SPOF | Performance bottleneck |
| Longhorn | Small-to-medium prod | Easy | Moderate | Replicated | Slow rebuilds on node reboot |
| Ceph (Rook) | Large-scale prod | Complex | High | Highly Resilient | High complexity & resource use |
| OpenEBS Mayastor | Performance-critical | Moderate | Very High | Replicated | Very high CPU usage |
K3s In The Trenches
K3s has unique behaviors that can trip you up when moving from a homelab to production. Understanding its datastore options and HA model is key to building a stable cluster.
Datastore Performance: SQLite vs. Embedded `etcd`
K3s defaults to SQLite for simplicity, but this is unsuitable for a multi-server HA cluster. The `kine` translation layer introduces overhead. For production, `etcd` is mandatory. This chart visualizes the performance cliff.
`etcd` is ~4x faster under load, with lower CPU usage. It demands faster disks but is the only option for a stable, multi-server production cluster.
The Three Pillars of K3s High Availability
True HA is more than just adding servers. Neglecting any of these pillars creates a hidden single point of failure and a false sense of security.
`etcd` Quorum
You must have an odd number of server nodes (3, 5, etc.). This allows the Raft consensus algorithm to maintain a majority (quorum) and tolerate node failures. A 3-node cluster can lose 1 server; a 5-node cluster can lose 2.
Stable API Endpoint
Agents and clients need a fixed IP address that floats between healthy servers. Without a Virtual IP (VIP), if the server you're targeting fails, your connection breaks. Use a load balancer or `keepalived` for the API server endpoint.
Performant Hardware
`etcd` is extremely sensitive to disk I/O latency. Running an HA cluster on slow storage like Raspberry Pi SD cards is a recipe for instability and data corruption. Use SSDs for your server nodes.
Production-Readiness Crucible
A functional cluster is not a production cluster. Use this checklist to systematically harden, monitor, and back up your system for mission-critical workloads.
- Harden Host OS: Set secure kernel parameters in /etc/sysctl.d/ and secure file permissions on K3s certs.
- Enable Audit Logging: K3s disables this by default. Enable it via kube-apiserver-args to create a forensic trail.
- Use RBAC Least Privilege: Avoid `cluster-admin`. Create narrowly-scoped Roles and RoleBindings for service accounts.
- Enforce Pod Security Standards (PSS): Label production namespaces with `pod-security.kubernetes.io/enforce: restricted`.
- Implement Network Policies: Start with a default-deny ingress policy in each namespace to prevent lateral movement.