I've been watching too much AI stuff, and too much coding stuff. Time to refresh. For now, system engineering, the ship that actually brings your package to the sea of internet.

If you click this you've probably already know what kubernetes are, in the simples term it's just a orchestration tool to help manage your docker cluster.

But "orchestration tool" doesn't really tell you anything useful. So let's actually go through it.

Why docker alone isn't enough

Docker solved something real. You package your app with everything it needs, it runs the same everywhere, "works on my machine" stops being an excuse.

The cracks show when things get bigger. You need to run multiple copies of your app at the same time. A container crashes at 3am and you want it back up without waking anyone. You want to push a new version without shutting everything down first. Docker doesn't handle any of that on its own.

Docker Compose is great for local dev but it was never meant to run production traffic. Docker Swarm tried and mostly faded. So Kubernetes filled the gap. It looks at your containers and asks "where should these live, how many, and what do we do when one goes down." You describe what you want, it works out the rest.

think of it like this

I'd skip this in most tutorials. Here it actually matters.

Kubernetes runs on a cluster, just a bunch of machines. One of them is the control plane (used to be called the master node). That's the one making all the decisions: which machine runs what, how many of each thing, what to do when something crashes. The rest are worker nodes, and those just run your containers.

Kubernetes cluster architecture: control plane and worker nodes
how a cluster is laid out. kubectl talks to the control plane, which decides where pods go.

The thing that trips people up: you're not telling Kubernetes "run this on machine 3". You're telling it "I want three copies of this running". It figures out the where, handles restarts, and keeps the count right.

Four things you need to know before touching the CLI:

  • Pod: the smallest unit. usually one container. pods are temporary, they come and go, don't treat them like servers
  • Deployment: keeps a group of pods at whatever count you want. if one dies, a new one comes up
  • Service: pods get new IPs every time they restart. a Service sits in front and stays stable, so other things always know where to look
  • Namespace: folders, basically. good for separating staging from prod, or team A from team B

That's the core. Everything else is variation on those four.

Setting up locally

Three good options:

  • minikube: spins up a full cluster on your laptop inside a VM or container. easiest starting point
  • kind: runs nodes as Docker containers. faster to start, I use it in CI
  • k3s: lighter build of Kubernetes. good if you're on a Raspberry Pi or a cheap VPS

Going with minikube. You'll also need kubectl, the command line tool you use to talk to Kubernetes.

# macOS
brew install kubectl
brew install minikube

On Linux, grab the binaries from the official releases page. On Windows, WSL2 then same as Linux.

minikube start

First run downloads some stuff and starts up, so give it a minute or two. After that it's fast. Check that everything came up:

kubectl cluster-info

You want to see the control plane URL listed. If it's there, you have a cluster running. Also check that kubectl is talking to minikube and not some other cluster you might have configured:

kubectl config current-context
# should say: minikube

You can have multiple clusters set up on one machine. kubectl uses "contexts" to track which one you're working with right now.

Your first deployment

We'll use NGINX as a stand-in. Simple, well-known, and it shows up in search when you need to debug something.

Create a file called deployment.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.25
        ports:
        - containerPort: 80
        resources:
          requests:
            memory: "64Mi"
            cpu: "100m"
          limits:
            memory: "128Mi"
            cpu: "200m"

Let's go through the important parts:

  • replicas: 3 — you want three copies of this pod running at all times
  • selector.matchLabels — how the Deployment finds the pods it's responsible for. it looks for pods with app: nginx
  • template.metadata.labels — the labels that get put on the pods. needs to match the selector above, otherwise the Deployment can't track its own pods
  • image: nginx:1.25 — always pin a version. using latest will give you surprises when the image updates
  • resources.requests — the minimum resources this container needs. Kubernetes uses this to decide which machine to put it on
  • resources.limits — the maximum. if the container goes over this, Kubernetes will kill and restart it

Apply it:

kubectl apply -f deployment.yaml

Watch the pods come up:

kubectl get pods -w

You'll see them go through Pending → ContainerCreating → Running. The -w flag keeps watching for changes, so you can see it live. Press Ctrl+C when all three say Running.

To see more detail about your deployment:

kubectl get deployments

NAME               READY   UP-TO-DATE   AVAILABLE   AGE
nginx-deployment   3/3     3            3           1m

READY 3/3 means all three pods are up. UP-TO-DATE means they're all running the version you asked for.

Looking inside

Now that things are running, kubectl describe is the command you'll use constantly. It gives you the full picture of any object.

kubectl describe deployment nginx-deployment

This shows you things like: how many pods are running, the image being used, recent events, and what happened when the deployment rolled out.

For a specific pod:

# first get the pod name
kubectl get pods

# then describe it
kubectl describe pod <pod-name>

The Events section at the bottom is where Kubernetes writes what actually happened — pulled the image, scheduled on this node, started the container, or failed to do any of those things. When something breaks, this is the first place to look.

To see logs from a running container:

kubectl logs <pod-name>

If there are multiple containers in a pod, specify which one:

kubectl logs <pod-name> -c nginx

To follow the logs in real time:

kubectl logs <pod-name> -f

And if you want to actually get inside the container to poke around:

kubectl exec -it <pod-name> -- /bin/bash

This opens a shell inside the container. Useful for checking if a config file is where you think it is, or if your app can reach a dependency.

Environment variables

Real apps need configuration. The basic way to pass it in is through environment variables in the pod spec.

spec:
  containers:
  - name: nginx
    image: nginx:1.25
    ports:
    - containerPort: 80
    env:
    - name: ENVIRONMENT
      value: "production"
    - name: MAX_CONNECTIONS
      value: "100"

Apply the updated file and Kubernetes will restart the pods with the new env vars.

For sensitive values like passwords or API keys, use a Secret instead of hardcoding in the YAML:

kubectl create secret generic app-secrets \
  --from-literal=db-password=mysecretpassword

Then reference it in the pod spec:

env:
- name: DB_PASSWORD
  valueFrom:
    secretKeyRef:
      name: app-secrets
      key: db-password

Secrets are stored separately from your deployment config, which means you're not accidentally committing passwords to git.

Making it reachable

The pods are running but nothing outside the cluster can reach them yet. That's by design. You need a Service.

Create service.yaml:

apiVersion: v1
kind: Service
metadata:
  name: nginx-service
spec:
  selector:
    app: nginx
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
  type: LoadBalancer

The selector: app: nginx is how the Service finds the right pods. It's the same label we put on the pods in the deployment. If those labels don't match exactly, no traffic will get through.

There are three Service types worth knowing:

  • ClusterIP: only reachable from inside the cluster. the default. good for internal services that other parts of your app talk to
  • NodePort: opens a port on every node in the cluster and forwards it to your pods. reachable from outside, but on a weird port number (30000+)
  • LoadBalancer: asks the cloud provider for a load balancer. on minikube, you tunnel to it manually
kubectl apply -f service.yaml

Check it was created:

kubectl get services

NAME            TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
nginx-service   LoadBalancer   10.96.123.45    <pending>     80:31234/TCP   30s

The EXTERNAL-IP shows <pending> on minikube because there's no real cloud provider to give you one. Use this instead:

minikube service nginx-service

Your browser opens. NGINX welcome page. That's your container, running in Kubernetes, reachable from the outside.

Service routing traffic to pods
the Service sits in front of the pods. traffic comes in on a stable address, gets routed to whichever pod is available.

Scaling

Want more pods? One command:

kubectl scale deployment nginx-deployment --replicas=5

Watch it happen:

kubectl get pods -w

Two new pods come up. Scale back down:

kubectl scale deployment nginx-deployment --replicas=2

Kubernetes picks which pods to terminate and removes them. The Service keeps routing traffic to whatever pods are left.

You can also update the replicas number in your deployment.yaml and run kubectl apply -f deployment.yaml again. Both work — but keeping the YAML file up to date is better for the long run since it's the source of truth for what you want.

Updating to a new version

Say a new version of NGINX came out and you want to update. Change the image in your deployment file:

image: nginx:1.27

Apply it:

kubectl apply -f deployment.yaml

Kubernetes does a rolling update. It doesn't kill all your pods at once. Instead it brings up new pods one by one, waits for them to be healthy, then terminates the old ones. Your app stays running the whole time.

Watch it happening:

kubectl rollout status deployment nginx-deployment

Waiting for deployment "nginx-deployment" rollout to finish: 1 out of 3 new replicas have been updated...
Waiting for deployment "nginx-deployment" rollout to finish: 2 out of 3 new replicas have been updated...
deployment "nginx-deployment" successfully rolled out

If something goes wrong with the new version, you can roll back to what was running before:

kubectl rollout undo deployment nginx-deployment

Kubernetes keeps the history of your previous rollouts. You can also roll back to a specific version:

kubectl rollout history deployment nginx-deployment
kubectl rollout undo deployment nginx-deployment --to-revision=2

When it breaks (and it will)

Pod lifecycle states
pod states you'll see while watching with kubectl get pods -w.

ImagePullBackOff: Kubernetes can't download the container image. Wrong image name, wrong tag, or it's a private registry and you haven't set up credentials. Check:

kubectl describe pod <pod-name>

Scroll to the Events section at the bottom. It'll usually say exactly what failed.

CrashLoopBackOff: the container starts, crashes, Kubernetes restarts it, crashes again. Something in your app is failing at startup. Look at the logs:

kubectl logs <pod-name>
# if the pod already restarted:
kubectl logs <pod-name> --previous

The --previous flag gets you logs from the last time the container ran, before it crashed. Very useful.

Pods stuck in Pending: Kubernetes can't find a machine to run the pod on. Usually it's asking for more CPU or memory than any machine has free right now. Check:

kubectl describe pod <pod-name>

If you see something like 0/1 nodes are available: 1 Insufficient memory, your resource requests are too high for the cluster you have. Either lower the requests in the pod spec or add more capacity.

Service not routing traffic: nothing crashes, things just silently don't connect. The most common cause is a label mismatch — the selector in your Service doesn't match the labels on your pods. One character off and they never find each other. Check both files side by side.

You can verify what endpoints a Service is pointing to:

kubectl get endpoints nginx-service

If it shows <none>, the Service can't find any matching pods. That's a label problem.

OOMKilled: your container was killed because it used more memory than the limit you set. It'll show up in kubectl describe pod under Last State. Either fix the memory leak in your app or raise the limit in the pod spec.

The thing that actually sells Kubernetes

Kill a pod on purpose:

kubectl delete pod <pod-name>

kubectl get pods -w

NAME                                READY   STATUS        RESTARTS   AGE
nginx-deployment-xxx-aaa            1/1     Running       0          5m
nginx-deployment-xxx-bbb            1/1     Running       0          5m
nginx-deployment-xxx-ccc            1/1     Terminating   0          5m
nginx-deployment-xxx-ddd            0/1     Pending       0          0s
nginx-deployment-xxx-ddd            1/1     Running       0          3s

You deleted one. Kubernetes noticed it only had two instead of three, so it started a new one. Back to three, and you didn't do anything. That's the whole point: you say how many you want, it keeps that number running, always.

One thing though: if your container is crashing because of a bug, Kubernetes will keep restarting it. That's not a rescue, it's a loop. Self-healing is about infrastructure, not your code.

Namespaces

So far everything has been in the default namespace. For a real setup you'll want to separate things.

kubectl create namespace staging

Deploy into a specific namespace:

kubectl apply -f deployment.yaml -n staging

List everything in a namespace:

kubectl get pods -n staging
kubectl get services -n staging

Or list across all namespaces at once:

kubectl get pods --all-namespaces

A common pattern is to have one namespace per environment (staging, production) or one per team. It keeps things from stepping on each other.

Cleaning up

Delete specific resources:

kubectl delete -f deployment.yaml
kubectl delete -f service.yaml

Delete everything in the default namespace (be careful):

kubectl delete all --all

Stop minikube without deleting the cluster:

minikube stop

Blow away the whole thing:

minikube delete

Where to go from here

ConfigMaps: once you need to pass non-secret config to containers without hardcoding it in the image. similar to env vars but stored as a Kubernetes object you can update without rebuilding the image.

Persistent Volumes: databases and anything stateful need storage that outlives a pod restart. this is honestly where it starts feeling complicated. the abstractions make sense eventually, but budget some time.

Ingress: one LoadBalancer per Service gets expensive fast on a real cloud account. Ingress puts a single controller in front and routes by domain or path, which is how most production setups actually work. you'd have one public IP and route api.yoursite.com to one service, yoursite.com to another.

Helm: package manager for Kubernetes. once you're managing more than a few components, raw YAML becomes a maintenance problem. Helm gives you versioned, templated charts you can install, diff, and roll back.

RBAC: when other people need cluster access, you'll want to define who can do what. don't skip this. the defaults are more open than you'd want anywhere near production data.

Liveness and Readiness Probes: tell Kubernetes how to check if your container is actually healthy, not just running. a liveness probe restarts the container if it stops responding. a readiness probe stops sending traffic to a pod that isn't ready yet. without these, Kubernetes doesn't know if your app is actually working.

Kubernetes has a reputation for being hard to learn. Some of that's fair: there's a lot of YAML and the error messages aren't always clear. But the idea itself is simple. You say what you want, it keeps it running.

Take your time with it. It clicks.