Rerun existing completed Job

kubectl replace deletes the old job, if there is any error, your job definition is lost, don’t forget to save it first!
Replace an existing Job with itself
kubectl get job JOBNAME -o yaml | kubectl replace --force -f -
Sometimes there are errors importing the job template due to auto-generated labels or selectors .. a quick and dirty hack is to filter them out using jq
Filtering out invalid generated labels and selectors with jq
kubectl get job JOBNAME -o json | jq 'del(.spec.template.metadata.labels)' | jq 'del(.spec.selector)' | kubectl replace --force -f -

k9s

Hotkeys to switch contexts

  1. Find out the path where your config is expected:

    k9s info
 ____  __.________
|    |/ _/   __   \______
|      < \____    /  ___/
|    |  \   /    /\___ \
|____|__ \ /____//____  >
        \/            \/

Version:           0.32.4
Config:            /Users/hascode/Library/Application Support/k9s/config.yaml
Custom Views:      /Users/hascode/Library/Application Support/k9s/views.yaml
Plugins:           /Users/hascode/Library/Application Support/k9s/plugins.yaml
Hotkeys:           /Users/hascode/Library/Application Support/k9s/hotkeys.yaml (1)
Aliases:           /Users/hascode/Library/Application Support/k9s/aliases.yaml
Skins:             /Users/hascode/Library/Application Support/k9s/skins
Context Configs:   /Users/hascode/Library/Application Support/k9s/clusters
Logs:              /Users/hascode/Library/Application Support/k9s/k9s.log
Benchmarks:        /Users/hascode/Library/Application Support/k9s/benchmarks
ScreenDumps:       /Users/hascode/Library/Application Support/k9s/screen-dumps
    1 This is the path we need to remember: /Users/hascode/Library/Application Support/k9s/hotkeys.yaml

  2. Create a new hotkeys.yaml at the designated path:

    hotKeys:
  f1:
    shortCut: F1
    description: change to hascode-dev
    command: ctx hascode-stage-dev
  f2:
    shortCut: F2
    description: change to hascode-test
    command: ctx hascode-stage-test
  f3:
    shortCut: F3
    description: change to hascode-uat
    command: ctx hascode-stage-uat
  f4:
    shortCut: F4
    description: change to hascode-prod
    command: ctx hascode-stage-prod

  3. Verify it works

    • when entering :? in k9s the hotkeys should be listed in the right side of the info screen

    • the hotkeys should be working when pressing F1, F2, etc ..

Resource Recommendations with krr

when using k8 with Prometheus/Thanos

https://github.com/robusta-dev/krr

Install with brew
brew tap robusta-dev/homebrew-krr
brew install krr
krr --help

 Usage: krr [OPTIONS] COMMAND [ARGS]...

 IMPORTANT: Run `krr simple --help` to see all cli flags!

╭─ Options ───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ --install-completion          Install completion for the current shell.                                                                                                                                                                                                     │
│ --show-completion             Show completion for the current shell, to copy it or customize the installation.                                                                                                                                                              │
│ --help                        Show this message and exit.                                                                                                                                                                                                                   │
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Strategies ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ simple                                                                                                                                                                                                                                                                      │
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Utils ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ version                                                                                                                                                                                                                                                                     │
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
Get resource recommendations
kubectl port-forward -n my_ns svc/prometheus 9090 (1)
krr simple -p http://127.0.0.1:9090 (2)

┏━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┓
┃ Number ┃ Namespace           ┃ Name                                        ┃ Pods ┃ Old Pods ┃ Type        ┃ Container                                   ┃ CPU Diff  ┃ CPU Requests          ┃ CPU Limits    ┃ Memory Diff ┃ Memory Requests            ┃ Memory Limits     ┃
┡━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━┩
│     1. │ llpo                │ llpo-nemo-llpo-workflows-server             │ 1    │ 7        │ Deployment  │ llpo-server                                 │ -190m     │ (-190m) 200m -> 10m   │ unset         │ +27Mi       │ (+27Mi) 450Mi -> 477Mi     │ 450Mi -> 477Mi    │
├────────┼─────────────────────┼─────────────────────────────────────────────┼──────┼──────────┼─────────────┼─────────────────────────────────────────────┼───────────┼───────────────────────┼───────────────┼─────────────┼────────────────────────────┼───────────────────┤
│     2. │ llpo                │ llpo-nemo-llpo-workflows-workflow-controll… │ 1    │ 12       │ Deployment  │ controller                                  │ -108m     │ (-108m) 200m -> 92m   │ unset         │ +838Mi      │ (+838Mi) 600Mi -> 1438Mi   │ 1500Mi -> 1438Mi  │
├────────┼─────────────────────┼─────────────────────────────────────────────┼──────┼──────────┼─────────────┼─────────────────────────────────────────────┼───────────┼───────────────────────┼───────────────┼─────────────┼────────────────────────────┼───────────────────┤
│     3. │ cattle-fleet-system │ fleet-agent                                 │ 1    │ 4        │ Deployment  │ fleet-agent                                 │ +23m      │ (+23m) unset -> 23m   │ unset         │ +507Mi      │ (+507Mi) unset -> 507Mi    │ unset -> 507Mi    │
├────────┼─────────────────────┼─────────────────────────────────────────────┼──────┼──────────┼─────────────┼─────────────────────────────────────────────┼───────────┼───────────────────────┼───────────────┼─────────────┼────────────────────────────┼───────────────────┤
│     4. │ cattle-system       │ cattle-cluster-agent                        │ 2    │ 0        │ Deployment  │ cluster-register                            │ +78m      │ (+39m) unset -> 39m   │ unset         │ +3246Mi     │ (+1623Mi) unset -> 1623Mi  │ unset -> 1623Mi   │
│        │                     │                                             │      │          │             │                                             │ (2 pods)  │                       │               │ (2 pods)    │                            │                   │
├────────┼─────────────────────┼─────────────────────────────────────────────┼──────┼──────────┼─────────────┼─────────────────────────────────────────────┼───────────┼───────────────────────┼───────────────┼─────────────┼────────────────────────────┼───────────────────┤
│     5. │ cattle-system       │ rancher-webhook                             │ 1    │ 2        │ Deployment  │ rancher-webhook                             │ +10m      │ (+10m) unset -> 10m   │ unset         │ +140Mi      │ (+140Mi) unset -> 140Mi    │ unset -> 140Mi    │
├────────┼─────────────────────┼─────────────────────────────────────────────┼──────┼──────────┼─────────────┼─────────────────────────────────────────────┼───────────┼───────────────────────┼───────────────┼─────────────┼────────────────────────────┼───────────────────┤
│     6. │ dev-mmzp            │ heapdump-manager                            │ 1    │ 1        │ Deployment  │ nginx                                       │ +9m       │ (+9m) 1m -> 10m       │ unset         │ +99Mi       │ (+99Mi) 1024Ki -> 100Mi    │ unset -> 100Mi    │
│        │                     │                                             │      │          │             │ cleanup                                     │ +9m       │ (+9m) 1m -> 10m       │ unset         │ +99Mi       │ (+99Mi) 1024Ki -> 100Mi    │ unset -> 100Mi    │
├────────┼─────────────────────┼─────────────────────────────────────────────┼──────┼──────────┼─────────────┼─────────────────────────────────────────────┼───────────┼───────────────────────┼───────────────┼─────────────┼────────────────────────────┼───────────────────┤
└────────┴─────────────────────┴─────────────────────────────────────────────┴──────┴──────────┴─────────────┴─────────────────────────────────────────────┴───────────┴───────────────────────┴───────────────┴─────────────┴────────────────────────────┴───────────────────┘
                                                                                                                                 34 points - D
1 Create a port forward to the prometheus pod
2 Run krr using the tunnel
Get resource recommendations with defined limits
krr simple --cpu_percentile=90 --memory_buffer_percentage=15 --history_duration=24 --timeframe_duration=0.5 -p http://127.0.0.1:9090

CLI Utilities, Prompt and Zsh Integration

Install

  • kctx (for kctx and kns tools)

  • kube-ps1 (for a nice prompt)

e.g.

brew install kctx kube-ps1

add zsh integration and prompt (adjust paths for linux)

cat << EOF >> ~/.zshrc
alias kctx="kubectx"
alias kns="kubens"
source "/opt/homebrew/opt/kube-ps1/share/kube-ps1.sh"
PS1='$(kube_ps1)'$PS1

Source

Find out if Pod belongs to a Deployment, ReplicaSet or StatefulSet

k describe po PODNAME | grep "Controlled By"
Example
$ kubectl describe po nginx-reverse-proxy-5b6c4764d5-fbhq8 | grep "Controlled By"
Controlled By:  ReplicaSet/nginx-reverse-proxy-5b6c4764d5

Get all running pods

kubectl get po --field-selector=status.phase=Running

Merge multiple Configurations

KUBECONFIG=~/.kube/config:~/.kube/myotherconfig

View merged configurations:

kubectl config view

Strimzi Operators for Kafka on Kubernetes Quick Install

kubectl create -f 'https://strimzi.io/install/latest?namespace=kafka'
clusterrole.rbac.authorization.k8s.io/strimzi-kafka-broker created
clusterrole.rbac.authorization.k8s.io/strimzi-cluster-operator-namespaced created
customresourcedefinition.apiextensions.k8s.io/kafkamirrormaker2s.kafka.strimzi.io created
rolebinding.rbac.authorization.k8s.io/strimzi-cluster-operator-leader-election created
customresourcedefinition.apiextensions.k8s.io/kafkaconnectors.kafka.strimzi.io created
customresourcedefinition.apiextensions.k8s.io/kafkabridges.kafka.strimzi.io created
customresourcedefinition.apiextensions.k8s.io/kafkamirrormakers.kafka.strimzi.io created
clusterrolebinding.rbac.authorization.k8s.io/strimzi-cluster-operator-kafka-broker-delegation created
rolebinding.rbac.authorization.k8s.io/strimzi-cluster-operator-watched created
clusterrolebinding.rbac.authorization.k8s.io/strimzi-cluster-operator created
customresourcedefinition.apiextensions.k8s.io/kafkatopics.kafka.strimzi.io created
customresourcedefinition.apiextensions.k8s.io/kafkaconnects.kafka.strimzi.io created
deployment.apps/strimzi-cluster-operator created
clusterrole.rbac.authorization.k8s.io/strimzi-cluster-operator-global created
customresourcedefinition.apiextensions.k8s.io/kafkarebalances.kafka.strimzi.io created
clusterrole.rbac.authorization.k8s.io/strimzi-cluster-operator-leader-election created
clusterrolebinding.rbac.authorization.k8s.io/strimzi-cluster-operator-kafka-client-delegation created
customresourcedefinition.apiextensions.k8s.io/kafkausers.kafka.strimzi.io created
clusterrole.rbac.authorization.k8s.io/strimzi-cluster-operator-watched created
clusterrole.rbac.authorization.k8s.io/strimzi-kafka-client created
configmap/strimzi-cluster-operator created
clusterrole.rbac.authorization.k8s.io/strimzi-entity-operator created
rolebinding.rbac.authorization.k8s.io/strimzi-cluster-operator-entity-operator-delegation created
rolebinding.rbac.authorization.k8s.io/strimzi-cluster-operator created
serviceaccount/strimzi-cluster-operator created
customresourcedefinition.apiextensions.k8s.io/strimzipodsets.core.strimzi.io created
customresourcedefinition.apiextensions.k8s.io/kafkas.kafka.strimzi.io created

View operator’s logs:

kubectl logs deployment/strimzi-cluster-operator -f

Link: Strimzi Homepage: Quickstart

Autoscale a deployment

Horizontal Pod Autoscaler (HPA)

Creates a scaling from 1 to 5 pods when 70% CPU is reached …​

The deployment is named d-ex1 
kubectl autoscale deployment d-ex1 --cpu-percent=70 --min=1 --max=5
List hpas
kubectl get hpa
NAME    REFERENCE          TARGETS         MINPODS   MAXPODS   REPLICAS   AGE
d-ex1   Deployment/d-ex1   <unknown>/70%   1         5         0          4s

Dirty quick Metrics Server install

kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml

e.g.

kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml
serviceaccount/metrics-server created
clusterrole.rbac.authorization.k8s.io/system:aggregated-metrics-reader created
clusterrole.rbac.authorization.k8s.io/system:metrics-server created
rolebinding.rbac.authorization.k8s.io/metrics-server-auth-reader created
clusterrolebinding.rbac.authorization.k8s.io/metrics-server:system:auth-delegator created
clusterrolebinding.rbac.authorization.k8s.io/system:metrics-server created
service/metrics-server created
deployment.apps/metrics-server created
apiservice.apiregistration.k8s.io/v1beta1.metrics.k8s.io created

Depending on the environment, sometimes the installation fails e.g.

k top pods
Error from server (ServiceUnavailable): the server is currently unable to handle the request (get pods.metrics.k8s.io)

Looking that one up…​

kubectl get apiservices | grep metrics
v1beta1.metrics.k8s.io                 kube-system/metrics-server   False (MissingEndpoints)   4m17s

Editing the deployment…​

kubectl edit deployments.apps -n kube-system metrics-server

adding the following options to the spec.containers.args:

      containers:
      - name: metrics-server
        image: k8s.gcr.io/metrics-server/metrics-server:v0.3.7
        imagePullPolicy: IfNotPresent
        args:
          - --cert-dir=/tmp
          - --secure-port=4443
          - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
          - --kubelet-insecure-tls

should then be running …​

kubectl get apiservices | grep metrics
v1beta1.metrics.k8s.io                 kube-system/metrics-server   True        19m

Manage Deployment Rollouts

View Rollout History

kubectl rollout history deployment my-deploy

View Details for specific rollout

kubectl rollout history deployment my-deploy --revision=1

Rollback to specific version

kubectl rollout undo deployment my-deploy --to-revision=1

Create a Pod or Deployment for Debugging

Using a simple Ubuntu Linux image

Creating a deployment
create deployment d-ex1 --image=ubuntu -- "sleep" "300000"
Creating a pod
create pod p-ex1 --image=ubuntu -- "sleep" "300000"

Now we can connnect to the pod ..

kubectl exec -it podname -- /bin/bash

Cluster Info

kubectl cluster-info

Start Container with Port exposed

kubectl run some-name --image=image:label --port=8080

Execute Command in POD

kubectl exec --it PODNAME -- COMMANF

Start HTTP Proxy to Kubernetes API

kubectl proxy

Expose as Kubernetes Service

using a deployment …

kubectl expose deployment/NAME --type="NodePort" --port 8080

Label a pod

kubectl label pod POD_NAME app=v1

Find pod by label

kubectl get pods -l app=v1

Create Autocompletion for ZSH Shell

The kubectl completion script for zsh can be generated using the following command:

kubectl completion zsh

So adding the following line to the ~/.zshrc is all we have to do here:

~/.zshrc
source <(kubectl completion zsh)

Link: Kubernetes Documentation: zsh auto-completion

General Tricks

Don’t type kubectl all the time, use an alias…​.

alias k=kubectl

Delete Service

kubectl delete service -l app=v1

Scale Deployment Replicas

kubectl scale deployments/NAME --replicas=4

Create Deployment from YAML File

kubectl apply -f deployment.yaml

YAML Descriptor

apiVersion: apps/v1beta2
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  selector:
    matchLabels:
      app: nginx
  replicas: 2
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.7.9
        ports:
        - containerPort: 80

Store Information as Environment Variable

export POD_NAME=$(kubectl get pods -o go-template --template '{{range .items}}{{.metadata.name}}{{"\n"}}{{end}}')\necho Name of the Pod: $POD_NAME
export NODE_PORT=$(kubectl get services/kubernetes-bootcamp -o go-template='{{(index .spec.ports 0).nodePort}}')\necho NODE_PORT=$NODE_PORT

Minikube

Starten

minikube start

Stoppen

minikube stop

IP Address

minikube ip

Services

minikube service nginx

Resume File Downloads from Pods

Helps when downloading bigger files from a pod fails 
kubectl cp --retries=10 ...

Common Errors

ImagePullBackoff

Something is wrong with the Docker image configured or its version or the repository or the connection to the repository

CrashLoopBackoff

There is some error in the container, app failing, resources missing, volume mounts missing, watch for the logs, describe pod, events

CreateContainerError

Often there is an issue with wiring secrets or a configmap etc…​

Logs from multiple pods and containers with stern

Stern on GitHub

Installation
brew install stern // or
go install github.com/stern/stern@latest
usage
stern --help
Tail multiple pods and containers from Kubernetes

Usage:
  stern pod-query [flags]

Flags:
  -A, --all-namespaces                  If present, tail across all namespaces. A specific namespace is ignored even if specified with --namespace.
      --color string                    Force set color output. 'auto':  colorize if tty attached, 'always': always colorize, 'never': never colorize. (default "auto")
      --completion string               Output stern command-line completion code for the specified shell. Can be 'bash', 'zsh' or 'fish'.
      --config string                   Path to the stern config file (default "~/.config/stern/config.yaml")
  -c, --container string                Container name when multiple containers in pod. (regular expression) (default ".*")
      --container-state strings         Tail containers with state in running, waiting, terminated, or all. 'all' matches all container states. To specify multiple states, repeat this or set comma-separated value. (default [all])
      --context string                  The name of the kubeconfig context to use
      --ephemeral-containers            Include or exclude ephemeral containers. (default true)
  -e, --exclude stringArray             Log lines to exclude. (regular expression)
  -E, --exclude-container stringArray   Container name to exclude when multiple containers in pod. (regular expression)
      --exclude-pod stringArray         Pod name to exclude. (regular expression)
      --field-selector string           Selector (field query) to filter on. If present, default to ".*" for the pod-query.
  -h, --help                            help for stern
  -H, --highlight stringArray           Log lines to highlight. (regular expression)
  -i, --include stringArray             Log lines to include. (regular expression)
      --init-containers                 Include or exclude init containers. (default true)
      --kubeconfig string               Path to the kubeconfig file to use for CLI requests.
      --max-log-requests int            Maximum number of concurrent logs to request. Defaults to 50, but 5 when specifying --no-follow (default -1)
  -n, --namespace strings               Kubernetes namespace to use. Default to namespace configured in kubernetes context. To specify multiple namespaces, repeat this or set comma-separated value.
      --no-follow                       Exit when all logs have been shown.
      --node string                     Node name to filter on.
      --only-log-lines                  Print only log lines
  -o, --output string                   Specify predefined template. Currently support: [default, raw, json, extjson, ppextjson] (default "default")
  -p, --prompt                          Toggle interactive prompt for selecting 'app.kubernetes.io/instance' label values.
  -l, --selector string                 Selector (label query) to filter on. If present, default to ".*" for the pod-query.
      --show-hidden-options             Print a list of hidden options.
  -s, --since duration                  Return logs newer than a relative duration like 5s, 2m, or 3h. (default 48h0m0s)
      --tail int                        The number of lines from the end of the logs to show. Defaults to -1, showing all logs. (default -1)
      --template string                 Template to use for log lines, leave empty to use --output flag.
  -T, --template-file string            Path to template to use for log lines, leave empty to use --output flag. It overrides --template option.
  -t, --timestamps string[="default"]   Print timestamps with the specified format. One of 'default' or 'short'. If specified but without value, 'default' is used.
      --timezone string                 Set timestamps to specific timezone. (default "Local")
      --verbosity int                   Number of the log level verbosity
  -v, --version                         Print the version and exit.
Example fetching logs of last two hours including xxx and excluding yyy
stern mypod -t --since 2h -i 'xxx' -e 'yyy'