Kubernetes

This is yet another container related article. You might ask, do we really need another tool? Why do we need it? Aren't things complicated enough? The answer is not very simple.

Containers as a concept and docker as an implementation have many practical applications, that span from the development cycle up to deployment in production environments.

Actually this one of the main selling points of docker. That you can run in production exactly the same artifact you run in your development environment.

The problem is that in todays world, applications are slightly more complicated than they used to be, we need ways to support auto scaling and high availability not only in isolated components (containers) but on whole environments as well. We also need self-healing and disaster with as much automation as possible. The paradigm of microservices relies in having large numbers of minimal systems, that talk to each other. Without automation it is impossible to manage them, and without monitoring and alerting it is impossible to catch any glitches in time in order to provide highly available services of high quality.

Some of the tools that we have today, cover some of these challenges, but leave others unaddressed.

Kubernetes tries to provide an enterprise level answer to most of these problems, and in this article we will try to take a glimpse of its functionality.

What is Kubernetes

Kubernetes is a system inspired by the Google's Borg and the datacenter as a Computer collection of papers
It is a joint project originally by Google and Linux foundation. Nowadays it is supported by the cloud native computing foundation
It is an open source container orchestration system that is loosely coupled with Google ecosystem.
It supported by Goocle Container Engine
Using GCE is not mandatory. There are alternative configurations:
The backend used during this investigation was Vagrant.

Features

Kubernetes provides tools to manage:

Service clusters
- Deploying and managing workloads
- DNS support
- Load balancing
- Auto scaling
- Health checking
- Service accounts
- Services management both via CLI and UI
Network
A wide variety of local and network based volumes are supported
- Google Compute Engine persistent disk
- AWS Elastic Block StoreAWS Elastic Block Store
- NFS
- RBD (Ceph Block Device)
- Glusterfs

More Kubernetes terminology

The main entity in Kubernetes is not the container. It is a Pod.

Pods are the smallest deployable units that can be created, scheduled, and managed. A pod corresponds to a colocated group of applications running with a shared context.

A pod models an application-specific "logical host" in a containerized environment. It may contain one or more applications which are relatively tightly coupled.

Pods overdose

Pods can be used to host vertically integrated application stacks, but their primary motivation is to support co-located, co-managed helper programs.

Individual pods are not intended to run multiple instances of the same application, in general.

Pods aren't intended to be treated as durable pets. They won't survive scheduling failures, node failures, or other evictions.

Users should almost always use controllers (e.g., replication controller), even for singletons.

Controllers provide self-healing with a cluster scope, as well as replication and rollout management.

The current best practice for pets is to create a replication controller with replicas equal to 1 and a corresponding service.

Kubernetes and networking

Kubernetes tries to solve 4 distinct networking problems:

Highly-coupled container-to-container communications:
- this is solved by pods and localhost communications.
Pod-to-Pod communications:
- See next section.
Pod-to-Service communications: this is covered by services.
External-to-Service communications:
- this is covered by services.

Pod networking properties

Kubernetes assumes that pods can communicate with other pods, regardless of which host they land on.
It provisions every pod its own IP address so you do not need to explicitly create links between pods and you almost never need to deal with mapping container ports to host ports.
This creates a clean, backwards-compatible model where pods can be treated much like VMs or physical hosts from the perspectives of port allocation, naming, service discovery, load balancing, application configuration, and migration.

Pod to pod communication

Kubernetes imposes the following fundamental requirements on any networking implementation (barring any intentional network segmentation policies):

all containers can communicate with all other containers without NAT
all nodes can communicate with all containers (and vice-versa) without NAT
the IP that a container sees itself as is the same IP that others see it as

What this means in practice is that you can not just take two computers running Docker and expect Kubernetes to work. You must ensure that the fundamental requirements are met.

In order to meet the requirements, we need to use software defined networks (SDN) and wire that up to our docker services in all of our nodes

Other popular software switches are openVSwitch, Weave or * Calico

Kubernetes internal architecture

Kubernetes with vagrant backend internally is using saltstack
- That means that there is a master server
- and multiple minions
- Etcd is used to store configuration data that can be used by each of the nodes in the cluster.
- API Server: the main management point of the entire cluster. It also is responsible for making sure that the etcd store and the service details of deployed containers are in agreement. A client called kubecfg is packaged along with the server-side tools and can be used from a local computer or by connecting to the master server.
- Controller Manager Service: used to handle the replication processes defined by replication tasks. The details of these operations are written to etcd, which the controller manager monitors for changes.
Scheduler Server: The process that actually assigns workloads to specific nodes in the cluster is the scheduler.
- Minion Server Components: Minion servers have a few requirements that are necessary to communicate with the master, configure the networking for containers, and run the actual workloads assigned to them.
- Minion services:
  - The first requirement of each individual minion server is docker
  - The main contact point for each minion with the cluster group is through a small service called kubelet. This service is responsible for relaying information to and from the master server, as well as interacting with the etcd store to read configuration details or write new values.
  - In order to deal with individual host subnetting and in order to make services available to external parties, a small proxy service is run on each minion server.

Kubernetes Work Units

Pods: A pod is the basic unit that Kubernetes deals with. Containers themselves are not assigned to hosts. Instead, closely related containers are grouped together in a pod. A pod generally represents one or more containers that should be controlled as a single "application".
Services: We have been using the term "service" throughout this guide in a very loose fashion, but Kubernetes actually has a very specific definition for the word when describing work units. A service, when described this way, is a unit that acts as a basic load balancer and ambassador for other containers. This allows you to deploy a service unit that is aware of all of the backend containers to pass traffic to. Services are an interface to a group of containers so that consumers do not have to worry about anything beyond a single access location.
Replication Controllers: A more complex version of a pod is a replicated pod. These are handled by a type of work unit known as a replication controller. A replication controller is a framework for defining pods that are meant to be horizontally scaled.
Labels: A Kubernetes organizational concept outside of the work-based units is labeling. A label is basically an arbitrary tag that can be placed on the above work units to mark them as a part of a group. These can then be selected for management purposes and action targeting.

Test run with vagrant

Installing kubernetes with vagrant

In order to install and run kubernetes we need to:

export KUBERNETES_PROVIDER=vagrant
curl -sS https://get.k8s.io | bash

It will create a master and a minion based on a fedora box. Kubernetes 1.0.4 was installed. For some reason kubelet.service failed to start a few times, eventually it managed to start. Docker service is also failing in master. After manuall fixing the docker service, salt managed to import all the necessary components for kubernetes (and the api service) to start properly.

The software network switch used by vagrant based kubernetes is openVSwitch.

Kubernetes usage examples

After getting Kubernetes up and running we can interact with it through its command line tools:

Nodes retrieval

[vagrant@kubernetes-master ~]$ kubectl get nodes
NAME         LABELS                              STATUS
10.245.1.3   kubernetes.io/hostname=10.245.1.3   Ready
[vagrant@kubernetes-master ~]$

Cluster info

[vagrant@kubernetes-master ~]$ kubectl cluster-info
Kubernetes master is running at http://localhost:8080
KubeDNS is running at http://localhost:8080/api/v1/proxy/namespaces/kube-system/services/kube-dns
KubeUI is running at http://localhost:8080/api/v1/proxy/namespaces/kube-system/services/kube-ui
[vagrant@kubernetes-master ~]$

Kubernetes services

[vagrant@kubernetes-master ~]$ kubectl get services
NAME         LABELS                                    SELECTOR   IP(S)        PORT(S)
kubernetes   component=apiserver,provider=kubernetes   <none>     10.247.0.1   443/TCP
[vagrant@kubernetes-master ~]$

Setting up Nginx as a kubernetes service

[vagrant@kubernetes-master ~]$ kubectl run my-nginx --image=nginx --replicas=3 --port=80
CONTROLLER   CONTAINER(S)   IMAGE(S)   SELECTOR       REPLICAS
my-nginx     my-nginx       nginx      run=my-nginx   3
[vagrant@kubernetes-master ~]$
[vagrant@kubernetes-master ~]$ kubectl get pods
NAME             READY     STATUS    RESTARTS   AGE
my-nginx-e2r0j   0/1       Pending   0          38s
my-nginx-ippdj   0/1       Pending   0          38s
my-nginx-s0nu7   0/1       Pending   0          38s
[vagrant@kubernetes-master ~]$

Looking at the vm state during service initialization

[vagrant@kubernetes-minion-1 ~]$ sudo docker ps
CONTAINER ID        IMAGE                                            COMMAND                  CREATED              STATUS              PORTS               NAMES
8dade8cd9b7a        nginx                                            "nginx -g 'daemon off"   7 seconds ago        Up 4 seconds                            k8s_my-nginx.f89646ea_my-nginx-ippdj_default_b9cdbd6d-669e-11e5-beed-080027fdddda_b5daab66
3d0e158e20cb        nginx                                            "nginx -g 'daemon off"   7 seconds ago        Up 4 seconds                            k8s_my-nginx.f89646ea_my-nginx-s0nu7_default_b9cc421e-669e-11e5-beed-080027fdddda_5b8962d4
7a72f793bc01        nginx                                            "nginx -g 'daemon off"   7 seconds ago        Up 4 seconds                            k8s_my-nginx.f89646ea_my-nginx-e2r0j_default_b9ce41d5-669e-11e5-beed-080027fdddda_247b2e9d
3662646b2f7b        gcr.io/google_containers/pause:0.8.0             "/pause"                 About a minute ago   Up About a minute                       k8s_POD.ef28e851_my-nginx-ippdj_default_b9cdbd6d-669e-11e5-beed-080027fdddda_3c4744cd
d851e7be10aa        gcr.io/google_containers/pause:0.8.0             "/pause"                 About a minute ago   Up About a minute                       k8s_POD.ef28e851_my-nginx-e2r0j_default_b9ce41d5-669e-11e5-beed-080027fdddda_34da036a
6d3a77f14422        gcr.io/google_containers/pause:0.8.0             "/pause"                 About a minute ago   Up About a minute                       k8s_POD.ef28e851_my-nginx-s0nu7_default_b9cc421e-669e-11e5-beed-080027fdddda_e1f203dc
781ac93991a8        gcr.io/google_containers/exechealthz:1.0         "/exechealthz '-cmd=n"   5 minutes ago        Up 5 minutes                            k8s_healthz.9183a299_kube-dns-v8-hsa7v_kube-system_cc3d0538-669d-11e5-beed-080027fdddda_4708cfdc
3e52ceb86952        gcr.io/google_containers/skydns:2015-03-11-001   "/skydns -machines=ht"   6 minutes ago        Up 5 minutes                            k8s_skydns.b5272dfc_kube-dns-v8-hsa7v_kube-system_cc3d0538-669d-11e5-beed-080027fdddda_da045f37
cae8358a9d37        gcr.io/google_containers/kube2sky:1.11           "/kube2sky -domain=cl"   6 minutes ago        Up 6 minutes                            k8s_kube2sky.68146a83_kube-dns-v8-hsa7v_kube-system_cc3d0538-669d-11e5-beed-080027fdddda_d2edcdf4
7bce50f6e30e        gcr.io/google_containers/kube-ui:v1.1            "/kube-ui"               6 minutes ago        Up 6 minutes                            k8s_kube-ui.593183d8_kube-ui-v1-k7ofv_kube-system_cc391e03-669d-11e5-beed-080027fdddda_0765df4b
93e33799923c        gcr.io/google_containers/etcd:2.0.9              "/usr/local/bin/etcd "   6 minutes ago        Up 6 minutes                            k8s_etcd.7b5ebf3b_kube-dns-v8-hsa7v_kube-system_cc3d0538-669d-11e5-beed-080027fdddda_cb8bff75
c776a62bcbd6        gcr.io/google_containers/pause:0.8.0             "/pause"                 6 minutes ago        Up 6 minutes                            k8s_POD.2688308a_kube-dns-v8-hsa7v_kube-system_cc3d0538-669d-11e5-beed-080027fdddda_a1d945be
8dc82410855c        gcr.io/google_containers/pause:0.8.0             "/pause"                 6 minutes ago        Up 6 minutes                            k8s_POD.3b46e8b9_kube-ui-v1-k7ofv_kube-system_cc391e03-669d-11e5-beed-080027fdddda_c2866347
[vagrant@kubernetes-minion-1 ~]$

As we can see multiple docker containers are started in the minion, some for the nginx pods others for kubernetes health monitoring, service discovery etc

After the services have started

we can retrieve the cluster state from the master:

[vagrant@kubernetes-master ~]$ kubectl get replicationcontrollers
CONTROLLER   CONTAINER(S)   IMAGE(S)   SELECTOR       REPLICAS
my-nginx     my-nginx       nginx      run=my-nginx   3
[vagrant@kubernetes-master ~]$

References

Nick Apostolakis
Tags: kubernetes
December 14th 2015