Coding On Kubernetes

Lately I have been working a lot with the core libraries inside of Kubernetes. As a developer, I highly recommend that anyone who works with Kubernetes take a peek inside. The code feels dense at first, but cracking it leads to a deeper intuition and understanding of what goes on inside of your clusters!

10,000 Foot View

Most standard interaction a developer will have with Kubernetes is through kubectl, which talks to the API Server. The API Server is a control plane component in Kubernetes, and its main job is to mediate requests defining an object or asking about an object's state.

To understand what goes on under the hood, let's start with a basic kubectl command with the output verbosity turned up:

> kubectl get pods -v=6

Among the output is the URL that gets called on the API Server:

I0325 18:08:37.052956   37863 round_trippers.go:553] GET https://127.0.0.1:41197/api/v1/namespaces/kube-system/pods?limit=500 200 OK in 11 milliseconds

Here the object we are interacting with is a resource of type pods. This resource is in the API Group called core/v1. In the api package we can see the struct defining a Pod at k8s.io/api@v0.23.0/core/v1/types.go :

type Pod struct {
   metav1.TypeMeta `json:",inline"`
   // +optional
   metav1.ObjectMeta `json:"metadata,omitempty" protobuf:"bytes,1,opt,name=metadata"`

   // +optional
   Spec PodSpec `json:"spec,omitempty" protobuf:"bytes,2,opt,name=spec"`

   // +optional
   Status PodStatus `json:"status,omitempty" protobuf:"bytes,3,opt,name=status"`
}

The TypeMeta gives the familiar Kind and APIVersion you might recognize from writing a YAML to deploy a Pod to Kubernetes. The ObjectMeta provides tons of metadata about the object itself, including the name and namespace of the Object. Both TypeMeta and ObjectMeta come from the meta package, because they describe metadata common to all objects in Kubernetes. Note also the marshaler tags for protobuf and json on all the fields.

Somewhat confusingly, this struct represents the Kind called Pod, not the resource called pod. So what is the difference? The Kind is the versioned object in the API Group that defines our contract. The resource can be defined as the subpath on the API Server that we hit. Many times there is a one-to-one mapping of Kind to Resource. Sometimes though, multiple Resources map to the same Kind.

Type <> API Group

There are many other types defined in core/v1 in types.go. How does Kubernetes store this information? How does it know which types belong in which API Groups?

Kubernetes maps this using a Scheme. Each combination of API Group, Version, and Kind maps an object uniquely (i.e. apps/v1/Deployment). These Schemes get registered by the packages that define them and the Kinds that they include.

Practically then, this means a couple of things:

to work with objects of a certain Kind, import the package where they (and their Scheme) are defined and you can create them as structs in your code
to create a CustomResourceDefinition (CRD) you write your own package that conforms to this pattern of using a Scheme (you must also implement some interfaces!). If you use a project like KubeBuilder to do this, it helps a lot with the boilerplate.

As an example, to create a Pod Kind in your code...

Import its package: metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
Create the object, using the API Group, Version, and Kind data:

obj := &corev1.Pod{
   TypeMeta: metav1.TypeMeta{
      Kind:       "Pod",
	  APIVersion: "v1",
 },
 ObjectMeta: metav1.ObjectMeta{
      Name:       "podzilla",
      Namespace:  "tools",
 },
 Spec: corev1.PodSpec{},
}

When You Don't Know The Type...

Recently, I was working with some library code that gives you the Resource, API Group, and Version for a Kubernetes object. The method I had to call didn't accept these though. It took in a runtime.Object instead.

runtime.Object is an interface that all structs representing Kinds must implement.

type Object interface {
    GetObjectKind() schema.ObjectKind
    DeepCopyObject() Object
}

GetObjectKind helps figure out the Kind to uniquely identify an object, and DeepCopyObject is necessary for serialization.

To provide something that has GetObjectKind, I need to know the Kind of the object I have. Given a GroupVersionResource (just all 3 variable squished in a struct) you can find out the corresponding Kind using a RESTMapper.

To create an object that satisfies the RESTMapper interface, I will:

create a discovery client
add a cached discovery interface (a caching layer in memory on that client)
instantiate a DeferredDiscoveryRESTMapper (a lazy lookup RESTMapper basically)

RESTMapper and Discovery

This seems daunting at first, but lets unpack it a little bit. The discovery client is able to discover what API groups and resources the API server supports. It can also do basic REST requests, get the server version and parse the Swagger doc on the API server.

If you follow which interfaces embed other interfaces, starting from the DeferredDiscoveryRESTMapper you end up with DiscoveryInterface:

type DiscoveryInterface interface {
   RESTClient() restclient.Interface
   ServerGroupsInterface ServerResourcesInterface ServerVersionInterface OpenAPISchemaInterface
}

The discovery client can store the results of what API groups and resources the API server supports into a cache. That cache can be periodically invalidated and refreshed. Remember, what the API Server offers can change over time!

So we have this client with caching that can ask the API Server what API groups it supports, and we have the GroupVersionResource . What we need is a way to map that GroupVersionResource to a Kind , and this is exactly what RESTMapper does: effectively a mapping of GroupVersionResource to GroupVersionKind.

If you issue kubectl api-resources -v=9 you can see how this mapping is obtained. Each API Group requested on the API Server is consulted and they return APIResourceList objects defining the mappings and what verbs you can use to perform actions against each resource.

{
  "kind": "APIResourceList",
  "groupVersion": "v1",
  "resources": [
    {
      "name": "bindings",
      "singularName": "",
      "namespaced": "true",
      "kind": "Binding",
      "verbs": [
        "create"
      ]
    }
  ]
}         // the rest is elided (pun intended)

GVR <> GVK <> Runtime.Object

Putting this together, we create our RESTMapper so we can get the GVK (GroupVersionKind) from the GVR (GroupVersionResource).

clientset := kubernetes.NewForConfigOrDie(config)
discoveryClient, err := discovery.NewDiscoveryClientForConfig(config)
if err != nil {
   panic(err.Error())
}
memcacheClient := memory.NewMemCacheClient(discoveryClient)
memcacheClient.Invalidate()
restmapper := restmapper.NewDeferredDiscoveryRESTMapper(memcacheClient)

// hard-coded a GVR, but this technique is helpful when you need to handle many
// different `Kinds` uniformly where a `Runtime.Object` is expected
gvr := schema.GroupVersionResource{
   Group:  "rbac.authorization.k8s.io",
 Version:  "v1",
 Resource: "clusterrolebindings",
}
gvk, _ := d.mapper.KindFor(gvr)      // make sure you handle the error ;p

Now that we have the Kind, how can we convert this into a Runtime.Object ? It turns out that we can use an unstructured.Unstructured{}. Unstructured{} exists to handle types when you dont have the corresponding struct that they map to.

The way that Unstructured{} works is kind of clever. Whatever Kind of object you have is effectively a mapping of the members on its struct to their values. If you know all Kinds in Kubernetes implement certain interfaces and have certain members on them, you can satisfy those constraints and store all of the struct's members on a map[string]interface{}.

For example, Unstructured{} implements GetObjectKind() by retrieving the kind key on the struct's map. In my particular case, I had to pass in a Runtime.Object and I knew that the library I used would call GetObjectKind() on it. That's why I set only the API Version and Kind on the struct -- I knew that was all it needed to use the library without it blowing up.

u := &unstructured.Unstructured{}
u.SetAPIVersion("v1")
u.SetKind(gvk.Kind)

Wrap Up

Having used Kubernetes for years, I found the libraries less intuitive than I imagined. After using them a bit I appreciate them more. Hopefully this blog post helps someone else who decides to look under the hood at how it works!

posts · about · home