Kubernetes Custom Resource

Introduction to Kubernetes Custom Resource(CR) in 2023

Custom resources (CR) is an excellent feature introduced in Kubernetes 1.7; with the help of CRD, we can create our own resources in Kubernetes and use them like any other native Kubernetes resource. In this blog post, I am going to talk about how to create a custom resource and will go over:

  • understand resources in Kubernetes
  • What is a custom resource(CR)
  • what is custom resource definition(CRD)
  • In which cases, do we need to create CRD?
  • ways to create a CR
  • we will be creating a new Custom resource.
  • Understand how to delete a CR.
  • A few extra tips.

To follow along, I assume you have prior knowledge of Kubernetes and kubectl. Before going into deep dive, Let’s first understand a few basic terminologies.

Resource in Kubernetes?

resource is an endpoint in Kubernetes API that allows you to store an API object of any kind. For example, the built-in deployment resource contains a collection of deployment objects.

There are many resources available in Kubernetes, some of the examples or Kubernetes resources are:

  • deployment
  • pods
  • replicaSet
  • service

What is a custom resource(CR)

As the name suggests, custom resource(CR) are the resources that are by default not present in the Kubernetes cluster. Therefore, We call them custom resources because we manually create them. By default, custom resources(CR) do not come with native Kubernetes installation. A new resource will be added to the Kubernetes cluster once you create custom resources.

To create a custom resource(CR), you need to have admin access to a Kubernetes cluster. Custom resources can appear and disappear in a running cluster through dynamic registration, and only the admin can update custom resources.

A resource called CRD defaults in Kubernetes, and we will create custom resources(CR) using that resource. Custom resources can be used like any other Kubernetes resource. Hence, once created, these CR can use all features of Kubernetes like security, RBAC, API service, etc.

Why create a custom resource definition(CRD)

As per Kubernetes’ official document, create a CRD if the following requirement satisfies:

  • You want to use Kubernetes client libraries and CLIs to create and update the new resource.
  • You want top-level support from kubectl; for example, kubectl get my-object object-name.
  • You want to build new automation that watches for updates on the new object and then CRUD other objects, or vice versa.
  • You want to write automation that handles updates to the object.
  • You want to use Kubernetes API conventions like .spec, .status, and .metadata.
  • You want the object to be an abstraction over a collection of controlled resources or a summarization of other resources.

Ways to add custom resources

There are two ways to add a custom resource to your cluster.

  • Using CRD
  • Using API Aggregation

CRD is a simple and quite popular way to create a custom resource. You can create a Custom resource(CR) using CRD without any programming language. It allows us to create new types of resources without adding another API server, and You do not need to understand API Aggregation to use CRDs.

On the other hand, Creating custom resources using API Aggregation requires programming, but you will have more control over storage and other attributes.

Before creating a crd, let us first understand Kubernetes API.

What is Kubernetes API

kubernetes api

Kubernetes API is a RESTFul API, that lets you create, update, delete and retrieve primary resources via HTTP. Various resource types in Kubernetes are pods, namespace, services, etc.


A resource type in Kubernetes follows the below pattern:

/apis/GROUP/VERSION/*

List the available API in the existing Kubernetes cluster using the below command:

Check if minikube is running

➜  ~ kubectl version
Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.7", GitCommit:"1dd5338295409edcfff11505e7bb246f0d325d15", GitTreeState:"clean", BuildDate:"2021-01-13T13:23:52Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.2", GitCommit:"faecb196815e248d3ecfb03c680a4507229c2a56", GitTreeState:"clean", BuildDate:"2021-01-13T13:20:00Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"}

start proxy

➜  ~ kubectl proxy
Starting to serve on 127.0.0.1:8001

Open another terminal and curl for all API.

➜ {
  "paths": [
    "/.well-known/openid-configuration",
    "/api",
    "/api/v1",
    "/apis",
    "/apis/",
    "/apis/admissionregistration.k8s.io",
    "/apis/admissionregistration.k8s.io/v1",
    "/apis/admissionregistration.k8s.io/v1beta1",
    "/apis/apiextensions.k8s.io",
    "/apis/apiextensions.k8s.io/v1",
    "/apis/apiextensions.k8s.io/v1beta1",
    "/apis/apiregistration.k8s.io",
    "/apis/apiregistration.k8s.io/v1",
    "/apis/apiregistration.k8s.io/v1beta1",
    "/apis/apps",
    "/apis/apps/v1",
    .......<suppressed output>.......
    .......<suppressed output>.......

By default, all these APIs are present in the Kubernetes cluster. Now let’s proceed and create a new custom resource.

Create simple custom(CR) resources using CRD

We will be using a simple minikube to demonstrate this example. Follow this link to download minikube download minikube on your machine.

Make sure the Kubernetes version is 1.16 or the latest. To check the version, enter the kubectl version

➜  ~ kubectl version
Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.7", GitCommit:"1dd5338295409edcfff11505e7bb246f0d325d15", GitTreeState:"clean", BuildDate:"2021-01-13T13:23:52Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.2", GitCommit:"faecb196815e248d3ecfb03c680a4507229c2a56", GitTreeState:"clean", BuildDate:"2021-01-13T13:20:00Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"}

Now let’s create a CR by using existing apiextensions custom resource definition.

apiVersion: "apiextensions.k8s.io/v1beta1"
kind: "CustomResourceDefinition"
metadata:
  name: "test.simple.io"
spec:
  group: "simple.io"
  version: "v1alpha1"
  scope: "Namespaced"
  names:
    kind: "test"
    plural: "test"
    singular: "test"

The above command will create a custom resource definition for the kind test. metadata. name should be a combination of (spec. group)+(spec.name.kind) spec. The scope should be either Namespaced or Cluster.

Copy the above command in a file simple-cr.yaml and run the below command to create CR.

➜ kubectl create -f simple-cr.yaml
Warning: apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
customresourcedefinition.apiextensions.k8s.io/test.simple.io created

Let’s verify if the CR gets created.

➜  kubectl get crd test.simple.io
NAME             CREATED AT
test.simple.io   2021-04-30T16:09:48Z

You can also check the CR by using an API reference as well.

➜ curl 127.0.0.1:8001 | grep "simple.io"
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  6000    0  6000    0     0   532k      0 --:--:-- --:--:-- --:--:--  532k
    "/apis/simple.io",
    "/apis/simple.io/v1alpha1",

Now Let’s create an object using the above CR.

apiVersion: "simple.io/v1alpha1"
kind: "test"
metadata:
  name: "testing"
spec:
  Key: "abc"
  value : 3

Copy the above code into a file and run the below command to create an object

 ➜  kubectl create -f test-cr.yaml
test.simple.io/testing created

Let’s verify if the object gets created.

➜  kubectl get test
NAME      AGE
testing   2m50s

Here we can see a testing object gets created inside test CR.

Create a custom resource with a schema

Here is the basic syntax of CR definition with schema:

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  # name must match the spec fields below, and be in the form: <plural>.<group>
  name: stable.example.com
spec:
  # group name to use for REST API: /apis/<group>/<version>
  group: example.com
  # list of versions supported by this CustomResourceDefinition
  versions:
    - name: v1
      # Each version can be enabled/disabled by Served flag.
      served: true
      # One and only one version must be marked as the storage version.
      storage: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                header:
                  type: string
                image:
                  type: string
                replicas:
                  type: integer
  # either Namespaced or Cluster
  scope: Namespaced
  names:
    # plural name to be used in the URL: /apis/<group>/<version>/<plural>
    plural: stable
    # singular name to be used as an alias on the CLI and for display
    singular: stable
    # kind is normally the CamelCased singular type. Your resource manifests use this.
    kind: stable

Copy the above command in a file simple-cr-schema.yaml and run the below command to create CR.

➜ kubectl create -f simple-cr-schema.yaml
customresourcedefinition.apiextensions.k8s.io/stable.example.com created

verify if the CR gets created by specifying the name:

➜ kubectl get crd stable.example.com
NAME                 CREATED AT
stable.example.com   2021-05-01T10:14:10Z

now, let’s create a custom object

apiVersion: "example.com/v1"
kind: stable
metadata:
  name: my-new-stable-object2
spec:
  header: "test-header"
  image: "my-image"
  replicas: 3


Copy the above command in a file, simple-object-schema.yaml, and run the below command to create CR.

➜ kubectl create -f simple-object-schema.yaml
stable.example.com/my-new-stable-object created


Verify if the custom object is created:

➜ kubectl get stable
NAME                   AGE
my-new-stable-object   2m1s

Now let’s try to create an object with an incorrect schema and see what happens.

apiVersion: "example.com/v1"
kind: stable
metadata:
  name: my-new-stable-object2
spec:
  header: "test-header"
  value: "my-image"
  replicas: 3

Copy the above command to a file simple-object-incorrect-schema.yaml

➜ kubectl create -f simple-object-incorrect-schema.yaml
error: error validating "simple-object-incorrect-schema.yaml": error validating data: ValidationError(stable.spec): unknown field "value" in com.example.v1.stable.spec; if you choose to ignore these errors, turn validation off with --validate=false

Here we got an error because we have not specified the value field in the CR schema

      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                header:
                  type: string
                image:
                  type: string
                replicas:
                  type: integer

Please modify the schema with the correct fields defined in the custom resource definition to resolve this issue.

How to delete a custom resource(CR)

Deleting a namespace deletes all custom objects in that namespace. Type the below command to delete a CR:

➜ kubectl delete crd test.simple.io
customresourcedefinition.apiextensions.k8s.io "test.simple.io" deleted

Conclusion

I hope you have learned something, and now you have a better understanding of the Kubernetes custom resource. I hope you enjoyed the article. Please let me know in the comment box if you need any help.

Happy Kubernetes exploring.

More to Read

Install Jenkins operator using Kubernetes custom resource

Jenkins Kubernetes operator

Docker vs Kubernetes

Leave a Comment

Your email address will not be published. Required fields are marked *

Scroll to Top