Managed Kubernetes provides a platform to automate the deployment, scaling, and management of containerized applications. With IONOS Cloud Managed Kubernetes, you can quickly set up Kubernetes clusters and manage Node Pools.

It offers a wide range of features for containerized applications without having to handle the underlying infrastructure details. It is a convenient solution for users who want to leverage the power of Kubernetes without dealing with the operational challenges of managing the cluster themselves.

Product Overview

Managed Kubernetes

DCD How-Tos

Frequently Asked Questions (FAQs)

To get answers to the most commonly encountered questions about Managed Kubernetes, see FAQs.

Managed Kubernetes facilitates the fully automated setup of Kubernetes clusters. Using Managed Kubernetes, several clusters can be quickly and easily deployed. For example, you can use it on the go to set up staging environments and then delete them if required. Managed Kubernetes simplifies and supports the automation of Continuous Integration and Continuous Delivery/Continuous Deployment (CI/CD) pipelines that help in testing and deployment.

IONOS Managed Kubernetes offers the following:

• Automatic updates and security fixes.

• Version and upgrade provisioning.

• Highly available and geo-redundant control plane.

• Full cluster administrator level access to Kubernetes API.

Supported Versions

Both Public and Private Node Pools support the same Kubernetes versions.

Components of Managed Kubernetes

The architecture of Managed Kubernetes includes the following main components that collectively provide a streamlined and efficient environment for deploying, managing, and scaling containerized applications.

• Control Plane: The control plane runs several key components, including the API server, scheduler, and controller manager. It is responsible for managing the cluster and its components, coordinates the scheduling and deployment of applications, monitors the health of the cluster, and enforces desired state management.

• Cluster: A cluster is a group of computing resources that are connected and managed as a single entity. It is the foundation of the Kubernetes platform and provides the environment for deploying, running, and managing containerized applications. Clusters can span multiple node pools that may be provisioned in different virtual data centers and across locations. For example, you can create a cluster consisting of multiple node pools where each pool is in a different location and achieve geo-redundancy. Each cluster consists of a control plane and a set of worker nodes.

• Node: A single (physical or virtual) machine in a cluster is part of the larger Kubernetes ecosystem. Each node is responsible for running containers, which are the encapsulated application units in Kubernetes. These nodes work together to manage and run containerized applications.

• Node Pool: A node pool is a group of nodes within a cluster with the same configuration. Nodes are the compute resources where applications run. All Kubernetes worker nodes are organized in node pools. All nodes within a node pool are identical in setup. The nodes of a pool are provisioned into virtual data centers at a location of your choice, and you can freely specify the properties of all the nodes at once before creation.

• kubectl: The command-line tool for interacting with Kubernetes clusters that serves as a powerful and versatile interface for managing and deploying applications on Kubernetes. With kubectl, you can perform various operations such as creating, updating, and deleting resources in a Kubernetes cluster.

• Kubeconfig: The kubeconfig file is a configuration file used by the Kubernetes command-line tool (kubectl) to authenticate and access a Kubernetes cluster. It contains information about the cluster, user credentials, and other settings.

• etcd: etcd is a distributed key-value store that is used as the primary data store for Kubernetes. It is responsible for storing the configuration data that represents the state of the cluster. This includes information about nodes in the cluster, configurations, and the current status of various resources.

The illustration shows the key components of the Managed Kubernetes.

In IONOS Managed Kubernetes, a Private Node Pool is a dedicated set of nodes within a Kubernetes cluster that is isolated for the exclusive use of a specific user, application, or organization. Private node pools of a cluster are deployed in a private network behind a NAT Gateway to enable connectivity from the nodes to the public internet but not vice-versa.

You can create Kubernetes clusters for Private Node Pools using the Configuration Management Tools or directly using the IONOS Cloud API. By using IONOS Kubernetes clusters for Private Node Pools, you can ensure the network traffic between your nodes and Kubernetes service stays on your private network only.

Features

The key features related to Private Node Pools include:

• Customized Configurations: The ability to customize networking configurations and define subnets provides flexibility to align the infrastructure with user-specific requirements.

• Isolation of Resources: Private Node Pools provide isolation of resources that improves the performance and reduces the risk of interference from external entities. The isolation of resources within a dedicated, private network environment.

• Security: The additional layer of security added by Private Node Pools ensures that nodes are only accessible within a private network. This helps in protecting sensitive data and applications from external threats.

• Scalability: The Private Node Pools are designed to be flexible and scalable based on your needs. This ensures that the resources are utilized efficiently, and you can adapt to varying levels of demand.

In IONOS Managed Kubernetes, a Public Node Pool provides a foundation for hosting applications and services that require external accessibility over the internet. These node pools consist of worker nodes that are exposed to the public network, enabling them to interact with external clients and services.

You can create Kubernetes clusters for Public Node Pools using the Configuration Management Tools or directly using the IONOS Cloud API.

Features

The key features related to Public Node Pools include:

• External Accessibility: Public Node Pools are designed to host workloads that need to be accessed from outside the Kubernetes cluster. This can include web applications, APIs, and other services that require internet connectivity.

• Load Balancing: Load balancers are used with IONOS Public Node Pools to distribute incoming traffic across multiple nodes. This helps to achieve high availability, scalability, and efficient resource utilization.

• Security: The Implementation of proper network policies, firewall rules, and user groups helps IONOS Public Node Pools mitigate potential risks and help in the protection of sensitive data.

• Scaling: The ability to dynamically scale the number of nodes in a Public Node Pool is crucial for handling varying levels of incoming traffic. This scalability ensures optimal performance during peak usage periods.

• Public Cloud Integration: Public Node Pools seamlessly integrate with IONOS Cloud services.

• Monitoring and Logging: Robust monitoring and logging solutions are essential for tracking the performance and health of applications hosted in Public Node Pools. This includes metrics related to traffic, resource utilization, and potential security incidents.

It is desirable to extend CoreDNS with additional configuration to make changes that survive control plane maintenance. It is possible to create a ConfigMap in the kube-system namespace. The ConfigMap must be named coredns-additional-conf and contain a data entry with the key extra.conf. The value of the entry must be a string containing the additional configuration.

The following example shows how to add a custom DNS entry for example.abc:

apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns-additional-conf
  namespace: kube-system
data:
    extra.conf: |
      example.abc:53 {
        hosts {
          1.2.3.4 example.abc
          2.3.4.5 server.example.abc
          fallthrough
        }
      }

You can use the Load Balancer to provide a stable and reliable IP address for your Kubernetes cluster. It will expose your application, such as Nginx deployment, to the internet. This IP address should remain stable as long as the service exists.

Example

Define type as LoadBalancer to create a service of type Load Balancer. When this service is created, most cloud providers will automatically provision a Load Balancer with a stable external IP address. Configure the ports that the service will listen on and forward the traffic to. Define the selector field to set the Pods to which the traffic will be forwarded.

apiVersion: v1
kind: Service
metadata:
  labels:
    run: nginx
  name: nginx
spec:
  ports:
    - port: 80
      targetPort: 80
  selector:
    run: nginx
  type: LoadBalancer
  loadBalancerIP: CRIP-IP

The release schedule outlines the timeline for Kubernetes versions, updates, availability, and the deployment of new features within the Managed Kubernetes environment. It also provides an estimated release and End of Life (EOL) schedule.

The Managed Kubernetes release schedule provides the following information:

• Kubernetes Version: This refers to a specific release of the Kubernetes, which includes updates, enhancements, and bug fixes.

• Kubernetes Release Date: The date when a specific version of the Kubernetes software is released, making it available for users to download and deploy.

• Availability Date: This is an estimate of the version release of the new feature that becomes accessible or ready for use.

• Kubernetes End of Life (EOL): The date when a specific version or release of Kubernetes reaches the end of its official support and maintenance period, after which it no longer receives updates, security patches, or bug fixes from the Kubernetes community or its maintainers. These versions may still be available in the Managed Kubernetes product but will soon be removed from the available versions.

• End of Life (EOL): The point in time when the Managed Kubernetes product reaches the end of its official support period, after which it will no longer receive updates, patches, or technical assistance.

Some applications require a Kubernetes service of type LoadBalancer, which preserves the source IP address of incoming packets. Example: . You can manually integrate a Network Load Balancer (NLB) by exposing and attaching a public IP address to a viable Kubernetes node. This node serves as a load balancer using kube-proxy.

To preserve the client source IP address, Kubernetes services with externalTrafficPolicy: Local need to be used. This configuration ensures that packets reaching a node are only forwarded to Pods that run on the same node, preserving the client source IP address. Therefore, the load balancer IP address of the service needs to be attached to the same node running the ingress controller pod.

This can be achieved with different strategies. One approach is to use a to ensure that a pod is running on each node. However, this approach is feasible only in some cases, and if a cluster has a lot of nodes, then using could lead to a waste of resources.

For an efficient setup, you can schedule Pods to be run only on nodes of a specific node pool using . The node pool needs to have labels that can be used in the node selector. To ensure that the service's load balancer IP is also attached to one of these nodes, annotate the service with cloud.ionos.com/node-selector: key=value, where the key and value are the labels of the node pool.

The following example shows how to install the as a DaemonSet with node selector and to configure the controller service with the required annotation.

1. Create a node pool with a label nodepool=ingress:

2. Create a values.yaml file for later use in the helm command with the following content:

3. Install ingress-nginx via helm using the following command:

Managed Kubernetes can be utilized to address the specific needs of its users. Here, you can find a list of common use cases and scenarios. Each use case is described in detail to highlight its relevance and benefits.

You can optimize the compute resources, such as CPU and RAM, along with storage volumes in Kubernetes through strategic usage of zones.To enhance the performance of your Kubernetes environment, consider implementing a strategic approach for resource allocation. You can intelligently distribute workloads across different zones to improve performance and enhance fault tolerance and resilience.

Example

Define a storage class named ionos-enterprise-ssd-zone-1, which specifies the provisioning of SSD-type storage with ext4 file system format, located in availability zone ZONE_2. Configure the volumeBindingMode and allowVolumeExpansion fields.

This implementation provides a robust and reliable Kubernetes infrastructure for your applications.

Kubernetes is organized in clusters and node pools. The node pools are created in the context of a cluster. The servers belonging to the node pool are provisioned into the . All servers within a node pool are identical in their configuration.

Nodes, also known as worker nodes, are the servers in your data center that are managed by Kubernetes and constitute your node pools. All Resources managed by Kubernetes in your data centers will be displayed by the DCD as read-only.

You can see, inspect, and position the managed resources as per your requirements. However, the specifications of the resources are locked for manual interactions to avoid undesirable results. To modify the managed resources, use the Kubernetes Manager. You can manage the following resource types based on your deployed pods and configurations:

• Servers

The Inspector for Managed Resources allows easy navigation between the data centers, clusters, and node pools in the Kubernetes Manager. Here, you can:

• Switch to the Kubernetes Manager and show the respective node pool.

• Download the kubeconfig to access the cluster.

• List all nodes in the data center belonging to the same node pool.

Cluster Status

All operations related to the infrastructure of clusters can be performed using the Kubernetes Manager, including cluster and node creation and scaling of node pools. The status of a cluster is indicated by different statuses.

Node Pool Status

All operations related to the infrastructure of node pools can be performed using the Kubernetes Manager. The status of a node pool is indicated by different statuses.

To set user privileges using the Cloud API for creating clusters, follow these steps:

1. Authenticate to the Cloud API using your API credentials.

2. Create a user using the POST /cloudapi/v6/um/users endpoint.

3. Set the following required parameters for the user: user's name, email address, and password.

4. Create a group using the POST /cloudapi/v6/um/groups endpoint.

5. Set createK8sCluster privilege to true.

6. Assign the user to the created group using POST /cloudapi/v6/um/groups/{groupId}/users endpoint and provide the user ID in the header.

The horizontal scaling of ingress network traffic over multiple Kubernetes nodes involves adjusting the number of running instances of your application to handle varying levels of load. This helps preserve the original client IP address forwarded by the Kubernetes ingress controller in the X-Forwarded-For HTTP header.

Ingress NGINX controller configuration

The Ingress NGINX Controller will be installed via Helm using a separate configuration file.

The following example contains a complete configuration file, including parameters and values to customize the installation:

controller:
  nodeSelector: 
    node-role.kubernetes.io/ingress-node: nginx
  replicaCount: 3
  service:
    loadBalancerIP: <static_IP_address>
    annotations:
      cloud.ionos.com/node-selector: node-role.kubernetes.io/ingress=<service_name>
    externalTrafficPolicy: Local
  affinity:
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchExpressions:
          - key: app.kubernetes.io/name
            operator: In
            values:
            - ingress-nginx
        topologyKey: "kubernetes.io/hostname"

The illustration shows the high-level architecture built using IONOS Managed Kubernetes.

Load balancing

The current implementation of the service of type LoadBalancer does not deploy a true load balancer in front of the Kubernetes cluster. Instead, it allocates a static IP address and assigns it to one of the Kubernetes nodes as an additional IP address. This node is, therefore, acting as an ingress node and takes over the role of a load balancer. If the pod of the service is not running on the ingress node, kube-proxy will NAT the traffic to the correct node.

Any individual Kubernetes node provides a throughput of up to 2 Gbit/s on the public interface. Scaling beyond that can be achieved by scaling the number of nodes horizontally. Additionally, the service LB IP address must also be distributed horizontally across those nodes. This type of architecture relies on Domain Name System (DNS) load balancing, as all LB IP addresses are added to the DNS record. During name resolution, the client will decide which IP address to connect to.

When using an ingress controller inside a Kubernetes cluster, web services will usually not be exposed as type LoadBalancer, but as type NodePort instead. The ingress controller is the component that will accept client traffic and distribute it inside the cluster. Therefore, usually only the ingress controller service is exposed as type LoadBalancer.

To scale traffic across multiple nodes, multiple LB IP addresses are required, which are then distributed across the available ingress nodes. This can be achieved by creating as many (dummy) services as nodes and IP addresses are required. It is best practice to reserve these IP addresses outside of Kubernetes in the IP Manager so that they are not unassigned when the service is deleted.

5 Gbit/s traffic demand

Let’s assume that our web service demands a throughput of close to 5 Gbit/s. Distributing this across 2 Gbit/s interfaces would require 3 nodes. Each of these nodes requires its own LB IP address, so in addition to the ingress controller service, one needs to deploy 2 additional (dummy) services.

To spread each IP address to a dedicated node, use a node label to assign the LB IP address to: node-role.kubernetes.io/ingress=<service_name>

Pin a Load Balancer IP address

To pin a LB IP address to a dedicated node, follow these steps:

1. Reserve an IP address in the IP Manager.

2. Create a node pool of only one node.

3. Apply the following label to the node:

   node-role.kubernetes.io/ingress=<service_name>

4. Add the following node selector annotation to the service:

   annotations.cloud.ionos.com/node-selector: node-role.kubernetes.io/ingress=<service_name>

In the case of our example, reserve 3 IP addresses in the IP Manager. Add these 3 IP addresses to the DNS A-record of your fully qualified domain name. Then, create 3 node pools, each containing only one node, and apply a different ingress node-role label to each node pool. We will call these 3 nodes as ingress nodes.

The first service will be the ingress NGINX controller service. Add the above-mentioned service annotation to it:

controller.service.annotations.cloud.ionos.com/node-selector: node-role.kubernetes.io/ingress=<service_name>

Also, add the static IP address (provided by the IP Manager) to the configuration:

controller.service.loadBalancerIP: <LB_IP_address>

Similarly, 2 additional (dummy) services of type LoadBalancer must be added to spread traffic across 3 nodes. These 2 services must point to the same ingress-nginx deployment, therefore the same ports and selectors of the standard ingress-nginx service are used.

Example

apiVersion: v1
kind: Service
metadata:
  name: ingress-dummy-service-01
  namespace: ingress-nginx
  annotations:
    cloud.ionos.com/node-selector: node-role.kubernetes.io/ingress=dummy-service-01
spec:
  ports:
  - appProtocol: http
    name: http
    nodePort: xyzxy
    port: 80
    protocol: TCP
    targetPort: http
  - appProtocol: https
    name: https
    nodePort: abcdef
    port: 443
    protocol: TCP
    targetPort: https
  selector:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/name: ingress-nginx
  type: LoadBalancer
  loadBalancerIP: <LB_IP_address>
  externalTrafficPolicy: Local

To avoid packets being forwarded using Network Address Translation (NAT) to different nodes (thereby lowering performance and losing the original client IP address), each node containing the LB IP address must also run an ingress controller pod. (This could be implemented by using a daemonSet, but this would waste resources on nodes that are not actually acting as ingress nodes.) First of all, as many replicas of the ingress controller as ingress nodes must be created (in our case 3): controller.replicaCount: 3

Then, the Pods must be deployed only on those ingress nodes. This is accomplished by using another node label. For example, node-role.kubernetes.io/ingress-node=nginx. The name and value can be set to any desired string. All 3 nodes must have the same label associated. The ingress controller must now be configured to use this nodeSelector:

controller.nodeSelector.node-role.kubernetes.io/ingress-node: nginx

This limits the nodes on which the Ingress Controller Pods are placed.

For the Ingress Controller Pods to spread across all nodes equally (one pod on each node), a pod antiAffinity must be configured:

controller.affinity:
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchExpressions:
          - key: app.kubernetes.io/name
            operator: In
            values:
            - ingress-nginx
        topologyKey: "kubernetes.io/hostname"

To force Kubernetes to forward traffic only to Pods running on the local node, the externalTrafficPolicy needs to be set to local. This will also guarantee the preservation of the original client IP address. This needs to be configured for the Ingress-NGINX service (controller.service.externalTrafficPolicy: Local) and for the 2 dummy services (see full-service example above).

The actual helm command via which the Ingress-NGINX Controller is deployed is as follows:

helm install ingress-nginx ingress-nginx --repo https://kubernetes.github.io/ingress-nginx --namespace ingress-nginx --create-namespace -f values.yaml

Verification of the Architecture

To verify the setup, ensure that:

• DNS load balancing works correctly.

• Fully Qualified Domain Name (FQDN) DNS lookup yields three IP addresses.

nslookup whoami.example.com
Non-authoritative answer:
Name:   whoami.example.com
Address: xx.xxx.xxx.xxx
Name:   whoami.example.com
Address: xx.xxx.xxx.xxx
Name:   whoami.example.com
Address: xx.xxx.xxx.xxx

The Whoami web application can be deployed using the following manifests:

kind: Deployment
apiVersion: apps/v1
metadata:
  namespace: default
  name: whoami
  labels:
    app: whoami
spec:
  replicas: 2
  selector:
    matchLabels:
      app: whoami
  template:
    metadata:
      labels:
        app: whoami
    spec:
      containers:
        - name: whoami
          image: containous/whoami
          ports:
            - name: web
              containerPort: 80

---
apiVersion: v1
kind: Service
metadata:
  name: whoami
  namespace: default
spec:
  ports:
    - protocol: TCP
      name: web
      port: 80
  selector:
    app: whoami
  type: NodePort

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-whoami
  namespace: default
spec:
  ingressClassName: nginx
  rules:
  - host: whoami.<your_domain>
    http:
      paths:
      - pathType: Prefix
        path: /
        backend:
          service:
            name: whoami
            port:
             number: 80

A curl with the below-mentioned flags to the hostname will show which Load Balancer IP address is used. You need to use the same curl command multiple times to verify connection to all 3 LB IP addresses is possible.

The response from the whoami application will also return the client IP address in the X-Forwarded-For HTTP header. Verify that it is your local public IP address.

curl -svk http://whoami.example.com
* Trying xx.xxx.xxx.xxx:xx...
* Connected to whoami.example.com (xx.xxx.xxx.xxx) port 80 (#0)
> GET / HTTP/1.1
> Host: whoami.example.com
> User-Agent: curl/8.1.2
> Accept: */*
> 
< HTTP/1.1 200 OK
< Date: Tue, 07 Nov 2023 11:19:35 GMT
< Content-Type: text/plain; charset=utf-8
< Content-Length: 450
< Connection: keep-alive
< 
Hostname: whoami-xxxxxxxxx-swn4z
IP: xxx.0.0.x
IP: ::1
IP: xx.xxx.xxx.xxx
IP: xxx0::xxxx:xxxx:xxxx:xxxx
RemoteAddr: xx.xxx.xxx.xx:xxxx
GET / HTTP/1.1
Host: whoami.example.com
User-Agent: curl/8.1.2
Accept: */*
X-Forwarded-For: xx.xxx.xxx.xx
X-Forwarded-Host: whoami.example.com
X-Forwarded-Port: 80
X-Forwarded-Proto: http
X-Forwarded-Scheme: http
X-Real-Ip: xx.xxx.xxx.xx
X-Request-Id: xxxx00xxxxxaa00000e00d0040adxxd00
X-Scheme: http

Essential Best Practices for Terraform Users

Update existing node pool configuration

Automatic migration

Existing Kubernetes node pools using AMD Opteron CPUs will be migrated to a “new CPU type” during scheduled maintenance windows.

How To Handle Errors

1. Run terraform plan -refresh-only to identify any remote changes related to your Kubernetes node pool configuration.

2. Check the plan output to confirm the change in the cpu_family attribute.

3. Update your Terraform configuration file (*.tf) to reflect the new CPU type. Ensure the cpu_family parameter aligns with the migrated node pool configuration.

Configure new node pools

When creating new Kubernetes node pools, ensure that the cpu_family parameter does not specify AMD_OPTERON. Instead, set the value to the available cpuFamily in your desired location.

Essential Best Practices for Crossplane Users

Update existing node pool configuration

Reconciliation behavior

Crossplane ignores the cpuFamily parameter during reconciliation for existing Kubernetes node pools. Clients can modify this parameter in the specification file without impacting the operational state of the node pool.

Configuration requirement for new node pools

When defining new Kubernetes node pools, ensure that the cpuFamily parameter in the specification file does not specify AMD_OPTERON. Failure to update this parameter may result in configuration errors during provisioning.

Next steps

Existing Node Pools

If you need to update the cpuFamily parameter for existing node pools, make the necessary changes in your Crossplane specification files as required.

New Node Pools

Ensure the cpuFamily parameter is set appropriately to align with the current CPU offerings during the creation of new Kubernetes node pools.

You can delete a cluster for Public and Private Node Pools with the Kubernetes Manager in DCD.

Managed Kubernetes has a group privilege called Create Kubernetes Clusters. The privilege must be enabled for a group so that the group members inherit this privilege through group privilege settings.

Once the privilege is granted, contract users can create, update, and delete Kubernetes clusters using Managed Kubernetes.

To set user privileges to create Kubernetes clusters, follow these steps:

1. In the DCD, open Management > Users & Groups under Users.

2. Select the Groups tab in the User Manager window.

3. Select the target group name from the Groups list.

4. Select the Create Kubernetes Clusters checkbox in the Privileges tab.

Revoke user privileges

You can revoke a user's Create Kubernetes Clusters privilege by removing the user from all the groups that have this privilege enabled.

To revoke this privilege from a contract administrator, disable the administrator option on the user account. On performing this action, the contract administrator gets the role of a contract user and the privileges that were set up for the user before being an administrator will then be in effect.

You can update a cluster for Public and Private Node Pools with the Kubernetes Manager in DCD.

A kubeconfig file is used to configure access to Kubernetes.

You can delete node pools with the Kubernetes Manager in DCD.

Create a node pool

You can update Public and Private Node Pools with the Kubernetes Manager in DCD.

You can add user groups and assign permissions for Public and Private Node Pools with the Kubernetes Manager in .

All Kubernetes API instructions can be found in the main Cloud API specification file.

To access the Kubernetes API, which the cluster provides, you can download the kubeconfig file and use it with tools such as kubectl.

Retrieve Kubernetes configuration files

GET https://api.ionos.com/cloudapi/v6/k8s/{k8sClusterId}/kubeconfig

Retrieve a configuration file for the specified Kubernetes cluster, in YAML or JSON format as defined in the Accept header; the default Accept header is application/yaml.

Path Parameters

Query Parameters

Headers

All Managed Kubernetes resources, such as clusters and node pools, are subject to an automated weekly maintenance process. All changes to a cluster or node pools that may cause service interruption, such as upgrades, are executed during maintenance. During the maintenance window, you may encounter uncontrolled disconnections and an inability to connect to the cluster.

The upgrade process during maintenance respects the selected Kubernetes version of the cluster. The upgrade process does not upgrade to another Kubernetes major, minor, or patch version unless the current cluster or node pool version reaches its end of life. In such instances, the cluster or node pool will be updated to the next minor version that is active.

Maintenance window configuration

The maintenance window consists of two parts. The first part specifies the day of the week, while the second part specifies the expected time. The following example shows a maintenance window configuration:

For more information, see .

Cluster maintenance

During cluster maintenance, control plane components are upgraded to the newest version available.

Node pool maintenance

During the maintenance of a particular node pool, all nodes of this pool will be replaced by new nodes. Nodes are replaced one after the other, starting with the oldest node. During the node replacement, first, a new node is created and added to the cluster. Then, the old node is drained and removed from the cluster. Node pool upgrade considers the four-hour maintenance window. The upgrade process will be continued in the next node pool maintenance if it fails to upgrade all nodes of the node pool within the four-hour maintenance window.

Node drain and PodDisruptionBudgets (PDBs)

The maintenance process first tries to drain a node gracefully, considering the given PDBs for one hour. If this fails, the node is drained, ignoring pod disruption budgets.

The following steps guide you through the process of connecting with Persistent Volume Claims (PVCs) in a Managed Kubernetes cluster.

Prerequisites:

• Ensure that the NFS volume and the Managed Kubernetes node pool are connected to the same private LAN.

• Node pools can only retrieve their IPs in the private LAN via Dynamic Host Configuration Protocol (DHCP). Each private LAN has its own subnet distributed by the DHCP server.

• The subnet of a private LAN becomes visible via the API when attaching a server to the LAN with a NIC, or by opening a node shell on the Kubernetes Cluster and inspecting the network interfaces.

To connect NFS with PVCs in a Managed Kubernetes cluster via the DCD (Data Center Designer), follow these steps:

1. Drag a vCPU Server into the workspace to add a new server in the DCD.

2. Click Add NIC. This action creates a new Network Inteface Controller (NIC) with a new private LAN. Note the LAN number.

3. Click PROVISION CHANGES.

4. Once the changes are provisioned, inspect the server’s NIC to see its primary IP in the private LAN. For example, 10.7.228.11. This reveals the private LAN’s DHCP subnet. For example, 10.7.228.0/24.

5. Provision a Kubernetes Cluster.

6. Provision a node pool for the Kubernetes Cluster and attach the previously created private LAN. Ensure that DHCP is enabled.

7. Provision an NFS Cluster with the same private LAN Attached.

8. Assign a static IP to the NFS cluster within the same subnet identified earlier. For example, 10.7.228.5/24.

9. Provision an NFS Share. For more information, see .

10. Add the 10.7.228.0/24 subnet and a client group to establish the necessary permissions for Kubernetes and any other hosts within that subnet to access NFS.

11. Delete the vCPU Server created in the first step.

Deploy an NFS Provisioner on Your Kubernetes Cluster

Choose one of the following NFS provisioners:

Both provisioners create a custom StorageClass configured with an NFS server. Use the static IP assigned earlier, for example, 10.7.228.5, as the server and /SHARE_UUID/SHARE_NAME as the share path.

Configure StorageClass for Managed Kubernetes

Managed Kubernetes requires a specific setting for the StorageClass because PersistentVolumes need a specific mount setting. Apply the following command:

Using nfs-subdir-external-provisioner

When creating PVCs, specify the custom StorageClass name. The PVCs should then get provisioned using the specified provisioner.

Using csi-driver-nfs

Install the helm chart for csi-driver-nfs:

Create a StorageClass with the necessary parameters:

Create a PVC:

A node pool upgrade generally happens automatically during weekly maintenance. You can also trigger it manually. For example, when upgrading to a higher version of Kubernetes. In any case, the node pool upgrade will result in rebuilding all nodes belonging to the node pool.

During the upgrade, an old node in a node pool is replaced by a new node. This may be necessary for several reasons:

• Software updates: Since the nodes are considered immutable, IONOS Cloud does not install software updates on the running nodes but replaces them with new ones.

• Configuration changes: Some configuration changes require replacing all included nodes.

Considerations: Multiple node pools of the same cluster can be upgraded at the same time. A node pool upgrade locks the affected node pool, and you cannot make any changes until the upgrade is complete. During a node pool upgrade, all of its nodes are replaced one by one, starting with the oldest one. Depending on the number of nodes and your workload, the upgrade can take several hours.

Rebuild a node

The rebuilding process consists of the following steps:

1. Provision a new node to replace the old one and wait for it to register in the control plane.

2. Exclude the old nodes from scheduling to avoid deploying additional pods to it.

3. Drain all existing workload from the old node.

At first, the IONOS Cloud tries to drain the node gracefully.

- PodDisruptionBudgets (PDBs) are enforced for up to 1 hour. For more information, see Specifying a Disruption Budget for your Application.

- GracefulTerminationPeriod for pods is respected for up to 1 hour. For more information, see Termination of Pods.

If the process takes more than 1 hour, all remaining pods are deleted.

1. Delete the old node from the node pool.

Drain nodes

You need to consider the following node drain updates and their impact on the maintenance procedure:

Under the current platform setup, a node drain considers PDBs. If a concrete eviction of a pod violates an existing PDB, the drain would fail. If the drain of a node fails, the attempt to delete this node will also fail.

The problems with unprepared workloads or misconfigured PDBs can lead to failing drains, node deletions, and resulting failure in node pool maintenance. To prevent this issue, the node drain will split into two stages. In the first stage, the system will continue to try to gracefully evict the pods from the node. If this fails, the second stage will forcefully drain the node by deleting all remaining pods. This deletion will bypass checking PDBs. This prevents nodes from failing during the drain.

How does this affect node pool maintenance?

As a result of the two-stage procedure, the process will stop failing due to unprepared workloads or misconfigured PDBs. However, this change may cause interruptions to workloads that are not prepared for maintenance. During maintenance, nodes are replaced one by one. For each node in a node pool, a new node is created. After that, the old node is drained and then deleted.

At times, a pod is not able to return to its READY state after being evicted from a node during maintenance. In such cases, a PDB was in place for a pod's workload. This leads to failed maintenance, and the rest of the workload is left untouched. With the force drain behavior, the maintenance process will proceed, and all parts of the workload will be evicted and potentially end up in a non-READY state. This might lead to an interruption of the workload. To prevent this, make sure that your workload pods are prepared for eviction at any time.

Following are a few limitations that you can encounter while using both Public and Private Node Pools:

• The maximum number of pods supported per node is 110. It is a Kubernetes default value.

• The recommended maximum number of nodes per node pool is 20.

• The maximum number of nodes in a node pool is 100.

• A total of 500 node pools per cluster is supported.

• The recommended maximum number of node pools per cluster are 50.

• The maximum number of supported nodes per cluster is 5000.

Private Node Pools

Following are a few limitations that you can encounter while using Private Node Pools:

• Managed Kubernetes clusters for Private Node Pools are bound to one region. When you create a node pool, you need to provide a data center, which has to be in the same location as defined in the cluster. You can create Private Node Pools only in a see Virtual Data Center (VDC) that shares the region with the managed Kubernetes cluster.

• Kubernetes services of type LoadBalancer are currently not supported.

• The static node Internet Protocol (IP) addresses are not supported.

What is the function of Managed Kubernetes?

Managed Kubernetes facilitates the fully automated setup of Kubernetes clusters. Managed Kubernetes also simplifies and carefully supports the automation of Continuous Integration and Continuous Delivery/Continuous Deployment (CI/CD) pipelines for testing and deployment.

What does Kubernetes offer for providing transparency and control?

IONOS Managed Kubernetes solution offers automatic updates, security fixes, versioning, upgrade provisioning, high availability, geo-redundant control plane, and full cluster administrator level access to Kubernetes API.

How does the Kubernetes Manager work?

Everything related to Managed Kubernetes can be controlled in the DCD via the dedicated Kubernetes Manager. The Manager provides a complete overview of your provisioned Kubernetes clusters and node pools, including their statuses. The Manager allows you to create and manage clusters, create and manage node pools, and download the kubeconfig file.

What is the control plane for?

The control plane manages the worker nodes and the pods in the cluster. In production environments, the control plane usually runs across multiple computers. A cluster usually runs multiple nodes, providing fault tolerance and high availability.

How does a kube-controller-manager work?

Kube-controller-manager manages controllers that provide functionalities such as deployments, services, etc. For more information, see kube-controller-manager.

What is the function of a kube-apiserver?

The API server is a component of the Kubernetes control plane that exposes the Kubernetes API. The API server is the front end for the Kubernetes control plane. Kube-apiserver is designed to scale horizontally. It scales by deploying more instances. You can run several instances of kube-apiserver and balance traffic between those instances. For more information, see kube-apiserver.

What is the function of a kube-scheduler?

Kube-scheduler distributes pods to nodes. Pods must be created with resource limits so that nodes are not over-commissioned. For more information, see kube-scheduler.

Where are the control plane nodes?

Managed Kubernetes supports regional control planes that provide a distributed and highly-available management infrastructure within a chosen region. It also offers a hidden control plane for both Public and Private Node Pools. Control plane components like kube-apiserver, kube-scheduler, and kube-controller-manager are not visible to the users and cannot be modified directly.

The kube-apiserver can only be interacted with by using its REST API.

The hidden control plane is deployed on Virtual Machines (VMs) running in a geo-redundant cluster in the chosen region. For more information about the control plane components, refer to the Kubernetes documentation. and Kubernetes API.

Which Calico Container Network Interface (CNI) plugin is installed in Managed Kubernetes clusters?

The Managed Kubernetes clusters have a Calico CNI plugin. Its primary function is to automatically assign IP addresses, set up network interfaces, and establish connectivity between the pods. Calico also allows the use of network policies in the Kubernetes cluster. For more information, see Kubernetes CNI and Network Policies.

Can I choose or install a different CNI plugin?

Managed Kubernetes does not currently offer an option to choose a different CNI plugin, nor does it support customers that do so on their own.

CNI affects the whole cluster network, so if changes are made to Calico or a different plugin is installed, it can cause cluster-wide issues and failed resources.

What is a Container Storage Interface (CSI)?

The CSI driver runs as a deployment in the control plane to manage volumes for Persistent Volume Claims (PVCs) in the IONOS Cloud and to attach them to nodes.

How to access Network File Storage (NFS) volumes?

Please refer to the Network File Storage (NFS) Kubernetes integration documentation for more information.

Why do I have leftover IONOS Cloud volumes in my Virtual Data Center (VDC)?

IONOS Cloud volumes are represented as Persistent Volume (PV) resources in Kubernetes. The PV reclaim policy determines what happens to the volume when the PV is deleted. The Retain reclaim policy skips deletion of the volume and is meant for manual reclamation of resources. In the case of dynamically provisioned volumes, the CSI driver manages the PV; the user cannot delete the volume even after the PV is deleted.

The PV has resource finalizers that ensure that Cloud resources are deleted. The finalizers are removed by the system after Cloud resources are cleaned up, so removing them prematurely is likely to leave resources behind.

How can I import unmanaged IONOS Cloud volumes into Kubernetes?

This may be desired when a dynamically provisioned volume is left over or if an external volume should be exposed to Kubernetes workload.

Dynamically provisioned PVs are created by the CSI driver, which populates the resource ownership annotations and information gathered from the IONOS Cloud API. For statically managed PVs, this data must be provided by the user. For statically managed PVs this data must be provided by the user.

Example for static PV spec

kind: PersistentVolume
apiVersion: v1
metadata:
  name: my-import
  annotations:
    pv.kubernetes.io/provisioned-by: cloud.ionos.com
spec:
  storageClassName: ionos-enterprise-hdd
  persistentVolumeReclaimPolicy: Delete
  accessModes:
    - ReadWriteOnce
  capacity:
    storage: 100G
  csi:
    driver: cloud.ionos.com
    fsType: ext4
    volumeHandle: datacenters/27871515-1527-443d-be94-f91b72fd557e/volumes/927c23f4-b7d1-4dd2-b29c-75b1e4b6b166
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: enterprise.cloud.ionos.com/datacenter-id
          operator: In
          values:
          - 27871515-1527-443d-be94-f91b72fd557e

The following fields should be modified according to the volume that is imported:

• spec.capacity.storage: Should contain the size of the volume with suffix G (Gigabyte).

• spec.csi.volumeHandle: Volume path in the IONOS Cloud API. Omit the leading slash (/)

• spec.nodeAffinity.required.nodeSelectorTerms.matchExpressions.values: Must contain the Virtual Data Center ID from the Volume path.

Creating this PV will allow it to be used in Pod by referencing it in a PVC's spec.volumeName.

What is a cluster autoscaler?

A cluster autoscaler is a tool that automatically adjusts the size of the Kubernetes cluster when one of the following conditions is true:

• There are pods that failed to run in the cluster due to insufficient resources.

• There are nodes in the cluster that have been underutilized for an extended period of time, and their pods can be placed on other existing nodes.

For more information, see Cluster Autoscaler and its FAQ.

When does the cluster autoscaler increase and reduce the node pool?

The cluster autoscaler increases the node pool if pods cannot be scheduled due to a lack of resources. In addition, adding a node (from the node pool that needs to be increased) should provide a remedy for the situation. If there are no node pools that provide enough nodes to schedule a pod, the autoscaler will not enlarge.

The cluster autoscaler reduces the node pool if a node is not fully utilized for an extended period of time. A node is underutilized when it has a light load, and all of its pods can be moved to other nodes.

Is it possible to mix node pools with and without active autoscaling within a cluster?

Yes, only node pools with active autoscaling are managed by the autoscaler.

Can the autoscaler enlarge/reduce node pools as required?

No, the autoscaler cannot increase the number of nodes in the node pool above the maximum specified by the user or decrease it below the specified minimum. In addition, the quota for a specific contract cannot be exceeded using the autoscaler. The autoscaler cannot reduce the number of nodes in the node pool to 0.

Is it possible to enable and configure encryption of secret data?

Yes, it is possible to enable and configure encryption of secret data. For more information, see Encrypting Confidential Data at Rest.

Are values and components of the cluster changed during maintenance?

All components installed in the cluster are updated. This includes the K8s control plane itself, CSI, CCM, Calico, and CoreDNS . With cluster maintenance, several components that are visible to customers are updated and reset to our values. For example, changes to CoreDNS are not permanent and will be removed at the next maintenance. It is currently not possible to set your own DNS records in the CoreDNS configuration, but this will be possible later.

Managed components that are regularly updated:

* `coredns`
* `csi-ionoscloud` (DaemonSet)
* `calico` (typha)
* `ionos-policy-validator`
* `snapshot-validaton-webhook`

Is there a limit on the node pool maintenance window?

The maintenance time window is limited to four hours for Public and Private Node Pools. If all of the nodes are not rebuilt within this time, the remaining nodes will be replaced at the next scheduled maintenance. To avoid taking more time for your updates, it is recommended to create node pools with no more than 20 nodes.

How to preset Internet Protocol (IP) addresses on new nodes for Public Node Pools?

If old nodes are replaced with new ones during maintenance, the new nodes will subsequently have different or new public IP addresses. You can pre-specify a list of public IP addresses from which entries for new nodes are taken. In such a way, the list of possible host addresses is limited and predictable. For example, to activate them differently through a whitelist.

Can a cluster and respective node pools have different Kubernetes versions?

The Kubernetes cluster control plane and the corresponding node pools can have different versions of Kubernetes. Node pools can use older versions than the control plane, but not vice versa. The difference between the minor versions must not be more than 1.

There is a distinction between patch version updates and minor version updates. You must initiate all version updates. Once initiated, the version updates are performed immediately. However, forced updates will also occur if the version used by you is so old that we can no longer support it. Typically, affected users receive a support notification two weeks before a forced update.

Is there traffic protection between nodes and the control plane?

The Kubernetes API is secured with Transport Layer Security (TLS). Traffic between the nodes and the control plane is secured by mutual TLS, which means that both sides check whether they are talking to the expected remote station.

Why is the status Failed displayed for clusters or node pools?

If clusters or node pools are created or modified, the operation may fail, and the cluster or node pool will go into a Failed status. In this case, our team is already informed because we monitor it. However, sometimes it can also be difficult for us to rectify the error since the reason can be a conflict with the client's requirements. For example, if a LAN is specified that does not exist at all or no longer exists, a service update becomes impossible.

If the node is in a NotReady state, and there is not enough RAM or the node runs out of the RAM space, an infinite loop occurs in which an attempt is made to free the RAM. This means the node cannot be used, and the executables must be reloaded from the disk. The node is busy with disk Input/Output (I/O). In such a situation, we recommend doing resource management to prevent such scenarios. For more information, see Requests and limits.

Can I publish my own Certificate Authorities (CAs) to the cluster?

The IONOS Kubernetes currently does not support the usage of your own CAs or your own TLS certificates in the Kubernetes cluster.

Is geo-redundancy implemented in Kubernetes for Public Node Pools?

You can reserve Public Node Pools in multiple locations in the same cluster. This allows simple geo-redundancy to be configured and implemented. The control plane is geo-reserved (within Germany). There are several replicas running in different locations.

How to access unresponsive nodes via the API?

If a node is unavailable, like too many pods are running on it without resource limits, it can be replaced. To do this, you can use the following API endpoint:

POST /k8s/{k8sClusterId}/nodepools/{nodePoolId}/nodes/{nodeId}/replace

Why do I need Private Node Pools?

A Private Node Pool ensures that the nodes are not connected directly to the internet; hence, the inter-node network traffic stays inside the private network. However, the control plane is still exposed to the internet and can be protected by restricting IP access.

When do the cluster status or node pools status turn yellow in the DCD?

Clusters and node pools turn yellow when a user or an automated maintenance process initiates an action on the resources. This locks the clusters and node pool resources from being updated until the process is finished, and they do not respond during this time.

Why is a Network Address Translation (NAT) Gateway deployed inside my cluster?

A NAT is required to enable outbound traffic between the cluster nodes and the control plane. For example, to be able to retrieve container images.

Can I attach public networks to my Kubernetes cluster?

Kubernetes clusters only support public networks only if they are VMs but not LAN networks.

Can I add nodes to Kubernetes clusters that consist of Private Node Pools?

Yes, if your node pool is configured to have a network interface in the same network as the VMs that you want to access, then you can add nodes.

How is a Public Node Pool configured within a Kubernetes cluster?

Public Node Pools within a Kubernetes cluster are configured by defining a public dedicated node pool. Networking settings are specified to include public IP addresses for external access.

How is a Private Node Pool configured within a Kubernetes cluster?

Private Node Pools within a Kubernetes cluster are configured by ensuring that each node in a pool has a distinct private network, while nodes within the same pool share a common private network.

It is crucial to set up these node pools with a network interface aligned with the network of the intended VMs when adding nodes to Kubernetes clusters.

Why is a Private Cross Connect deployed inside my VDC?

The Private Cross Connect is required to enable node-to-node communication across all node pools belonging to the same Kubernetes cluster. This ensures that node pools in different VDCs can communicate.

Can I attach an additional private network to the NAT Gateway?

No, the private NAT Gateway is not intended to be used for arbitrary nodes.

Is a service of type LoadBalancer supported, and how do I deploy a service of type LoadBalancer?

The Public Node Pools support the LoadBalancer service type. However, the Private Node Pools currently do not support the LoadBalancer service type.