Managed Kubernetes provides a platform to automate the deployment, scaling, and management of containerized applications. With IONOS Cloud Managed Kubernetes, you can quickly set up Kubernetes clusters and manage Node Pools.

It offers a wide range of features for containerized applications without having to handle the underlying infrastructure details. It is a convenient solution for users who want to leverage the power of Kubernetes without dealing with the operational challenges of managing the cluster themselves.

Product Overview

Recommended How-Tos

Kubernetes is organized in clusters and node pools. The node pools are created in the context of a cluster. The servers belonging to the node pool are provisioned into the Virtual Data Center (VDC). All servers within a node pool are identical in their configuration.

Nodes, also known as worker nodes, are the servers in your data center that are managed by Kubernetes and constitute your node pools. All Resources managed by Kubernetes in your data centers will be displayed by the DCD as read-only.

You can see, inspect, and position the managed resources as per your requirements. However, the specifications of the resources are locked for manual interactions to avoid undesirable results. To modify the managed resources, use the Kubernetes Manager. You can manage the following resource types based on your deployed pods and configurations:

• Servers

• Network Interface Card (NIC)

• Local Area Network (LAN)

• IP addresses

The Inspector for Managed Resources allows easy navigation between the data centers, clusters, and node pools in the Kubernetes Manager. Here, you can:

• Switch to the Kubernetes Manager and show the respective node pool.

• Download the kubeconfig to access the cluster.

• List all nodes in the data center belonging to the same node pool.

Cluster Status

All operations related to the infrastructure of clusters can be performed using the Kubernetes Manager, including cluster and node creation and scaling of node pools. The status of a cluster is indicated by different statuses.

Node Pool Status

All operations related to the infrastructure of node pools can be performed using the Kubernetes Manager. The status of a node pool is indicated by different statuses.

In IONOS Managed Kubernetes, a Public Node Pool provides a foundation for hosting applications and services that require external accessibility over the internet. These node pools consist of worker nodes that are exposed to the public network, enabling them to interact with external clients and services.

You can create Kubernetes clusters for Public Node Pools using the Configuration Management Tools or directly using the IONOS Cloud API.

Features

The key features related to Public Node Pools include:

• External Accessibility: Public Node Pools are designed to host workloads that need to be accessed from outside the Kubernetes cluster. This can include web applications, APIs, and other services that require internet connectivity.

• Load Balancing: Load balancers are used with IONOS Public Node Pools to distribute incoming traffic across multiple nodes. This helps to achieve high availability, scalability, and efficient resource utilization.

• Security: The Implementation of proper network policies, firewall rules, and user groups helps IONOS Public Node Pools mitigate potential risks and help in the protection of sensitive data.

• Scaling: The ability to dynamically scale the number of nodes in a Public Node Pool is crucial for handling varying levels of incoming traffic. This scalability ensures optimal performance during peak usage periods.

• Public Cloud Integration: Public Node Pools seamlessly integrate with IONOS Cloud services.

• Monitoring and Logging: Robust monitoring and logging solutions are essential for tracking the performance and health of applications hosted in Public Node Pools. This includes metrics related to traffic, resource utilization, and potential security incidents.

In IONOS Managed Kubernetes, a Private Node Pool is a dedicated set of nodes within a Kubernetes cluster that is isolated for the exclusive use of a specific user, application, or organization. Private node pools of a cluster are deployed in a private network behind a NAT Gateway to enable connectivity from the nodes to the public internet but not vice-versa.

You can create Kubernetes clusters for Private Node Pools using the Configuration Management Tools or directly using the IONOS Cloud API. By using IONOS Kubernetes clusters for Private Node Pools, you can ensure the network traffic between your nodes and Kubernetes service stays on your private network only.

Features

The key features related to Private Node Pools include:

• Customized Configurations: The ability to customize networking configurations and define subnets provides flexibility to align the infrastructure with user-specific requirements.

• Isolation of Resources: Private Node Pools provide isolation of resources that improves the performance and reduces the risk of interference from external entities. The isolation of resources within a dedicated, private network environment.

• Security: The additional layer of security added by Private Node Pools ensures that nodes are only accessible within a private network. This helps in protecting sensitive data and applications from external threats.

• Scalability: The Private Node Pools are designed to be flexible and scalable based on your needs. This ensures that the resources are utilized efficiently, and you can adapt to varying levels of demand.

Managed Kubernetes facilitates the fully automated setup of Kubernetes clusters. Using Managed Kubernetes, several clusters can be quickly and easily deployed. For example, you can use it on the go to set up staging environments and then delete them if required. Managed Kubernetes simplifies and supports the automation of Continuous Integration and Continuous Delivery/Continuous Deployment (CI/CD) pipelines that help in testing and deployment.

IONOS Managed Kubernetes offers the following:

• Automatic updates and security fixes.

• Version and upgrade provisioning.

• Highly available and geo-redundant control plane.

• Full cluster administrator level access to Kubernetes API.

Supported Versions

Both Public and Private Node Pools support the same Kubernetes versions.

Components of Managed Kubernetes

The architecture of Managed Kubernetes includes the following main components that collectively provide a streamlined and efficient environment for deploying, managing, and scaling containerized applications.

• Control Plane: The control plane runs several key components, including the API server, scheduler, and controller manager. It is responsible for managing the cluster and its components, coordinates the scheduling and deployment of applications, monitors the health of the cluster, and enforces desired state management.

• Cluster: A cluster is a group of computing resources that are connected and managed as a single entity. It is the foundation of the Kubernetes platform and provides the environment for deploying, running, and managing containerized applications. Clusters can span multiple node pools that may be provisioned in different virtual data centers and across locations. For example, you can create a cluster consisting of multiple node pools where each pool is in a different location and achieve geo-redundancy. Each cluster consists of a control plane and a set of worker nodes.

• Node: A single (physical or virtual) machine in a cluster is part of the larger Kubernetes ecosystem. Each node is responsible for running containers, which are the encapsulated application units in Kubernetes. These nodes work together to manage and run containerized applications.

• Node Pool: A node pool is a group of nodes within a cluster with the same configuration. Nodes are the compute resources where applications run. All Kubernetes worker nodes are organized in node pools. All nodes within a node pool are identical in setup. The nodes of a pool are provisioned into virtual data centers at a location of your choice, and you can freely specify the properties of all the nodes at once before creation.

• kubectl: The command-line tool for interacting with Kubernetes clusters that serves as a powerful and versatile interface for managing and deploying applications on Kubernetes. With kubectl, you can perform various operations such as creating, updating, and deleting resources in a Kubernetes cluster.

• Kubeconfig: The kubeconfig file is a configuration file used by the Kubernetes command-line tool (kubectl) to authenticate and access a Kubernetes cluster. It contains information about the cluster, user credentials, and other settings.

• etcd: etcd is a distributed key-value store that is used as the primary data store for Kubernetes. It is responsible for storing the configuration data that represents the state of the cluster. This includes information about nodes in the cluster, configurations, and the current status of various resources.

The illustration shows the key components of the Managed Kubernetes.

Managed Kubernetes has a group privilege called Create Kubernetes Clusters. The privilege must be enabled for a group so that the group members inherit this privilege through group privilege settings.

Once the privilege is granted, contract users can create, update, and delete Kubernetes clusters using Managed Kubernetes.

To set user privileges to create Kubernetes clusters, follow these steps:

1. In the DCD, open Management > Users & Groups under Users.

2. Select the Groups tab in the User Manager window.

3. Select the target group name from the Groups list.

4. Select the Create Kubernetes Clusters checkbox in the Privileges tab.

Revoke user privileges

You can revoke a user's Create Kubernetes Clusters privilege by removing the user from all the groups that have this privilege enabled.

To revoke this privilege from a contract administrator, disable the administrator option on the user account. On performing this action, the contract administrator gets the role of a contract user and the privileges that were set up for the user before being an administrator will then be in effect.

Create a cluster

You can delete a cluster for Public and Private Node Pools with the Kubernetes Manager in .

You can update a cluster for Public and Private Node Pools with the Kubernetes Manager in .

You can update Public and Private Node Pools with the Kubernetes Manager in .

A kubeconfig file is used to configure access to Kubernetes.

Create a node pool

To set user privileges using the Cloud API for creating clusters, follow these steps:

1. Authenticate to the Cloud API using your API credentials.

2. Create a user using the POST /cloudapi/v6/um/users endpoint.

3. Set the following required parameters for the user: user's name, email address, and password.

4. Create a group using the POST /cloudapi/v6/um/groups endpoint.

5. Set createK8sCluster privilege to true.

6. Assign the user to the created group using POST /cloudapi/v6/um/groups/{groupId}/users endpoint and provide the user ID in the header.

A node pool upgrade generally happens automatically during weekly maintenance. You can also trigger it manually. For example, when upgrading to a higher version of Kubernetes. In any case, the node pool upgrade will result in rebuilding all nodes belonging to the node pool.

During the upgrade, an old node in a node pool is replaced by a new node. This may be necessary for several reasons:

• Software updates: Since the nodes are considered immutable, IONOS Cloud does not install software updates on the running nodes but replaces them with new ones.

• Configuration changes: Some configuration changes require replacing all included nodes.

Considerations: Multiple node pools of the same cluster can be upgraded at the same time. A node pool upgrade locks the affected node pool, and you cannot make any changes until the upgrade is complete. During a node pool upgrade, all of its nodes are replaced one by one, starting with the oldest one. Depending on the number of nodes and your workload, the upgrade can take several hours.

Rebuild a node

The rebuilding process consists of the following steps:

1. Provision a new node to replace the old one and wait for it to register in the control plane.

2. Exclude the old nodes from scheduling to avoid deploying additional pods to it.

3. Drain all existing workload from the old node.

At first, the IONOS Cloud tries to drain the node gracefully.

- PodDisruptionBudgets (PDBs) are enforced for up to 1 hour. For more information, see Specifying a Disruption Budget for your Application.

- GracefulTerminationPeriod for pods is respected for up to 1 hour. For more information, see Termination of Pods.

If the process takes more than 1 hour, all remaining pods are deleted.

1. Delete the old node from the node pool.

Drain nodes

You need to consider the following node drain updates and their impact on the maintenance procedure:

Under the current platform setup, a node drain considers PDBs. If a concrete eviction of a pod violates an existing PDB, the drain would fail. If the drain of a node fails, the attempt to delete this node will also fail.

The problems with unprepared workloads or misconfigured PDBs can lead to failing drains, node deletions, and resulting failure in node pool maintenance. To prevent this issue, the node drain will split into two stages. In the first stage, the system will continue to try to gracefully evict the pods from the node. If this fails, the second stage will forcefully drain the node by deleting all remaining pods. This deletion will bypass checking PDBs. This prevents nodes from failing during the drain.

How does this affect node pool maintenance?

As a result of the two-stage procedure, the process will stop failing due to unprepared workloads or misconfigured PDBs. However, this change may cause interruptions to workloads that are not prepared for maintenance. During maintenance, nodes are replaced one by one. For each node in a node pool, a new node is created. After that, the old node is drained and then deleted.

At times, a pod is not able to return to its READY state after being evicted from a node during maintenance. In such cases, a PDB was in place for a pod's workload. This leads to failed maintenance, and the rest of the workload is left untouched. With the force drain behavior, the maintenance process will proceed, and all parts of the workload will be evicted and potentially end up in a non-READY state. This might lead to an interruption of the workload. To prevent this, make sure that your workload pods are prepared for eviction at any time.

All Managed Kubernetes resources, such as clusters and node pools, are subject to an automated weekly maintenance process. All changes to a cluster or node pools that may cause service interruption, such as upgrades, are executed during maintenance. During the maintenance window, you may encounter uncontrolled disconnections and an inability to connect to the cluster.

The upgrade process during maintenance respects the selected Kubernetes version of the cluster. The upgrade process does not upgrade to another Kubernetes major, minor, or patch version unless the current cluster or node pool version reaches its end of life. In such instances, the cluster or node pool will be updated to the next minor version that is active.

Maintenance window configuration

The maintenance window consists of two parts. The first part specifies the day of the week, while the second part specifies the expected time. The following example shows a maintenance window configuration:

...
"maintenanceWindow": {
      "time": "15:39:01",
      "dayOfTheWeek": "Friday"
    },
 ...

For more information, see Create a Cluster.

Cluster maintenance

During cluster maintenance, control plane components are upgraded to the newest version available.

Node pool maintenance

During the maintenance of a particular node pool, all nodes of this pool will be replaced by new nodes. Nodes are replaced one after the other, starting with the oldest node. During the node replacement, first, a new node is created and added to the cluster. Then, the old node is drained and removed from the cluster. Node pool upgrade considers the four-hour maintenance window. The upgrade process will be continued in the next node pool maintenance if it fails to upgrade all nodes of the node pool within the four-hour maintenance window.

Node drain and PodDisruptionBudgets (PDBs)

The maintenance process first tries to drain a node gracefully, considering the given PDBs for one hour. If this fails, the node is drained, ignoring pod disruption budgets.

You can delete node pools with the Kubernetes Manager in DCD.

You can add user groups and assign permissions for Public and Private Node Pools with the Kubernetes Manager in DCD.

All Kubernetes API instructions can be found in the main Cloud API specification file.

To access the Kubernetes API, which the cluster provides, you can download the kubeconfig file and use it with tools such as kubectl.

Retrieve Kubernetes configuration files

GET https://api.ionos.com/cloudapi/v6/k8s/{k8sClusterId}/kubeconfig

Retrieve a configuration file for the specified Kubernetes cluster, in YAML or JSON format as defined in the Accept header; the default Accept header is application/yaml.

Path Parameters

Query Parameters

Headers

{}

{
    "httpStatus": 400,
    "messages": {
        "errorCode": "123",
        "message": "Error message example."
    }
}

Scenario 1: Preserve Source Internet Protocol (IP) when using Ingress

Some applications require a Kubernetes service of type LoadBalancer, which preserves the source Internet Protocol (IP) of incoming packets. Example: . You can manually integrate a Network Load Balancer (NLB) by exposing and attaching a public IP to a viable Kubernetes node. This node serves as a load balancer using kube-proxy.

To preserve the client source IP address, Kubernetes services with externalTrafficPolicy: Local need to be used. This configuration ensures that packets reaching a node are only forwarded to pods that run on the same node, preserving the client source IP. Therefore, the load balancer IP address of the service needs to be attached to the same node running the ingress controller pod.

This can be achieved with different strategies. One approach is to use a to ensure that a pod is running on each node. However, this approach is feasible only in some cases, and if a cluster has a lot of nodes, then using could lead to a waste of resources.

For an efficient setup, you can schedule pods to be run only on nodes of a specific node pool using . The node pool needs to have labels that can be used in the node selector. To ensure that the service's load balancer IP is also attached to one of these nodes, annotate the service with cloud.ionos.com/node-selector: key=value, where the key and value are the labels of the node pool.

The following example shows how to install the as a DaemonSet with node selector and to configure the controller service with the required annotation.

1. Create a node pool with a label nodepool=ingress:

2. Create a values.yaml file for later use in the helm command with the following content:

3. Install ingress-nginx via helm using the following command:

Scenario 2: Add Custom CoreDNS Configuration

It is desirable to extend CoreDNS with additional configuration to make changes that survive control plane maintenance. It is possible to create a ConfigMap in the kube-system namespace. The ConfigMap must be named coredns-additional-conf and contain a data entry with the key extra.conf. The value of the entry must be a string containing the additional configuration.

The following example shows how to add a custom DNS entry for example.abc:

Scenario 3: Horizontal scaling of network traffic

The horizontal scaling of ingress network traffic over multiple Kubernetes nodes involves adjusting the number of running instances of your application to handle varying levels of load. This helps preserve the original client IP address forwarded by the Kubernetes ingress controller in the X-Forwarded-For HTTP header.

Ingress NGINX controller configuration

The following example contains complete configuration file including parameters and values to customize the installation:

The illustration shows the high-level architecture built using IONOS Managed Kubernetes.

Load balancing

The current implementation of service of type LoadBalancer does not deploy a true load balancer in front of the Kubernetes cluster. Instead, it allocates a static IP address and assigns it to one of the Kubernetes nodes as an additional IP address. This node is therefore acting as an ingress node and takes over the role of a load balancer. If the pod of the service is not running on the ingress node, kube-proxy will NAT the traffic to the correct node.

Any individual Kubernetes node provides a throughput of up to 2 Gbit/s on the public interface. Scaling beyond that can be achieved by scaling the number of nodes horizontally. Additionally, the service LB IP address must also be distributed horizontally across those nodes. This type of architecture relies on Domain Name System (DNS) load balancing, as all LB IPs are added to the DNS record. During name resolution, the client will decide which IP address to connect to.

When using an ingress controller inside a Kubernetes cluster, web services will usually not be exposed as type LoadBalancer, but as type NodePort instead. The ingress controller is the component that will accept client traffic and distribute it inside the cluster. Therefore, usually only the ingress controller service is exposed as type LoadBalancer.

To scale traffic across multiple nodes, multiple LB IPs are required, which are then distributed across the available ingress nodes. This can be achieved by creating as many (dummy) services as nodes and IP addresses are required. It is best practice to reserve these IP addresses outside of Kubernetes, in the IP Manager, so that they are not unassigned when the service is deleted.

5 Gbit/s traffic demand

Let’s assume that our web service demands a throughput of close to 5 Gbit/s. Distributing this across 2 Gbit/s interfaces would require 3 nodes. Each of these nodes requires its own LB IP address, so in addition to the ingress controller service, one needs to deploy 2 additional (dummy) services.

To spread each IP address to a dedicated node, use a node label to assign the LB IP address to: node-role.kubernetes.io/ingress=<service_name>

Pin a Load Balancer IP address

To pin a LB IP address to a dedicated node, follow these steps:

1. Reserve an IP address in the IP Manager.

2. Create a node pool of only one node.

3. Apply the following label to the node:

   node-role.kubernetes.io/ingress=<service_name>

4. Add the following node selector annotation to the service:

   annotations.cloud.ionos.com/node-selector: node-role.kubernetes.io/ingress=<service_name>

In the case of our example, reserve 3 IP addresses in the IP Manager. Add these 3 IP addresses to the DNS A-record of your fully qualified domain name. Then create 3 node pools, each containing only one node and apply a different ingress node-role label to each node pool. We will call these 3 nodes as ingress nodes.

The first service will be the ingress NGINX controller service. Add the above-mentioned service annotation to it:

controller.service.annotations.cloud.ionos.com/node-selector: node-role.kubernetes.io/ingress=<service_name>

Also add the static IP address (provided by the IP Manager) to the configuration:

controller.service.loadBalancerIP: <LB_IP_address>

Similarly, 2 additional (dummy) services of type LoadBalancer must be added to spread traffic across 3 nodes. These 2 services must point to the same ingress-nginx deployment, therefore the same ports and selectors of the standard ingress-nginx service are used.

Example:

To avoid packets being forwarded using Network Address Translation (NAT) to different nodes (thereby lowering performance and losing the original client IP address), each node containing the LB IP address must also run an ingress controller pod. (This could be implemented by using a daemonSet, but this would waste resources on nodes that are not actually acting as ingress nodes.) First of all, as many replicas of the ingress controller as ingress nodes must be created (in our case 3): controller.replicaCount: 3

Then, the pods must be deployed only on those ingress nodes. This is accomplished by using another node label. For example, node-role.kubernetes.io/ingress-node=nginx. The name and value can be set to any desired string. All 3 nodes must have the same label associated. The ingress controller must now be configured to use this nodeSelector:

controller.nodeSelector.node-role.kubernetes.io/ingress-node: nginx

This limits the nodes on which the ingress controller pods are placed.

For the ingress controller pods to spread across all nodes equally (one pod on each node), a pod antiAffinity must be configured:

To force Kubernetes to forward traffic only to pods running on the local node, the externalTrafficPolicy needs to be set to local. This will also guarantee the preservation of the original client IP address. This needs to be configured for the Ingress-NGINX service (controller.service.externalTrafficPolicy: Local), and for the 2 dummy services (see full-service example above).

The actual helm command via which the Ingress-NGINX Controller is deployed is as follows:

helm install ingress-nginx ingress-nginx --repo https://kubernetes.github.io/ingress-nginx --namespace ingress-nginx --create-namespace -f values.yaml

Verification of the Architecture

To verify the setup, first make sure that DNS load balancing is working correctly. Verify that 3 IP addresses are returned when performing a DNS lookup on your Fully Qualified Domain Name (FQDN).

The Whoami web application can be deployed using the following manifests:

A curl with the below-mentioned flags to the hostname will show which Load Balancer IP address is used. You need to use the same curl command multiple times to verify connection to all 3 LB IP addresses is possible.

The response from the whoami application will also return the client IP address in the X-Forwarded-For HTTP header. Verify that it is your local public IP address.

Following are a few limitations that you can encounter while using both Public and Private Node Pools:

• The maximum number of pods supported per node is 110. It is a Kubernetes default value.

• The recommended maximum number of nodes per node pool is 20.

• The maximum number of nodes in a node pool is 100.

• A total of 500 node pools per cluster is supported.

• The recommended maximum number of node pools per cluster are 50.

• The maximum number of supported nodes per cluster is 5000.

Private Node Pools

Following are a few limitations that you can encounter while using Private Node Pools:

• Managed Kubernetes clusters for Private Node Pools are bound to one region. When you create a node pool, you need to provide a data center, which has to be in the same location as defined in the cluster. You can create Private Node Pools only in a see that shares the region with the managed Kubernetes cluster.

• Kubernetes services of type LoadBalancer are currently not supported.

• The static node Internet Protocol (IP) addresses are not supported.

What is the function of Managed Kubernetes?

Managed Kubernetes facilitates the fully automated setup of Kubernetes clusters. Managed Kubernetes also simplifies and carefully supports the automation of Continuous Integration and Continuous Delivery/Continuous Deployment (CI/CD) pipelines for testing and deployment.

What does Kubernetes offer for providing transparency and control?

IONOS Managed Kubernetes solution offers automatic updates, security fixes, versioning, upgrade provisioning, high availability, geo-redundant control plane, and full cluster administrator level access to Kubernetes API.

How does the Kubernetes Manager work?

Everything related to Managed Kubernetes can be controlled in the DCD via the dedicated Kubernetes Manager. The Manager provides a complete overview of your provisioned Kubernetes clusters and node pools, including their statuses. The Manager allows you to create and manage clusters, create and manage node pools, and download the kubeconfig file.

What is the control plane for?

The control plane manages the worker nodes and the pods in the cluster. In production environments, the control plane usually runs across multiple computers. A cluster usually runs multiple nodes, providing fault tolerance and high availability.

How does a kube-controller-manager work?

Kube-controller-manager manages controllers that provide functionalities such as deployments, services, etc. For more information, see .

What is the function of a kube-apiserver?

The API server is a component of the Kubernetes control plane that exposes the Kubernetes API. The API server is the front end for the Kubernetes control plane. Kube-apiserver is designed to scale horizontally. It scales by deploying more instances. You can run several instances of kube-apiserver and balance traffic between those instances. For more information, see .

What is the function of a kube-scheduler?

Where are the control plane nodes?

Managed Kubernetes offers a hidden control plane for both Public and Private Node Pools. Control plane components like kube-apiserver, kube-scheduler, and kube-controller-manager are not visible to the users and cannot be modified directly.

The kube-apiserver can only be interacted with by using its REST API.

Which Calico Container Network Interface (CNI) plugin is installed in Managed Kubernetes clusters?

Can I choose or install a different CNI plugin?

Managed Kubernetes does not currently offer an option to choose a different CNI plugin, nor does it support customers that do so on their own.

CNI affects the whole cluster network, so if changes are made to Calico or a different plugin is installed, it can cause cluster-wide issues and failed resources.

The CSI driver runs as deployment in the control plane to manage volumes for Persistent Volume Claims (PVCs) in the IONOS Cloud and to attach them to nodes.

Mount options in the PersistentVolume specification can also be set using the StorageClass.

Example for PV spec:

Example for annotation:

Example for using StorageClass:

What is a cluster autoscaler?

A cluster autoscaler is a tool that automatically adjusts the size of the Kubernetes cluster when one of the following conditions is true:

• There are pods that failed to run in the cluster due to insufficient resources.

• There are nodes in the cluster that have been underutilized for an extended period of time, and their pods can be placed on other existing nodes.

When does the cluster autoscaler increase and reduce the node pool?

The cluster autoscaler increases the node pool if pods cannot be scheduled due to a lack of resources. In addition, adding a node (from the node pool that needs to be increased) should provide a remedy for the situation. If there are no node pools that provide enough nodes to schedule a pod, the autoscaler will not enlarge.

The cluster autoscaler reduces the node pool if a node is not fully utilized for an extended period of time. A node is underutilized when it has a light load, and all of its pods can be moved to other nodes.

Is it possible to mix node pools with and without active autoscaling within a cluster?

Yes, only node pools with active autoscaling are managed by the autoscaler.

Can the autoscaler enlarge/reduce node pools as required?

No, the autoscaler cannot increase the number of nodes in the node pool above the maximum specified by the user or decrease it below the specified minimum. In addition, the quota for a specific contract cannot be exceeded using the autoscaler. The autoscaler cannot reduce the number of nodes in the node pool to 0.

Is it possible to enable and configure encryption of secret data?

Are values and components of the cluster changed during maintenance?

All components installed in the cluster are updated. This includes the K8s control plane itself, CSI, CCM, Calico, and CoreDNS . With cluster maintenance, several components that are visible to customers are updated and reset to our values. For example, changes to CoreDNS are not permanent and will be removed at the next maintenance. It is currently not possible to set your own DNS records in the CoreDNS configuration, but this will be possible later.

Managed components that are regularly updated:

Is there a limit on the node pool maintenance window?

The maintenance time window is limited to four hours for Public and Private Node Pools. If all of the nodes are not rebuilt within this time, the remaining nodes will be replaced at the next scheduled maintenance. To avoid taking more time for your updates, it is recommended to create node pools with no more than 20 nodes.

How to preset Internet Protocol (IP) addresses on new nodes for Public Node Pools?

If old nodes are replaced with new ones during maintenance, the new nodes will subsequently have different or new public IP addresses. You can pre-specify a list of public IP addresses from which entries for new nodes are taken. In such a way, the list of possible host addresses is limited and predictable. For example, to activate them differently through a whitelist.

Can a cluster and respective node pools have different Kubernetes versions?

The Kubernetes cluster control plane and the corresponding node pools can have different versions of Kubernetes. Node pools can use older versions than the control plane, but not vice versa. The difference between the minor versions must not be more than 1.

There is a distinction between patch version updates and minor version updates. You must initiate all version updates. Once initiated, the version updates are performed immediately. However, forced updates will also occur if the version used by you is so old that we can no longer support it. Typically, affected users receive a support notification two weeks before a forced update.

Is there traffic protection between nodes and the control plane?

The Kubernetes API is secured with Transport Layer Security (TLS). Traffic between the nodes and the control plane is secured by mutual TLS, which means that both sides check whether they are talking to the expected remote station.

Why is the status Failed displayed for clusters or node pools?

If clusters or node pools are created or modified, the operation may fail, and the cluster or node pool will go into a Failed status. In this case, our team is already informed because we monitor it. However, sometimes it can also be difficult for us to rectify the error since the reason can be a conflict with the client's requirements. For example, if a LAN is specified that does not exist at all or no longer exists, a service update becomes impossible.

Can I publish my own Certificate Authorities (CAs) to the cluster?

The IONOS Kubernetes currently does not support the usage of your own CAs or your own TLS certificates in the Kubernetes cluster.

Is geo-redundancy implemented in Kubernetes for Public Node Pools?

You can reserve Public Node Pools in multiple locations in the same cluster. This allows simple geo-redundancy to be configured and implemented. The control plane is geo-reserved (within Germany). There are several replicas running in different locations.

How to access unresponsive nodes via the API?

If a node is unavailable, like too many pods are running on it without resource limits, it can be replaced. To do this, you can use the following API endpoint:

Why do I need Private Node Pools?

A Private Node Pool ensures that the nodes are not connected directly to the internet; hence, the inter-node network traffic stays inside the private network. However, the control plane is still exposed to the internet and can be protected by restricting IP access.

When do the cluster status or node pools status turn yellow in the DCD?

Clusters and node pools turn yellow when a user or an automated maintenance process initiates an action on the resources. This locks the clusters and node pool resources from being updated until the process is finished, and they do not respond during this time.

Can I attach public networks to my Kubernetes cluster?

Kubernetes clusters only support public networks only if they are VMs but not LAN networks.

Can I add nodes to Kubernetes clusters that consist of Private Node Pools?

Yes, if your node pool is configured to have a network interface in the same network as the VMs that you want to access, then you can add nodes.

How is a Public Node Pool configured within a Kubernetes cluster?

Public Node Pools within a Kubernetes cluster are configured by defining a public dedicated node pool. Networking settings are specified to include public IP addresses for external access.

How is a Private Node Pool configured within a Kubernetes cluster?

Private Node Pools within a Kubernetes cluster are configured by ensuring that each node in a pool has a distinct private network, while nodes within the same pool share a common private network.

It is crucial to set up these node pools with a network interface aligned with the network of the intended VMs when adding nodes to Kubernetes clusters.

The Private Cross Connect is required to enable node-to-node communication across all node pools belonging to the same Kubernetes cluster. This ensures that node pools in different VDCs can communicate.

No, the private NAT Gateway is not intended to be used for arbitrary nodes.

Is a service of type LoadBalancer supported, and how do I deploy a service of type LoadBalancer?

The Public Node Pools support the LoadBalancer service type. However, the Private Node Pools currently do not support the LoadBalancer service type.