Horizontal Scaling of Network Traffic

The horizontal scaling of ingress network traffic over multiple Kubernetes nodes involves adjusting the number of running instances of your application to handle varying levels of load. This helps preserve the original client IP address forwarded by the Kubernetes ingress controller in the X-Forwarded-For HTTP header.

Ingress NGINX controller configuration

The Ingress NGINX Controller will be installed via Helm using a separate configuration file.

The following example contains a complete configuration file, including parameters and values to customize the installation:

controller:
  nodeSelector: 
    node-role.kubernetes.io/ingress-node: nginx
  replicaCount: 3
  service:
    loadBalancerIP: <static_IP_address>
    annotations:
      cloud.ionos.com/node-selector: node-role.kubernetes.io/ingress=<service_name>
    externalTrafficPolicy: Local
  affinity:
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchExpressions:
          - key: app.kubernetes.io/name
            operator: In
            values:
            - ingress-nginx
        topologyKey: "kubernetes.io/hostname"

The illustration shows the high-level architecture built using IONOS Managed Kubernetes.

Load balancing

The current implementation of the service of type LoadBalancer does not deploy a true load balancer in front of the Kubernetes cluster. Instead, it allocates a static IP address and assigns it to one of the Kubernetes nodes as an additional IP address. This node is, therefore, acting as an ingress node and takes over the role of a load balancer. If the pod of the service is not running on the ingress node, kube-proxy will NAT the traffic to the correct node.

Problem: The NAT operation will replace the original client IP address with an internal node IP address.

Any individual Kubernetes node provides a throughput of up to 2 Gbit/s on the public interface. Scaling beyond that can be achieved by scaling the number of nodes horizontally. Additionally, the service LB IP address must also be distributed horizontally across those nodes. This type of architecture relies on Domain Name System (DNS) load balancing, as all LB IP addresses are added to the DNS record. During name resolution, the client will decide which IP address to connect to.

When using an ingress controller inside a Kubernetes cluster, web services will usually not be exposed as type LoadBalancer, but as type NodePort instead. The ingress controller is the component that will accept client traffic and distribute it inside the cluster. Therefore, usually only the ingress controller service is exposed as type LoadBalancer.

To scale traffic across multiple nodes, multiple LB IP addresses are required, which are then distributed across the available ingress nodes. This can be achieved by creating as many (dummy) services as nodes and IP addresses are required. It is best practice to reserve these IP addresses outside of Kubernetes in the IP Manager so that they are not unassigned when the service is deleted.

5 Gbit/s traffic demand

Let’s assume that our web service demands a throughput of close to 5 Gbit/s. Distributing this across 2 Gbit/s interfaces would require 3 nodes. Each of these nodes requires its own LB IP address, so in addition to the ingress controller service, one needs to deploy 2 additional (dummy) services.

To spread each IP address to a dedicated node, use a node label to assign the LB IP address to: node-role.kubernetes.io/ingress=<service_name>

Note: You can always set labels and annotations via the DCD, API, Terraform, or other DevOps tools.

Pin a Load Balancer IP address

To pin a LB IP address to a dedicated node, follow these steps:

Reserve an IP address in the IP Manager.
Create a node pool of only one node.
Apply the following label to the node:
node-role.kubernetes.io/ingress=<service_name>
Add the following node selector annotation to the service:
annotations.cloud.ionos.com/node-selector: node-role.kubernetes.io/ingress=<service_name>

In the case of our example, reserve 3 IP addresses in the IP Manager. Add these 3 IP addresses to the DNS A-record of your fully qualified domain name. Then, create 3 node pools, each containing only one node, and apply a different ingress node-role label to each node pool. We will call these 3 nodes as ingress nodes.

The first service will be the ingress NGINX controller service. Add the above-mentioned service annotation to it:

controller.service.annotations.cloud.ionos.com/node-selector: node-role.kubernetes.io/ingress=<service_name>

Also, add the static IP address (provided by the IP Manager) to the configuration:

controller.service.loadBalancerIP: <LB_IP_address>

Similarly, 2 additional (dummy) services of type LoadBalancer must be added to spread traffic across 3 nodes. These 2 services must point to the same ingress-nginx deployment, therefore the same ports and selectors of the standard ingress-nginx service are used.

Example

apiVersion: v1
kind: Service
metadata:
  name: ingress-dummy-service-01
  namespace: ingress-nginx
  annotations:
    cloud.ionos.com/node-selector: node-role.kubernetes.io/ingress=dummy-service-01
spec:
  ports:
  - appProtocol: http
    name: http
    nodePort: xyzxy
    port: 80
    protocol: TCP
    targetPort: http
  - appProtocol: https
    name: https
    nodePort: abcdef
    port: 443
    protocol: TCP
    targetPort: https
  selector:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/name: ingress-nginx
  type: LoadBalancer
  loadBalancerIP: <LB_IP_address>
  externalTrafficPolicy: Local

Note:

Make sure to add your specific LB IP address to the manifest.
Notice the service is using the service specific node selector label as annotation.
This spreads 3 IP addresses across 3 different nodes.

To avoid packets being forwarded using Network Address Translation (NAT) to different nodes (thereby lowering performance and losing the original client IP address), each node containing the LB IP address must also run an ingress controller pod. (This could be implemented by using a daemonSet, but this would waste resources on nodes that are not actually acting as ingress nodes.) First of all, as many replicas of the ingress controller as ingress nodes must be created (in our case 3): controller.replicaCount: 3

Then, the Pods must be deployed only on those ingress nodes. This is accomplished by using another node label. For example, node-role.kubernetes.io/ingress-node=nginx. The name and value can be set to any desired string. All 3 nodes must have the same label associated. The ingress controller must now be configured to use this nodeSelector:

controller.nodeSelector.node-role.kubernetes.io/ingress-node: nginx

This limits the nodes on which the Ingress Controller Pods are placed.

For the Ingress Controller Pods to spread across all nodes equally (one pod on each node), a pod antiAffinity must be configured:

controller.affinity:
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchExpressions:
          - key: app.kubernetes.io/name
            operator: In
            values:
            - ingress-nginx
        topologyKey: "kubernetes.io/hostname"

To force Kubernetes to forward traffic only to Pods running on the local node, the externalTrafficPolicy needs to be set to local. This will also guarantee the preservation of the original client IP address. This needs to be configured for the Ingress-NGINX service (controller.service.externalTrafficPolicy: Local) and for the 2 dummy services (see full-service example above).

The actual helm command via which the Ingress-NGINX Controller is deployed is as follows:

helm install ingress-nginx ingress-nginx --repo https://kubernetes.github.io/ingress-nginx --namespace ingress-nginx --create-namespace -f values.yaml

Verification of the Architecture

To verify the setup, ensure that:

DNS load balancing works correctly.
Fully Qualified Domain Name (FQDN) DNS lookup yields three IP addresses.

nslookup whoami.example.com
Non-authoritative answer:
Name:   whoami.example.com
Address: xx.xxx.xxx.xxx
Name:   whoami.example.com
Address: xx.xxx.xxx.xxx
Name:   whoami.example.com
Address: xx.xxx.xxx.xxx

The Whoami web application can be deployed using the following manifests:

kind: Deployment
apiVersion: apps/v1
metadata:
  namespace: default
  name: whoami
  labels:
    app: whoami
spec:
  replicas: 2
  selector:
    matchLabels:
      app: whoami
  template:
    metadata:
      labels:
        app: whoami
    spec:
      containers:
        - name: whoami
          image: containous/whoami
          ports:
            - name: web
              containerPort: 80

---
apiVersion: v1
kind: Service
metadata:
  name: whoami
  namespace: default
spec:
  ports:
    - protocol: TCP
      name: web
      port: 80
  selector:
    app: whoami
  type: NodePort

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-whoami
  namespace: default
spec:
  ingressClassName: nginx
  rules:
  - host: whoami.<your_domain>
    http:
      paths:
      - pathType: Prefix
        path: /
        backend:
          service:
            name: whoami
            port:
             number: 80

Note: Ensure that both Whoami Pods are running, the service is created, and the Ingress returns an external IP address and a hostname.

A curl with the below-mentioned flags to the hostname will show which Load Balancer IP address is used. You need to use the same curl command multiple times to verify connection to all 3 LB IP addresses is possible.

The response from the whoami application will also return the client IP address in the X-Forwarded-For HTTP header. Verify that it is your local public IP address.

curl -svk http://whoami.example.com
* Trying xx.xxx.xxx.xxx:xx...
* Connected to whoami.example.com (xx.xxx.xxx.xxx) port 80 (#0)
> GET / HTTP/1.1
> Host: whoami.example.com
> User-Agent: curl/8.1.2
> Accept: */*
> 
< HTTP/1.1 200 OK
< Date: Tue, 07 Nov 2023 11:19:35 GMT
< Content-Type: text/plain; charset=utf-8
< Content-Length: 450
< Connection: keep-alive
< 
Hostname: whoami-xxxxxxxxx-swn4z
IP: xxx.0.0.x
IP: ::1
IP: xx.xxx.xxx.xxx
IP: xxx0::xxxx:xxxx:xxxx:xxxx
RemoteAddr: xx.xxx.xxx.xx:xxxx
GET / HTTP/1.1
Host: whoami.example.com
User-Agent: curl/8.1.2
Accept: */*
X-Forwarded-For: xx.xxx.xxx.xx
X-Forwarded-Host: whoami.example.com
X-Forwarded-Port: 80
X-Forwarded-Proto: http
X-Forwarded-Scheme: http
X-Real-Ip: xx.xxx.xxx.xx
X-Request-Id: xxxx00xxxxxaa00000e00d0040adxxd00
X-Scheme: http

PreviousAdd Custom CoreDNS Configuration NextOptimize Kubernetes Deployments for Compute Resources and Storage Volumes

Last updated 8 months ago