Yang-Xin Cao (Eason Cao)

Yang-Xin Cao (Eason Cao)

Engineer@Amazon
- AWS All-5 certified / ECS & EKS SME / Marathon Runner / PADI Open Water Diver

[AWS][EKS] Best practice for load balancing - 3. what controller should I use

5 minute read

This article is sharing the best practice for doing load balancing on Amazon EKS, learn what is advantage and disadvantage of using different controller. We will discuss what controller should you use.

Compare different controller options

Here are some common load balancing solutions that can be applied on Amazon EKS:

Kubernetes in-tree load balancer controller

This is the easiest way to provision your Elastic Load Balancer resource, which could be done by using default Kubernetes service deployment with type: LoadBalancer. In most case, the in-tree controller can quickly spin up the load balancer for experiment purpose; or, offers production workload.

However, you need to aware the problem as we mentioned in the previous posts 1 2 because it generally can add a hop for your load balancing behavior on AWS and also can increase the complexity for your traffic.

In addition, you need to aware this method only applies for creating Classic Load Balancer and Network Load Balancer (by using annotation 3).

nginx ingress controller

If you are using nginx Ingress controller in AWS, it will deploy Network load balancer (NLB) to expose the NGINX Ingress controller behind a Service of type=LoadBalancer. Here is an example for deploying Kubernetes service of nginx Ingress controller 1.1.3:

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-backend-protocol: tcp
    service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"
    service.beta.kubernetes.io/aws-load-balancer-type: nlb
  labels:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/version: 1.1.3
  name: ingress-nginx-controller
  namespace: ingress-nginx
spec:
  externalTrafficPolicy: Local
  ports:
  - appProtocol: http
    name: http
    port: 80
    protocol: TCP
    targetPort: http
  - appProtocol: https
    name: https
    port: 443
    protocol: TCP
    targetPort: https
  selector:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/name: ingress-nginx
  type: LoadBalancer

Guess what, yes, it is still can rely on the in-tree controller. On the other hand, the problem we were mentioning can persist. It can be hard to expect which Pods will receive the traffic; however, the main issue is that an Ingress controller does not typically eliminate the need for an external load balancer, it simply adds an additional layer of routing and control behind the load balancer.

An architecture overview of using nginx Ingress controller
Figure 1. An architecture overview of using nginx Ingress controller

So why to choose Nginx Ingress controller? It probably can be the reason why as mentioned in the post 4 as mentioned on the AWS Blog:

  • By default, the NGINX Ingress controller will listen to all the ingress events from all the namespaces and add corresponding directives and rules into the NGINX configuration file. This makes it possible to use a centralized routing file which includes all the ingress rules, hosts, and paths.
  • With the NGINX Ingress controller you can also have multiple ingress objects for multiple environments or namespaces with the same network load balancer.

AWS Load Balancer Controller

AWS Load Balancer Controller is similar to the in-tree Kubernetes controller and use native AWS APIs to provision and manage Elastic Load Balancers. The controller was an open-source project originally named ALB Ingress Controller because it was only provides capability to manage Application Load Balancer at the intial stage, lately, it officially renamed as AWS Load Balancer Controller 5, which is maintaining by AWS product team and open-source community.

Unlike in-tree Kubernetes controller needs to wait the upstream code to be updated, which requires you to upgrade Kubernetes control plane version if the controller has any bug or any new ELB features need to be supported. Using AWS Load Balancer Controller, it can gracefully be replaced because it will be running as Kubernetes deployment instead of relying on Kubernetes upstream source code integration.

The controller directly maintain your Elastic Load Balancer resources with up-to-date annotations. For nginx ingress controller, it can provision and add an extra load balancing layer with the Network Load Balancer, in this case, the traffic generally will pass through the controller itself (nginx-ingress); instead, for AWS Load Balancer Controller, it doesn’t play as a gateway. The AWS Load Balancer Controller will directly control the Elastic Load Balancer resource, which can register your Pod (by using IP mode) so the request can directly forward to your backend application.

The AWS Load Balancer Controller also starts to support TargetGroupBinding 6 and IngressGroup 7 feature since v2.2. It enables you can group multiple Ingress resources together, which allows multiple service deployments can share the same Elastic Load Balancer resource.

Conclusion: What controller should I use?

After comparing different load balancer controllers, generally speaking, using AWS Load Balancer basically can have better feature supports as well as adopt with the performance optimization by configuring AWS Load Balancer attributes correctly. It is essential to enable IP mode when applying the Kubernetes service deployment with AWS Load Balancer Controller to reduce unnecessary hop that can be caused by Kubernetes networking itself, which is generally not totally suitable for AWS networking and elastic load balancing feature.

However, the disadvantage of using AWS Load Balancer can be all features require to be supported by Elastic Load Balancer itself because the controller doesn’t involve additional functions to extend the traffic control. Using other controller still can have its benefit and provide different features that Elastic Load Balancer doesn’t have, such as using nginx Ingress controller you may be able to define forward service to external FastCGI targets, using Regular Expression to perform path matching … etc.

By the end of this article, I hope the comparison and information can better help you understand how to select load balancer controller that will be running in Amazon EKS, and choose the right option for your environment.

Thanks for reading! If you have any feedback or opinions, please feel free to leave the comment below.

References

  1. [AWS][EKS] Best pratice load balancing - Let’s start with an example from Kubernetes document ↩

  2. [AWS][EKS] Best pratice load balancing - imbalanced problem ↩

  3. in-tree controller - Network Load Balancer support on AWS ↩

  4. Using a Network Load Balancer with the NGINX Ingress Controller on Amazon EKS ↩

  5. Introducing the AWS Load Balancer Controller ↩

  6. AWS Load Balancer controller v2.2 - TargetGroupBinding ↩

  7. AWS Load Balancer controller v2.2 - IngressGroup ↩

Leave a comment

You May Also Have Interest

You may also enjoy