From Kubernetes Ingress to Kubernetes Gateway API

If you've ever touched on application networking in Kubernetes, it's more than likely you've come across Ingress. However, it is worth knowing that Ingress has a worthy successor in the form of Kubernetes’ Gateway API. If you want to get familiar with this new API, this article is what you need.

Kubernetes Ingress - lessons learned

Ingress is a Kubernetes API object that has been widely used for many years. It allows you to handle traffic entering the Kubernetes cluster from outside and to route it to multiple Services running in the cluster. This is sometimes called north-south traffic. Underneath, Ingress is a kind of proxy that performs L7 routing.

Ingress itself is quite simple and therefore a bit limited in terms of the functionalities it can offer. It can only handle HTTP(S) traffic. The routing it performs is based solely on host and/or path matching. Therefore, to serve more demanding scenarios, controllers from different vendors offer a whole range of additional functionalities beyond the native version of Ingress. They are provided in the form of CRDs (Custom Resource Definitions) and annotations. What is interesting is that they often appear to be quite similar to each other, though they differ in syntax.

Therefore, in 2019, discussions began on the evolution of Ingress and the need for a new API of this kind. One of the goals for such an API was to deliver most of the additional capabilities as part of the common core API standard. This resulted in the initiative which today is called Gateway API and which has been developed by Kubernetes’ SIG-NETWORK (Special Interest Group - Network).

Kubernetes Gateway API concepts

The Gateway API provides a collection of resources among which three main object types can be indicated:

GatewayClass
Gateway
xRoutes (e.g. HTTPRoute, TCPRoute, GRPCRoute, etc.)

Gateway is a logical representation of the underlying network infrastructure that processes the traffic (e.g. load balancer, proxy, etc.). Gateway, however, only offers a basic configuration. More granular configurations are defined within xRoutes that are attached to a given Gateway instance. Moreover, xRoutes can be configured separately by different users in the cluster. It is therefore a kind of decentralized configuration model. In short, it can be said that Gateway and xRoutes together perform the same role that Ingress performs as a single resource. GatewayClass, in turn, is something similar to the previous IngressClass. It indicates a controller that implements the functionality defined within the Gateway API.

Roles in the Gateway API

Such an approach may not seem entirely clear at first. It does become more justified, however, when we discover that the intention of the Gateway API designers was to create a resource model that would allow different roles to be distinguished within the configuration process. Which roles did they have in mind, exactly? Well, there are three primary roles defined in the Gateway API:

Infrastructure Provider - responsible for Kubernetes cluster provisioning
Cluster Operator - responsible for managing the cluster once it has been provisioned (this includes managing policies, permissions, etc.)
Application Developer - responsible for application configuration and Service composition

Configuring resources

When it comes to configuring the resources defined within the Gateway API their tasks would be as follows:

Infrastructure Provider creates GatewayClasses (they describe types of load-balancing/proxying infrastructure that can be provisioned in the cluster).
Cluster Operator creates Gateway instances derived from GatewayClasses (defined by the Infrastructure Provider).
Application Developer creates xRoutes (attached to existing Gateways) in order to define how traffic should be routed to applications.

Functionality & implementations

As mentioned, the Gateway API covers a much wider set of functionalities than Ingress. First, it provides support for various L4-L7 protocols, not just HTTP(S) as Ingress does. For each protocol a dedicated xRoute resource is defined (offering specific capabilities):

HTTPRoute (for HTTP)
GRPCRoute (for gRPC)
TLSRoute (for TLS)
TCPRoute (for TCP)
UDPRoute (for UDP)

Second, compared to Ingress, the set of HTTP(S) functionalities is also much richer. In particular, the following features are covered:

path-based matching
header-based matching
query parameters-based matching
method-based matching
request and response headers modification
request mirroring
request REDIRECT
URL rewrite

As was the case with Ingress, the Gateway API does not provide any default controller implementation built into Kubernetes. You are free to choose one of several available implementations. It is no big surprise that these are usually implementations offered by the same vendors that provide Ingress controllers or service mesh solutions. The list of existing implementations and their status can be found here . You should also know that individual implementations generally do not have to support literally all the features that are defined within the Gateway API. This is because so-called support levels have been introduced. They are assigned to individual features. Gateway API offers three support levels:

Core - features assigned to this category will be portable and they are supposed to be supported by all implementations eventually.
Extended - these features will also be portable but support is not mandatory.
Implementation-specific - these features will not be portable and are vendor-specific.

Ebook Application networking in Kubernetes

Example use case: Ingress vs. Gateway API

To understand what the Gateway API can offer, let’s consider an example use case. Imagine you are building a system for observability. It is a modular application where its individual components perform different tasks. You want to deploy this system in Kubernetes. Two application modules are supposed to be accessible from the public network. The first is a module for graphical data visualization, offering tools for statistical data analysis and data presentation (in the form of various types of charts and dashboards). The second module is the API to the data collector. This way, access to raw data is ensured as well. Here's how you can configure this using the well-known Kubernetes Ingress object.

Configuration of Kubernetes Ingress

Fig.1: Kubernetes Ingress-based configuration scheme

We assume that the routing to two application modules reachable from outside the cluster (e.g. from the Internet) takes place at the HTTP layer and is based on URL paths. For the "/data" prefix, HTTP requests to "my.analytics.example.com" will be forwarded to Pods representing the data collector's API module. For the "/visualize" prefix, the requests will go to Pods responsible for data presentation and graphical visualization. Of course, we assume that each Pod group has a dedicated Kubernetes Service instance created. The Ingress instance uses these Services (.spec.rules[].http.paths[].backend.service) to forward traffic to these Pod groups. The complete Ingress configuration is shown below.

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
 name: ingress1
spec:
 rules:
 - host: my.analytics.example.com
   http:
     paths:
     - path: /data
       pathType: Prefix
       backend:
         service:
           name: data
           port:
             number: 8000
     - path: /visualize
       pathType: Prefix
       backend:
         service:
           name: visualize
           port:
             number: 9000

It should also be remembered that configuration at the Ingress level is responsible only for HTTP routing within the cluster. It does not make individual resources deployed in the cluster automatically available from the Internet. For this, additional network mechanisms are needed. But we will not go into detail about them since they may differ depending on the environment in which the Kubernetes cluster is running (cluster in the cloud vs. on-premise cluster).

Now let’s use the Kubernetes Gateway API to reflect the above configuration.

Configuration of Kubernetes Gateway API

Fig.2: Kubernetes Gateway API-based configuration scheme

The starting point is to create a GatewayClass to indicate which controller will be used underneath. Next, in the context of traffic processing, you need to use two objects defined within the Gateway API. The first is a Gateway instance listening for a certain type of traffic. Within its configuration, you define a single listener that listens to HTTP traffic on port 80. Next, you need to create an HTTPRoute instance. You specify a single entry "my.analytics.example.com" under the .spec.hostnames[] array and then you provide a set of rules (under .spec.rules[]) for HTTP requests matching such a host name. Similar to Ingress, you also configure path-based HTTP routing here. You can refer to individual groups of Pods (representing modules of your application) through the Service instances corresponding to them. You indicate these under .spec.rules[].backendRefs[]. Once you have this part of the configuration ready, all you have to do is to attach your HTTPRoute instance to one of the existing Gateways. You can do this in the .spec.parentRefs[] section where you point to the name of the Gateway you created. The entire configuration is presented below.

---
kind: GatewayClass
apiVersion: gateway.networking.k8s.io/v1beta1
metadata:
 name: my-controller
spec:
 controllerName: my.company.example.io/my-gateway-controller
 
 
---
kind: Gateway
apiVersion: gateway.networking.k8s.io/v1beta1
metadata:
 name: web-gateway
spec:
 gatewayClassName: my-controller
 listeners:
   - name: http
     protocol: HTTP
     port: 80
     allowedRoutes:
       namespaces:
         from: All
 
---
kind: HTTPRoute
apiVersion: gateway.networking.k8s.io/v1beta1
metadata:
 name: httproute-basic
 labels:
   scenario: basic
spec:
 parentRefs:
 - group: gateway.networking.k8s.io
   kind: Gateway
   name: web-gateway
 hostnames:
 - "my.analytics.example.com"
 rules:
 - matches:
   - path:
       type: PathPrefix
       value: /data
   backendRefs:
   - name: data
     port: 8000
 - matches:
   - path:
       type: PathPrefix
       value: /visualize
   backendRefs:
   - name: visualize
     port: 9000

A more demanding scenario

At this point, you might be thinking: it's nice that I was able to use the Gateway API for my application, but why do I actually need it when I can accomplish exactly the same using Ingress? Well, you're basically right. However, you must remember that your scenario was relatively simple. All you needed was the efficient path-based HTTP routing that Ingress offers out of the box.

Now think about a slightly more complex scenario. Suppose you want to deploy a brand new version of the data collector API within your application. It is supposed to be an extended version of this module, which is divided into sub-modules, each being a separate API for metrics, traces and logs respectively. At the same time, you want to keep offering your users the old version of this module. But you want to limit its functionality a bit. First, you decide that they will only be able to access certain subsets of the metrics, not the complete data. Second, you don't want the API to allow users to perform CRUD operations, but instead offer read-only functionality. You have also decided that only a selected group of users will gain access to the new API. Such verification will be based on the value of cookies in the header of HTTP requests incoming to the old API. For a given value of cookies, a REDIRECT containing URL to the new API has to be made.

As for the module for data presentation and visualization, here you also plan some modifications, but not so extensive. You just want to add a few minor features to the current release. To test the stability of this latest version of the module, you want to use an approach based on the so-called canary deployment strategy, where both versions of the module will run in parallel, but the latest one is allowed to handle only a small percentage (e.g. 5%) of all requests coming from users.

If you start analyzing this scenario from the technical point of view, you will find out that it requires the following mechanisms to be executed:

path-based HTTP matching (as you still want to use URL prefixes to distinguish traffic destined to data collector API module and data visualization module).
header-based HTTP matching (since you want to use cookies values for matching).
method-based HTTP matching (since you plan to allow only the HTTP GET method to be executed on the old data collector API module).
request REDIRECT (your scenario requires a REDIRECT to be executed).
weighted load balancing (since you need non-symmetrical traffic split for data visualization module versions).

Now ask yourself: which of these functions does Ingress cover? The correct answer is: only the first from the above list. Therefore, Ingress will not be suitable for your complex scenario. Unless you apply some custom Ingress controller implementation that offers advanced functionalities. Remember, however, that then you will not really be using the native version of Ingress, but the so-called CRDs and annotations instead, which are actually something extra.

You can find a detailed description of these functions in this video:

Now it's time to ask the same question again, but now in the context of Gateway API: which of the functions required in your scenario does Gateway API cover? The correct answer is: all of them. Here's what it might look like.

Fig.3: Kubernetes Gateway API-based configuration scheme - advanced scenario

Here you can use exactly the same Gateway instance as for the basic scenario but now the previous HTTPRoute instance is replaced by two other routes attached to that Gateway. The first HTTPRoute instance will handle the requests to “my.analytics.example.com”. It has three rules configured:

The first matches requests containing the "/data" prefix and the Cookie header value equal to "version=fullfeatured". Such requests will not go to any backend. Instead, a REDIRECT will be executed to a new location equal to "fullfeatured.my.analytics.example.com/data". Requests to the "fullfeatured.my.analytics.example.com" host will, in turn, be handled by a separate HTTPRoute instance. Routing to the backends corresponding to the individual sub-modules of the new data collector API is handled here based on the URL path prefix:
- requests with “/data” prefix will be forwarded to the “pilot” web-page containing the URLs to the new APIs
- requests with “/metrics” prefix will be forwarded to the metrics API
- requests with “/traces” prefix will be forwarded to the traces API
- requests with “/logs” prefix will be forwarded to the logs API
The second rule matches only HTTP GET requests containing the "/data" prefix and the Cookie header value equal to "version=standard". Such traffic will be forwarded to the simplified version of the old data collector API.
The third rule matches requests containing the "/visualize" prefix. They will be forwarded both to the current version and the test (updated) version of the data visualization and presentation module. The traffic distribution will be based on the applied canary rollout strategy (95% to 5% respectively).

The entire configuration is presented below.

---
kind: GatewayClass
apiVersion: gateway.networking.k8s.io/v1beta1
metadata:
 name: my-controller
spec:
 controllerName: my.company.example.io/my-gateway-controller
 
 
---
kind: Gateway
apiVersion: gateway.networking.k8s.io/v1beta1
metadata:
 name: web-gateway
spec:
 gatewayClassName: my-controller
 listeners:
   - name: http
     protocol: HTTP
     port: 80
     allowedRoutes:
       namespaces:
         from: All
 
---
apiVersion: gateway.networking.k8s.io/v1alpha2
kind: HTTPRoute
metadata:
 name: httproute-advanced
 labels:
   route: advanced
spec:
 parentRefs:
 - group: gateway.networking.k8s.io
   kind: Gateway
   name: web-gateway
 hostnames:
 - my.analytics.example.com
 rules:
 - matches:
   - path:
       type: PathPrefix
       value: /data
     headers:
     - type: Exact
       name: Cookie
       value: "version=fullfeatured"
   filters:
   - type: RequestRedirect  
     requestRedirect:
       hostname: fullfeatured.my.analytics.example.com
       statusCode: 302
 - matches:
   - path:
       type: PathPrefix
       value: /data
     headers:
     - type: Exact
       name: Cookie
       value: "version=standard"
     method: GET 
   backendRefs:
   - name: data
     port: 8000
 - matches:
   - path:
       type: PathPrefix
       value: /visualize
   backendRefs:
   - name: visualize
     port: 9000
     weight: 95
   - name: visualize2
     port: 9000
     weight: 5
 
 
---
apiVersion: gateway.networking.k8s.io/v1alpha2
kind: HTTPRoute
metadata:
 name: httproute-advanced-data
 labels:
   route: advanced-data
spec:
 parentRefs:
 - group: gateway.networking.k8s.io
   kind: Gateway
   name: web-gateway
 hostnames:
 - fullfeatured.my.analytics.example.com
 rules:
 - matches:
   - path:
       type: PathPrefix
       value: /data
   backendRefs:
   - name: data-desc
     port: 80
 - matches:
   - path:
       type: PathPrefix
       value: /metrics
   backendRefs:
   - name: metrics-ff
     port: 8001
 - matches:
   - path:
       type: PathPrefix
       value: /traces
   backendRefs:
   - name: traces-ff
     port: 8002
 - matches:
   - path:
       type: PathPrefix
       value: /logs
   backendRefs:
   - name: logs-ff
     port: 8003

Summary

The Gateway API is an interesting initiative. It allows you to express in a standard way a wide set of advanced functionalities that were not covered by the native Ingress. At the same time, it still gives flexibility and allows a certain level of customization for implementers. It can therefore be expected that the Gateway API will become more and more popular over time.

DESIGN

SOFTWARE ENGINEERING

NETWORK & CLOUD ENGINEERING

DATA

RESEARCH AND DEVELOPMENT

FOR STARTUPS

EXPERTISE

PUBLICATIONS

NEWS