This post was inspired by listening to the February 19, 2019, Kubernetes Podcast, “Ingress, with Tim Hockin.” The Kubernetes Podcast is turning out to be a very well done podcast overall, and well worth the listen. In the Ingress episode, the podcasters interview Tim Hockin who’s one of the original Kubernetes co-founders, a team lead on the Kubernetes predecessor Borg/Omega, and is still very active within the Kubernetes community such as chairing the Kubernetes Network Special Interest Group that currently own the Ingress resource specification. Tim talks in the podcast about the history of the Kubernetes Ingress, current developments around Ingress, and proposed futures. This talk inspired me to reflect on both Ingress Controllers (realizes the implementation of Ingress manifest) and Ingress the concept (allow client outside the Kubernetes cluster to access services running in the Kubernetes cluster).
So what’s a Kubernetes Ingress?
To paraphrase from the Kubernetes Ingress documentation, Ingress is an L7 network service that exposes HTTP(S) routes from outside to inside a Kubernetes cluster. A Kubernetes cluster may have one or more Ingress Controllers running, and each controller manages service reachability, load balancing, TLS/SSL termination, and other services for that controller’s associated routes.
Each Ingress manifest includes an annotation that indicates which Ingress controller should manage that Ingress resource.
For example, to have Solo.io Gloo manage a specific Ingress resource, you
would specify the following. Note the included annotation
apiVersion: extensions/v1beta1 kind: Ingress metadata: annotations: kubernetes.io/ingress.class: gloo labels: chart: jsonplaceholder-v0.1.0 name: jsonplaceholder-jsonplaceholder namespace: default spec: rules: - host: gloo.example.com http: paths: - path: /.* backend: serviceName: jsonplaceholder-jsonplaceholder servicePort: 8080
Ingress has existed as a beta extension since Kubernetes 1.1, and it’s proven to be a lowest common denominator API. For
example, the NGNIX community Ingress Controller is used by many in production, but that NGNIX Ingress controller
requires the use of many NGNIX specific Ingress Annotations
for all but the simplest use cases. The current Kubernetes Ingress resource specification has many limitations like that all
referenced services and secrets MUST be in the same namespace as the Ingress, i.e., no cross namespace referencing.
And there have been long debates about how exactly to interpret the
path attribute; is it a regular expression like
the documentation implies OR is it a path prefix like some controllers like NGNIX implement. These challenges have made
it, in practice, difficult to have an Ingress manifest that is portable across implementations. The current Ingress
manifest has also proven difficult to round trip sync with Custom Resources (CRD)
which is unfortunate as CRDs are proving to be a beneficial way to extend Kubernetes.
What’s Next for Ingress?
In the podcast, Tim Hockin says given how many are using the current beta Ingress spec in production, there is a push
to move the existing Ingress spec to GA status, and then start work on a next-generation specification, either an Ingress v2 or
breaking up Ingress across multiple CRDs. Tim mentions how the Kubernetes community is looking at several
Envoy based Ingress implementations for inspiration for the next generation of Ingress. For
example, Heptio Contour has created a very interesting, and implementation
neutral CRD called Ingress Route. An Ingress Route
looks to address the governance challenges with Ingress, for example, if a company wants to expose a
/eng route path there are many challenges with the current Ingress model as you can have conflicting Ingress
manifests for the route
/eng. Ingress Route provides a way to create governance and delegation such as cluster admins
can define a virtual host
/eng and delegate implementation explicitly to the
eng namespace, and this prevents others
from overriding that route path.
The Istio community, also based on Envoy like Heptio Contour, are also defining Ingress CRDs.
It will be fascinating to see how Ingress evolves in the not too distant future.
Related reading: API Gateways are going through an identity crisis.
I find it helpful to see working code to help make concepts more real, so let’s run through a few examples of Ingress and beyond.
For this example, I’m going to use a Kubernetes service created from https://jsonplaceholder.typicode.com/, which
provides a quick set of REST APIs that provide different JSON output that can be helpful for testing. It’s based on
a Node.js json-server - it’s very cool and worth looking at independently. I
forked the original GitHub jsonplaceholder repository, ran
on the project, and made a couple of tweaks to the generated Helm chart. Draft
is a super fast and easy way to bootstrap existing code into Kubernetes. I’m running all of this example locally using
The jsonplaceholder service comes with six common resources each of which returns several JSON objects. For this
example, we’ll be getting the first user resource at
Following is a script to try this example yourself, and there’s also an asciinema playback so you can see what it looks like running on my machine. We’ll unpack what’s happening following the playback.
# Install tooling brew update brew cask install minikube brew install kubernetes-cli \ kubernetes-helm \ azure/draft/draft \ glooctl # Create and set up local Kubernetes Cluster minikube start helm init draft init glooctl install ingress # Draft runs better locally if you configure # against minikube docker daemon eval $(minikube docker-env) # Get and run the example git clone https://github.com/scranton/jsonplaceholder.git cd jsonplaceholder draft up # Validate all is running kubectl get all --namespace default kubectl get all --namespace gloo-system kubectl get ingress --namespace default curl --header "Host: gloo.example.com" \ $(glooctl proxy url --name ingress-proxy)/users/1
We installed local tooling (you can check respective websites for full install details)
We then started up a local Kubernetes cluster (minikube) and initialized Helm and Draft. We also installed Gloo ingress into our local cluster.
git clone our example and used
draft up to build and deploy it to our cluster. Let’s spend a minute on
what happened in this step. I originally forked the
jsonplaceholder GitHub repository and ran
draft create against
its code. Draft autodetects the source code language, in this case, Node.js, and creates both a
Dockerfile that builds
our example application into an image container and creates a default Helm chart. I then made a few minor tweaks to the
Helm chart to enable its Ingress. Let’s look at that Ingress manifest. The main changes are the addition of the
ingress.class: gloo annotation to mark this Ingress for Gloo’s Ingress Controller. And the host is set to
gloo.example.com, which is why our curl statement set
curl --header "Host: gloo.example.com".
For more examples of using Gloo as an basic Ingress controller you can check out Kubernetes Ingress Control using Gloo.
You may have also noticed the call to
$(glooctl proxy url --name ingress-proxy) in the curl command. This is needed
when you’re running in a local environment like minikube and you need to get the host IP and port of the Gloo proxy
server. When Gloo is deployed to a Cloud Provider like Google or AWS, then those environments would associate a static IP and
allow port 80 (or port 443 for HTTPS) to be used, and that static IP would be registered with a DNS server, i.e.,
when Gloo is deployed to a cloud-managed Kubernetes you could do
Ingress Example Challenges
Let’s say we wanted to remap the exiting
/people/1 as users are people too. That becomes tricky with
Ingress manifests as we can set up a second rule for
/people, but we need to rewrite that path to
sending to our service as it doesn’t know how to handle requests for
/people. If you were using the NGNIX ingress, you
could add another annotation
nginx.ingress.kubernetes.io/rewrite-target: /, but now we’re adding implementation
specific annotations, that is, the nginx annotation won’t be recognized by other Ingress Controllers. And annotations
are a flat name space so adding lots of annotations can get quite messy, which is part of why
Custom Resources (CRD) was
created. Let’s see what the original route, and our path re-writing route, would look like in a CRD based Ingress: Gloo.
Gloo Virtual Services
Gloo uses a concept called Virtual Service that is derived from similar ideas in Istio and Envoy and is conceptually equivalent to an Ingress resource. Easiest to show you the equivalent of the example Ingress we’ve created so far in a Gloo Virtual Service.
apiVersion: gateway.solo.io/v1 kind: VirtualService metadata: name: default namespace: gloo-system spec: virtualHost: domains: - gloo.example.com routes: - matcher: prefix: / routeAction: single: upstream: name: default-jsonplaceholder-jsonplaceholder-8080 namespace: gloo-system
You’ll notice that it looks very similar to the Ingress we had previously created with a few subtle changes. The path
prefix: / which is generally what people intend, i.e., if the beginning of the request message path
matches the route path specifier than apply the route actions. If we wanted to exactly match the previous Ingress, we could use
regex: /.* instead. Virtual Services allow you to specify paths by: prefix, exact, and regular expression. You can
also see that instead of
servicePort, a Virtual Service has a
delegates to a
upstream. Gloo upstreams are auto-discovered and can refer to Kubernetes Services AND
REST/gRPC function, cloud functions like AWS Lambda and Google Functions, and other external to Kubernetes services.
More details on Gloo at:
Let’s go back to our example, and update our Virtual Service to do the path rewrite we wanted, i.e.,
apiVersion: gateway.solo.io/v1 kind: VirtualService metadata: name: default namespace: gloo-system spec: virtualHost: domains: - gloo.example.com routes: - matcher: prefix: /people routeAction: single: upstream: name: default-jsonplaceholder-jsonplaceholder-8080 namespace: gloo-system routePlugins: prefixRewrite: prefixRewrite: /users - matcher: prefix: / routeAction: single: upstream: name: default-jsonplaceholder-jsonplaceholder-8080 namespace: gloo-system
We’ve added a second route matcher, just like adding a second route path in an Ingress, and specified
This will match all requests that start with
/people, and all other calls to the
gloo.example.com domain will be
handled by the other route matcher. We also added a
routePlugins section that will rewrite the request path to
that our service will now correctly handle our request. Route Plugins
allow you to perform many operations on both the request to the upstream service and the response back from the upstream
service. Best shown with an example, so for our new
/people route let’s also transform the response to both add
a new header
x-test-phone with a value from the response body, and let’s transform the response body to return a
couple of fields: name, and the address/street and address/city.
Let’s see what that looks like. My example GitHub repository already included
the full Gloo Virtual Service we just examined. We need to configure Gloo for
gateway which means adding another proxy to handle Virtual Services in
addition to Ingress resources. We’ll use
draft up to ensure our example is
fully deployed including the full Virtual Service, and then we’ll call both
/people/1 to see the differences.
# Install Gloo and update example glooctl install gateway draft up # Call service curl --verbose --header "Host: gloo.example.com" \ $(glooctl proxy url --name gateway-proxy)/users/1 curl --verbose --header "Host: gloo.example.com" \ $(glooctl proxy url --name gateway-proxy)/people/1
Ok, well not that mind-blowing if you’ve used other L7 networking products or done other integration work, but still pretty cool relative to standard Ingress objects. Gloo is using Inja Templates to process the JSON response body. More details in the Gloo documentation.
In this article, we touched on some of the history and difficulties with the existing Kubernetes Ingress resources. Ingress resources continue to play a role within Kubernetes deployments despite the many challenges that annotation-based extensions have. Kubernetes Custom Resources (CRDs) was created to address some of those extension challenges and can provide a cleaner way to extend Kubernetes as you saw in the Gloo Ingress and Gateway examples. I’m a big believer in the potential of Envoy based solutions as are others in the Istio and Contour communities, and it will be exciting to see how the Kubernetes community decides to evolve Ingress after they finally move the existing resource spec to GA status.