Machine Learning
Kubeflow: Authentication with Istio + Dex

by | July 2019

5 min read

Compatible with Static Users, LDAP/AD and External IdPs Multi-tenancy is a central feature of Kubeflow 0.6 and providing a flexible authentication scheme is important for enterprise use-cases. In this article, we will explore how we leveraged the power of Istio and open-source components to create a flexible, robust and clean authentication solution. This solution is […]

Compatible with Static Users, LDAP/AD and External IdPs

Multi-tenancy is a central feature of Kubeflow 0.6 and providing a flexible authentication scheme is important for enterprise use-cases.

In this article, we will explore how we leveraged the power of Istio and open-source components to create a flexible, robust and clean authentication solution.

This solution is especially powerful because:

  • It leverages Istio features and doesn’t add extra components.
  • It can work on any Kubernetes cluster and doesn’t rely on having access to the API-Server.
  • It is a general solution and can be applied to other projects besides Kubeflow.

With that said, let’s dive into the solution.

The end-goal

We will focus on using Istio to perform end-user authentication, meaning that our apps won’t contain any authentication logic!
Authentication, for user access to an application, will be done at the Istio Gateway: the one point where all traffic enters the cluster.

Authentication is a major area that developers may choose to leave up to Istio. The idea is simple:

  • Incoming traffic includes a JSON Web Token (JWT) for authentication.
  • The JWT is verified by the Istio Gateway.
  • Apps inside the cluster trust the JWT because it has been verified by the Gateway.

Istio Gateway enforces Auth for the Kubeflow apps

This way, our apps contain no authentication logic at all!
Unfortunately, it’s not that simple.

Authentication in Istio

At first glance, Istio seems to support end-user authentication.
However, notice how Istio can only perform the last part, token verification (i.e. verify the JWT and allow the request).
The question is: how are we going to get that token in the first place?

Enter OpenID Connect (OIDC): a way to authenticate a user using a standardized OAuth2 flow.

The OIDC Flow

Because a picture is worth a thousand words, let’s take a look at what an OIDC flow looks like.

The OIDC Flow — Istio Gateway only supports JWT verification

Notice how Istio can only perform the last part, token verification.
Because of this, we need a new entity that will act as the OIDC client and execute the flow.

Envoy — The power behind Istio

As many of you will already know, Istio is mainly in the control path.
Under the hood, the data is handled by Envoy, a very efficient and versatile proxy.

Why is this relevant?
Istio lets you configure its underlying Envoy Proxies using an EnvoyFilter object.
The EnvoyFilter object enables us to insert Envoy Filters in the data path of certain proxies.

In our case, we want to configure the Envoy Proxy of the Istio Gateway.
But what filter should we use?

The ext_authz HTTP Filter

The ext_authz HTTP filter is an extremely powerful and versatile capability of Envoy.

Its function is very simple:
Every request is forwarded to an Authorization Service (AuthService). Depending on the answer, that response can be:

  • Accepted
  • Rejected
  • Redirected

The OIDC AuthService

We just need to implement the component that will handle those actions.
Luckily, there is an open-source tool that we can reuse for this exact purpose, the ambassador-auth-oidc project.

Putting it all Together

Our Istio Gateway can now act as an OIDC client and execute the whole flow to authenticate a user.
Let’s test it out using Dex, a popular OIDC provider.
Dex supports many authentication backends, including static users, LDAP and external Identity Providers, so you can have the power of choice.

The final architecture is the following:

Demo

All of the aforementioned functionality is available for Kubeflow v0.6.
At this time, the way to install this reference architecture is to use the standard kfctl tool, and define platform “existing_arrikto”, to install on an existing Kubernetes Cluster.

https://www.kubeflow.org/docs/started/getting-started-k8s/

Simply follow the instructions from the Kubeflow docs and you should have a Kubernetes Cluster with Kubeflow, Istio and Dex for Authentication.

Image for post
You should be greeted by the Dex login screen

Final Thoughts

In this article, we unlocked the powerful feature of the Envoy Proxy and used Istio along with Dex and the OIDC AuthService to form a complete Authentication architecture. This enables applications to offload all authentication logic to Istio and focus on the business logic, which works great for Kubeflow’s microservice-oriented architecture.

The ext_authz feature of Envoy is not very well-known within the Istio userbase, however it is an incredibly powerful feature that enables us to integrate with virtually any authentication scheme.

Yannis Zarkadas

Yannis is a software engineer at Arrikto, working with Kubeflow. He loves contributing to open source projects and has authored the Cassandra Operator in Rook and the official Scylla Operator.

More Like This

Avoiding an ML Migraine!

Here are some simple observations: setting up an ML stack/pipeline is incredibly hard, setting up a production ML stack/pipeline is even harder, and setting up an ML stack/pipeline that works across multi-cloud environments is a full-on headache. That’s not just the...

The double whammy when it comes to ML in an enterprise

Observations on the drivers of success and failure Imagine you’re about to have heart surgery. Your surgeon drives to your house, bathes you, drives you to the hospital, preps you for surgery, administers anesthesia, performs the surgery, moves you to the...

You May Also Like

k3d + GitHub Actions: Kubernetes E2E Testing Made Easy

Combining k3d and GitHub Actions for easy, quick and cheap E2E testing The Old Way During my first year of Software Engineering, I understood the importance of two things: E2E tests running with CI. They save a ton of manual effort testing the software, increase...

Kubeflow 1.1 Community Release Update

September, 2020 @ Kubeflow Community Update Call Josh Bottum, Kubeflow Community Product Manager, leads the community call giving us all an update on the great things that came in Kubeflow 1.1 This includes updates from; Josh Bottum, Arrikto, Kubeflow Product...

Kubeflow 1.0 Update By A Kubeflow Community Product Manager

August 20, 2020 @ KubeCon + CloudNativeCon Europe Amsterdam 2020 This session will provide a Kubeflow 1.0 Update by a Kubeflow Community Product Manager. The presentation will include a review of the Kubeflow Community and feature development process, the Kubeflow...