How to provide Host to Kubernetes service discovery provider?

[chrisbecke]

The Problem

While navigating the docs I saw this rather concerning note:

Note 1: Host, Port and Token are no longer in use.

As my situation is I am routing traffic to pods on a EKS cluster from an Ocelot based gateway running on docker on an EC2 node external to the cluster, this capability was important to me.

Expectation

How to usePodServiceAccount to configure and use an external Kubernetes cluster should be documented.

Reality

I needed the following code in my Startup.cs to connect to a cluster the way the documentation implies should be possible

var cfg = Configuration.GetSection("GlobalConfiguration:ServiceDiscoveryProvider");
  var Host = cfg.GetValue<string>("Host");
  var Port = cfg.GetValue<int>("Port", 443);
  var Token = cfg.GetValue<string>("Token");
  services.Configure<KubeClientOptions>(options => {
    options.ApiEndPoint = new UriBuilder("https",Host,Port).Uri;
    options.AuthStrategy = KubeAuthStrategy.BearerToken;
    options.AccessToken = Token;
    options.AllowInsecure = true;
  });
services.AddOcelot(Configuration).AddKubernetes(false);

[raman-m]

Hello, Chris!

Our NuGet Ocelot.Provider.Kubernetes extension package is based on the KubeClient package. For a comprehensive understanding, it is essential refer to the KubeClient documentation.

The Problem

While navigating the docs I saw this rather concerning note:

Note 1: Host, Port and Token are no longer in use.

As my situation is I am routing traffic to pods on a EKS cluster from an Ocelot based gateway running on docker on an EC2 node external to the cluster, this capability was important to me.

So, the subject of discussion is this line in docs:

Ocelot/docs/features/kubernetes.rst

Line 83 in ab119f8

**Note 1**: ``Host``, ``Port`` and ``Token`` are no longer in use.

This phrase was added in the following 👇

• Commit d43383b: see R61-R62
• PR update k8s doc & samples #885

By default, usePodServiceAccount = true, as noted in Install section.
It appears you do not use this mode, and in your scenario, useServiceAccount = false.

The sentence in the documentation can confuse developers. Would you like us to rewrite the documentation and provide more explanation?

[raman-m]

Expectation

How to usePodServiceAccount to configure and use an external Kubernetes cluster should be documented.

Sorry?

useServiceAccount = true

It is already documented in the Install section.

useServiceAccount = false

You are correct; the documentation for this mode and its configuration can be improved. A code sample is necessary.
Currently this mode is tested in the KubernetesServiceDiscoveryTests class.
Here is configuration:

Ocelot/test/Ocelot.AcceptanceTests/ServiceDiscovery/KubernetesServiceDiscoveryTests.cs

Lines 31 to 37 in ab119f8

	_kubeClientOptionsConfigure = opts =>
	{
	opts.ApiEndPoint = new Uri(_kubernetesUrl);
	opts.AccessToken = "txpc696iUhbVoudg164r93CxDTrKRVWG";
	opts.AuthStrategy = KubeAuthStrategy.BearerToken;
	opts.AllowInsecure = true;
	};

Here is adding services:

Ocelot/test/Ocelot.AcceptanceTests/ServiceDiscovery/KubernetesServiceDiscoveryTests.cs

Lines 314 to 317 in ab119f8

	private IOcelotBuilder AddKubernetes(IServiceCollection services) => services
	.Configure(_kubeClientOptionsConfigure)
	.Replace(GetValidateScopesDescriptor())
	.AddOcelot().AddKubernetes(false);

So, useServiceAccount is false.

[chrisbecke]

Now I am embarrassed. I looked everywhere except the tests.

[raman-m]

Anything else, Chris?

[chrisbecke]

satisfied now thanks. especially as I have it working in our environment.

[raman-m]

My final answer is here

[raman-m]

Reality

I needed the following code in my Startup.cs to connect to a cluster the way the documentation implies should be possible

var cfg = Configuration.GetSection("GlobalConfiguration:ServiceDiscoveryProvider");
  var Host = cfg.GetValue<string>("Host");
  var Port = cfg.GetValue<int>("Port", 443);
  var Token = cfg.GetValue<string>("Token");
  services.Configure<KubeClientOptions>(options => {
    options.ApiEndPoint = new UriBuilder("https",Host,Port).Uri;
    options.AuthStrategy = KubeAuthStrategy.BearerToken;
    options.AccessToken = Token;
    options.AllowInsecure = true;
  });
services.AddOcelot(Configuration).AddKubernetes(false);

It should work, but you forgot to replace the configuration service in DI for the IOptions<KubeClientOptions> interface.
When you add services using this code: services.AddOcelot(Configuration).AddKubernetes(false); you are utilizing our default AddKubernetes(bool) method:

Ocelot/src/Ocelot.Provider.Kubernetes/OcelotBuilderExtensions.cs

Lines 31 to 34 in ab119f8

	KubeClientOptions options = sp.GetRequiredService<IOptions<KubeClientOptions>>().Value;
	options.LoggerFactory ??= sp.GetService<ILoggerFactory>();
	return KubeApiClient.Create(options);

If you bind the configuration section to the IOptions<KubeClientOptions> interface, then your solution will be correct.

[raman-m]

I meant you can remove the IKubeApiClient interface from the DI container 👇

builder.Services
    .AddOcelot(Configuration).AddKubernetes(false)
    .RemoveAll<IKubeApiClient>()
    .AddSingleton<IKubeApiClient, KubeApiClient>(MyApiClientFactory);

KubeApiClient MyApiClientFactory(IServiceProvider sp)
{
    if (usePodServiceAccount)
    {
        return KubeApiClient.CreateFromPodServiceAccount(sp.GetService<ILoggerFactory>());
    }

    KubeClientOptions options = // create your options here!
    options.LoggerFactory ??= sp.GetService<ILoggerFactory>();
    return KubeApiClient.Create(options);
}

Is it clear to you? I hope this will help!

[chrisbecke]

That is also an approach. given the default AddKubernetes method was retrieving a IOptions from DI it just seemed easier and safer to just make sure there was actually one present.

[raman-m]

Indeed, utilizing the default implementation based on the IOptions<KubeClientOptions> is preferable as it aligns with best practices in ASP.NET development. Therefore, avoid replacing the services!

I have come to the conclusion that your solution, which is based on IOptions<KubeClientOptions>, is indeed entirely correct

  var cfg = builder.Configuration.GetSection("GlobalConfiguration:ServiceDiscoveryProvider");
  var Host = cfg.GetValue<string>("Host");
  var Port = cfg.GetValue<int>("Port", 443);
  var Token = cfg.GetValue<string>("Token");
  services.Configure<KubeClientOptions>(options => {
    options.ApiEndPoint = new UriBuilder("https",Host,Port).Uri;
    options.AuthStrategy = KubeAuthStrategy.BearerToken;
    options.AccessToken = Token;
    options.AllowInsecure = true;
  });
builder.Services.AddOcelot(builder.Configuration).AddKubernetes(false);

Let us add a new extension method.
Would you be able to contribute, Chris, by opening a pull request?

[raman-m]

@chrisbecke Will the following extension be useful?

    public static IOcelotBuilder AddKubernetes(this IOcelotBuilder builder, Action<KubeClientOptions> configureOptions = null,
        string defaultScheme = null, int defaultPort = 443)
    {
        configureOptions ??= Configure;
        builder.Services.Configure(configureOptions);
        return builder.AddKubernetes(false);

        void Configure(KubeClientOptions options)
        {
            var section = builder.Configuration.GetSection("GlobalConfiguration:ServiceDiscoveryProvider");
            var scheme = section.GetValue<string>("Scheme", defaultScheme ?? Uri.UriSchemeHttps);
            var host = section.GetValue<string>("Host");
            var port = section.GetValue<int>("Port", defaultPort);
            var token = section.GetValue<string>("Token");
            options.ApiEndPoint = new UriBuilder(scheme, host, port).Uri;
            options.AuthStrategy = KubeAuthStrategy.BearerToken;
            options.AccessToken = token;
            options.AllowInsecure = true;
        }
    }

I was inspired by your solution 😄

[raman-m]

• #2255 #2256 Overload AddKubernetes method with KubeClientOptions param for the discovery provider #2257

Therefore, it will be released in the .NET 9 version