How to work with multiple configuration files per microservice?

[sachinvaid1604]

          > Ok guys, so I have found out what was causing the strange behaviour.

It was the folder structure that we are using in Kubernetes for our configuration files.

Generally placing configuration files in the folder structure described below will cause described issue.

This could be written in the documentation as a note.

Folder structure: etc/config/

Hi, Can you please provide a working code that can read multiple route configurations for each microservice and can run both. In my case, I can only access the last added route configuration.

Here is my program.cs file

using Microsoft.AspNetCore;
using Microsoft.AspNetCore.Hosting;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;
using Ocelot.DependencyInjection;
using Ocelot.Middleware;
using System.IO;

public class Program
{
    public static void Main(string[] args)
    {
        CreateWebHostBuilder(args)
            .Build()
            .Run();
    }

    static IWebHostEnvironment _hostingContextEnvironment = null;

    public static IWebHostBuilder CreateWebHostBuilder(string[] args) =>
        WebHost.CreateDefaultBuilder(args)
            .UseContentRoot(Directory.GetCurrentDirectory())
            .ConfigureAppConfiguration((hostingContext, config) =>
            {
                _hostingContextEnvironment = hostingContext.HostingEnvironment;

                config
                    .SetBasePath(hostingContext.HostingEnvironment.ContentRootPath)
                    .AddJsonFile("appsettings.json", true, true)
                    .AddJsonFile("ocelot.json", false, true)
                    .AddJsonFile("student.ocelot.Development.json", false, true)
                    .AddJsonFile("teacher.ocelot.Development.json", false, true)
                    .AddEnvironmentVariables();
            })
            .ConfigureServices(s =>
            {
                s.AddOcelot();
            })
            .Configure(app =>
            {
                app.UseOcelot().Wait();
            });
}

Originally posted by @sachinvaid1604 in #1186 (comment)

[raman-m]

Sachin, fork Ocelot repo first to look into the Ocelot code!

After forking Ocelot you will figure out that AddJsonFile method is a part of JsonConfigurationExtensions class from Microsoft.Extensions.Configuration namespace!
If you press F12 key multiple times in your Visual Studio, you will get to this Microsoft ASP.NET implementation:

        /// <summary>
        /// Adds a JSON configuration source to <paramref name="builder"/>.
        /// </summary>
        /// <param name="builder">The <see cref="IConfigurationBuilder"/> to add to.</param>
        /// <param name="provider">The <see cref="IFileProvider"/> to use to access the file.</param>
        /// <param name="path">Path relative to the base path stored in
        /// <see cref="IConfigurationBuilder.Properties"/> of <paramref name="builder"/>.</param>
        /// <param name="optional">Whether the file is optional.</param>
        /// <param name="reloadOnChange">Whether the configuration should be reloaded if the file changes.</param>
        /// <returns>The <see cref="IConfigurationBuilder"/>.</returns>
        public static IConfigurationBuilder AddJsonFile(this IConfigurationBuilder builder, IFileProvider? provider, string path, bool optional, bool reloadOnChange)
        {
            ThrowHelper.ThrowIfNull(builder);

            if (string.IsNullOrEmpty(path))
            {
                throw new ArgumentException(SR.Error_InvalidFilePath, nameof(path));
            }

            return builder.AddJsonFile(s =>
            {
                s.FileProvider = provider;
                s.Path = path;
                s.Optional = optional;
                s.ReloadOnChange = reloadOnChange;
                s.ResolveFileProvider();
            });
        }

I have no idea how inner MS implementation merges config JSON.

If you call

                    .AddJsonFile("student.ocelot.Development.json", false, true)
                    .AddJsonFile("teacher.ocelot.Development.json", false, true)

and they describe the same routes, seems the second file will overwrite the routes from the first one.
Microsoft classes don't merge!

[sachinvaid1604]

The documentation says there is a merging facility available, but I used it the way they suggested, but the behaviour is totally different.

Merging Configuration Files
This feature was requested in issue 296 and allows users to have multiple configuration files to make managing large configurations easier.

Instead of adding the configuration directly e.g. AddJsonFile("ocelot.json") you can call AddOcelot() like below:

ConfigureAppConfiguration((hostingContext, config) =>
{
config
.SetBasePath(hostingContext.HostingEnvironment.ContentRootPath)
.AddJsonFile("appsettings.json", true, true)
.AddJsonFile($"appsettings.{hostingContext.HostingEnvironment.EnvironmentName}.json", true, true)
.AddOcelot(hostingContext.HostingEnvironment)
.AddEnvironmentVariables();
})
In this scenario Ocelot will look for any files that match the pattern (?i)ocelot.([a-zA-Z0-9]*).json and then merge these together. If you want to set the GlobalConfiguration property, you must have a file called ocelot.global.json.

[raman-m]

Exactly!
If you know "how", why are you asking? 😄

[raman-m]

You described exactly the same user scenario of #296 for merging multiple files of the same environment.

You have to read this Ocelot doc: Merging Configuration Files

So, you need this setup:

            .ConfigureAppConfiguration((hostingContext, config) =>
            {
                config
                    // ...
                    .AddOcelot(hostingContext.HostingEnvironment)
                    .AddEnvironmentVariables();
            })

[raman-m]

If you need to load multi-files config from other folder:

.AddOcelot("/foo/bar", hostingContext.HostingEnvironment)

Clear?

[sachinvaid1604]

The reason I asked this is based on my finding of the actual behaviour I am getting from muktiple configurations. it is only picking up last configuration file always, like in my case student and teacher microservices are added but it will only pick the last on added configuration file and ignoring other routes.

[sachinvaid1604]

As of now there is no overloaded method that takes folder path, I am using .net 6

[sachinvaid1604]

@raman-m Thanks a lot for guiding me in the right direction. I was using the wrong convention of the file names as it never matches with what Ocelot looks for. The configuration files should always start with ocelot along with the environment name. Thank you for showing me the path to internally go into the working of Ocelot's AddOcelot method. Its working perfectly fine now.

[sachinvaid1604]

There is a flaw in the Ocelot's AddOcelot method, it only traverse the current directory, rather than subdirectories, so It's better to write your own extension method that can traverse subdirectories so as to keep the files segregated in their respective folders and provide better management. Also, in my case I renamed the configuration files as follow.

[sachinvaid1604]

There is a flaw in the Ocelot's AddOcelot method, it only traverse the current directory, rather than subdirectories, so It's better to write your own extension method that can traverse subdirectories so as to keep the files segregated in their respective folders and provide better management. Also, in my case I renamed the configuration files as follow.

[sachinvaid1604]

 private static FileConfiguration GetOcelotFiles(string folder, IWebHostEnvironment env)
 {
     string excludeConfigName = ((env != null && env.EnvironmentName != null) ? ("ocelot." + env.EnvironmentName + ".json") : string.Empty);
     Regex reg = SubConfigRegex();
     FileInfo[] array = (from fi in new DirectoryInfo(folder).EnumerateFiles("*", SearchOption.AllDirectories)
                         where reg.IsMatch(fi.Name) && fi.Name != excludeConfigName
                         select fi).ToArray();
     FileConfiguration fileConfiguration = new FileConfiguration();
     FileInfo[] array2 = array;
     foreach (FileInfo fileInfo in array2)
     {
         if (array.Length <= 1 || !fileInfo.Name.Equals("ocelot.json", StringComparison.OrdinalIgnoreCase))
         {
             FileConfiguration fileConfiguration2 = JsonConvert.DeserializeObject<FileConfiguration>(File.ReadAllText(fileInfo.FullName));
             if (fileInfo.Name.Equals("ocelot.global.json", StringComparison.OrdinalIgnoreCase))
             {
                 fileConfiguration.GlobalConfiguration = fileConfiguration2.GlobalConfiguration;
             }

             fileConfiguration.Aggregates.AddRange(fileConfiguration2.Aggregates);
             fileConfiguration.Routes.AddRange(fileConfiguration2.Routes);
         }
     }

     return fileConfiguration;
 }

This is the utility method I created and just call the .AddOcelot(GetOcelotFiles(hostingContext.HostingEnvironment.ContentRootPath + "\\ocelotfiles", hostingContext.HostingEnvironment))
It will get all the files from FileConfiguration and will write it to ocelot.json file, which is the ultimate truth for Ocelot. I wish, Ocelot should have considered to iterate files recursively, but no worries, at the end, I got what I wanted. Thanks to @raman-m

[raman-m]

You are welcome!
Hope for your future contributions.