It seems that IEvaluator.Reset does not actually clear the referenced assemblies

[frank-thole]

This is the (very simplified) code I am using:

class ScriptContextManager
{
    private IEvaluator _evaluator = CSScript.Evaluator.With(eval => eval.IsAssemblyUnloadingEnabled = true);

    public void LoadAssembly(string assemblyPath)
    {
        var list1 = _evaluator.GetReferencedAssemblies();

        _evaluator.With(eval => eval.IsAssemblyUnloadingEnabled = true).ReferenceAssembly(assemblyPath);

        var list2 = _evaluator.GetReferencedAssemblies();
    }

    public void UnloadAndWait()
    {
        _evaluator = _evaluator.With(eval => eval.IsAssemblyUnloadingEnabled = true).Reset(false);

        var list3 = _evaluator.GetReferencedAssemblies();

        for (int i = 0; i < 1000; i++)
        {
            GC.Collect();
            GC.WaitForPendingFinalizers();
            Thread.Sleep(10);
        }

        var list4 = _evaluator.GetReferencedAssemblies();
    }
}

class Program
{
    static void Main(string[] args)
    {
        var manager = new ScriptContextManager();
        manager.LoadAssembly("C:\\Users\\frankt\\AppData\\Local\\Temp\\CSScriptTest\\2c808b71-b1c0-4c52-9ba1-2e07e971ef76\\MQTTnet.dll");
        manager.UnloadAndWait();
    }
}

list1 contains 23 assemblies.
list2 contains 24 assemblies, because MQTTnet.dll gets added.

list3 and list4 still contain 24 assemblies. In Visual Studio it can also be seen that MQTTnet.dll stays around in the Modules view.

Documentation says on Reset function: "Resetting means clearing all referenced assemblies, recreating evaluation infrastructure (e.g. compiler setting) and reconnection to or recreation of the underlying compiling services."

I am also unable to find in the cs-script repository where the "refAssemblies" list is cleared.

However, it seems this is causing the application to cling on to the MQTTnet.dll, and thus doesn't allow replacing the MQTTnet.dll for another version during the lifetime of the application.

I am not sure if the IsAssemblyUnloadingEnabled = true option is needed in this scenario. Without it there is no difference in behavior.

Am I doing something wrong, should I use some other code? Or is this an actual issue in the cs-script code?

[oleg-shilo]

Thank you. Will look at it

[oleg-shilo]

Indeed, it is a defect.
There is a shadow copy of the referenced assemblies that were not cleared on the Reset(). This is fixed now, and the fix will be available with the very next release. Until the fix is available, you can achieve the desired behavier by creating a fresh copy of the evaluator when you call Reset. This is teh workaround:

// instead of 
_evaluator.Reset(false);

// use this workaround
lock (CSScript.EvaluatorConfig)
{
    var oldValue = CSScript.EvaluatorConfig.ReferenceDomainAssemblies;

    CSScript.EvaluatorConfig.ReferenceDomainAssemblies = false;
    _evaluator = CSScript.Evaluator;

    CSScript.EvaluatorConfig.ReferenceDomainAssemblies = oldValue;
}
_evaluator.Reset(false);

Though I am not sure it will help you to achieve what you want.
As I understand, what you want is to allow loading a new version of the assembly that is already loaded in the current AppDomain.

.NET Core does not support such a scenario directly. Though you could do it in .NET Framework.

You can only achieve this by executing your script from the host assembly that is loaded in isolation.

• Load Isolation: You can create multiple AssemblyLoadContext instances. Each context can only load one version of an assembly, but different contexts can load different versions of the same assembly simultaneously.

But it means that your application will need to fork multiple AssemblyLoadContext instances with the routine that executes the script. And this is a serious architectural change.

I waould encourage you to consider if you really want to go this way as in >NET loading multile versions of a given assembly is rathe considered an anti-pattern.

Key Risks & LimitationsType

• Identity Issues: If you pass an object from one version of an assembly to a method expecting the same type from a different version, it will throw an InvalidCastException.Binding Redirects: Usually, developers avoid loading multiple versions by using
• binding redirects to force the application to use a single, typically newer, version.

[frank-thole]

What I want to achieve is the following:

1. Reference the assemblies needed for script compilation
2. Compile the script
3. Load the required assemblies into a (newly created) AssemblyLoadContext
4. Load the compiled assembly into the same AssemblyLoadContext

Then at a moment in time, I want to unload that AssemblyLoadContext. As result, all assemblies should be unloaded from the process.

Then I want to be able to replace the script (and any required assemblies), and do steps 1-4 again, all while the process keeps on running.

But as cs-script currently locks the referenced assemblies, it is not possible to delete or replace them.

Does this make it clear what I want to achieve?

[frank-thole]

I have tried your workaround.

public void LoadAssembly(string assemblyPath)
{
    var list1 = _evaluator.GetReferencedAssemblies();

    _evaluator.With(eval => eval.IsAssemblyUnloadingEnabled = true).ReferenceAssembly(assemblyPath);

    var list2 = _evaluator.GetReferencedAssemblies();
}

public void UnloadAndWait()
{
    lock (CSScript.EvaluatorConfig)
    {
        var oldValue = CSScript.EvaluatorConfig.ReferenceDomainAssemblies;

        CSScript.EvaluatorConfig.ReferenceDomainAssemblies = false;
        _evaluator = CSScript.Evaluator;

        CSScript.EvaluatorConfig.ReferenceDomainAssemblies = oldValue;
    }

    _evaluator.Reset(false);
    
    //var list3 = _evaluator.GetReferencedAssemblies();

    for (int i = 0; i < 1000; i++)
    {
        GC.Collect();
        GC.WaitForPendingFinalizers();
        Thread.Sleep(10);
    }

    var list4 = _evaluator.GetReferencedAssemblies();
}

I do now see that list4 goes down to 23 referenced assemblies.
However... I do see that the MQTTnet.dll is not unloaded from the process. And thus it can't be deleted/replaced.

I was hoping that the "fixed" Reset would release the MQTTnet.dll, but doesn't seem to be the case?

[oleg-shilo]

No, Reset() cannot possibly fix your problem. This is what I was trying to convey. Not, unless you start using AssemblyLoadContext (ALC), which is a complicated approach that you normally do not want to go with.

But let's clarify your Reset expectations first.

Note, when you call RoslynEvaluator.ReferenceAssembly(assemblyPath), you are simply instructing the instance of the evaluator to prepare the referencing metadata to be used as a reference when you execute the script. When preparing the metadata, the assembly will need to be loaded into the current AppDomain.

When you execute the script, it will be compiled into an assembly and then loaded in the very same current AppDomain, and an entry point (a method) of the assembly will be invoked.

If this method needs the referenced assembly (e.g. MQTTnet.dll), the CLR will automatically supply it at runtime since it is already loaded in the current AppDomain.

This is how script execution looks under the hood. BTW, it is the same concept that Roslyn implements.

So this is what is expected in your test.

var _evaluator = CSScript.Evaluator;

var refAsms1 = evaluator.GetReferencedAssemblies(); 
// the above line returns as many assemblies as are currently loaded in the 
// current AppDomain, as by default evaluator references them all

evaluator.With(eval => eval.IsAssemblyUnloadingEnabled = true);
// the above line does not do anything for the referenced assemblies 
// it only instructs the engine to load the future script with ALC so it can be unloaded later

var refAsms2 = evaluator.GetReferencedAssemblies(); 
// the above line returns as many assemblies as var refAsms1 + 1 

_evaluator.Reset(false);
// the above line clears all prepared assemblies referencing metadata stored in teh evaluator for future referencing
// this call is broken in your version but the workaround would fix it 

var refAsms3 = evaluator.GetReferencedAssemblies(); 
// the above line returns 0 as you just cleared them all and did not reference any new assembly

for (int i = 0; i < 1000; i++)
        GC....
// the above loop does not do anything at all

var refAsms4 = evaluator.GetReferencedAssemblies(); 
// the above line returns 0 as expected. Nothing has changed; you did not reference any new assembly since refAsms3 call. 

All this has been captured in this unit test:

cs-script/src/Tests.CSScriptLib/IssuesTests.cs

Lines 109 to 129 in 1af2e2c

	public void issue_464()
	{
	var assemblyPath = this.GetType().Assembly.Location;
	IEvaluator evaluator = CSScript.Evaluator
	.With(eval => eval.IsAssemblyUnloadingEnabled = true)
	.Reset(referenceDomainAssemblies: false);
	var list1 = evaluator.GetReferencedAssemblies();
	evaluator.ReferenceAssembly(assemblyPath);
	var list2 = evaluator.GetReferencedAssemblies();
	evaluator = evaluator.Reset(false);
	var list3 = evaluator.GetReferencedAssemblies();
	Assert.Equal(list1.Count(), list2.Count() - 1);
	Assert.Empty(list3);
	}

I suspect that if you do load and execute a script from the assembly that is collectable (loaded with ALC), then when you unload the assembly after the execution, you will be able to repeat the execution, but with a different version of the MQTTnet.dll.
But it goes so against the .NET assembly loading fundamentals that implementing it (and even testing) seems an unjustifiable effort. But, it's your call.

I hope this helps.

[frank-thole]

So you are saying that it is impossible to unload a DLL that is loaded through ReferenceAssembly?
What is the use of the "IsAssemblyUnloadingEnabled" option then, if the assemblies stay locked in the process, and can't be removed or replaced?

If we would load the DLL in an ALC ourselves, would we be able to use the CompileAssemblyFromFile function to compile our code (which references that DLL)?
Because else I don't understand your comment: "I suspect that if you do load and execute a script from the assembly that is collectable (loaded with ALC), then when you unload the assembly after the execution, you will be able to repeat the execution, but with a different version of the MQTTnet.dll." --> If we would still have to use ReferenceAssembly, the DLL would be kept locked, and we thus can't change it.

[oleg-shilo]

So you are saying that it is impossible to unload a DLL that is loaded through ReferenceAssembly?

No. But the question implies that the constraint is imposed by Evaluator.ReferenceAssembly(path). The constraint is actually a CLR constraint. You would face this problem even if you are not using scripting at all.

What is the use of the "IsAssemblyUnloadingEnabled" option then?

To allow unloading the script assembly. The script engine loads the script as an assembly, and you have an option to unload it.
However, the dependency assemblies (e.g. MQTTnet.dll) are loaded by your CLR, and the script engine has no control over them.

Basically, when your script is compiled, the script engine instructs Rosln: "Compile this script into assembly. When this assembly is going to be executed, it will need the host application to supply MQTTnet.dll (referenced assembly)."

Then you call your script, and CLR finds MQTTnet.dll file (assembly probing process) and loads it. After that, you cannot unload it.

I tried this trick. Before referencing the MQTTnet.dll file, I load it with ALC and then reference Evaluator.ReferenceAssembly(loadedAssembly). It works perfectly.

But when you try to call teh script assembly method, the CLR cannot locate MQTTnet.dll even though it is right there in the same appdomain. Custom assembly probing (AppDomain.CurrentDomain.AssemblyResolve +=...) does not help as returning the live MQTTnet throws NotSupportedException: Resolving to a collectible assembly is not supported.. It's just not intended to be done this way due to teh CLR design.

Because else I don't understand your comment: "I suspect that if you do load and execute a script from the assembly...

I meant to use ALC to "load and execute a script". Create a small assembly (e.g. JobAssembly.dll) that loads and executes your script with the corresponding MQTTnet.dll. Let's call this. Then you can load JobAssembly.dll with ALC, execute the job and unload it.

After that, I expect you to be able to repeat this even if the next time your job (script execution) loads a new version of MQTTnet.dll. But you probably can see how convoluted it is. And without the guarantee of work.

[frank-thole]

Thanks for your extensive reply.

I tried this trick. Before referencing the MQTTnet.dll file, I load it with ALC and then reference Evaluator.ReferenceAssembly(loadedAssembly). It works perfectly.

It works indeed, but the DLL then gets loaded twice:

And then on unload of the ALC, the one loaded in the ALC disappears and the one reference by the evaluator remains.

Let me try your other suggestion next.

[oleg-shilo]

Ironically, on .NET Framework, this task would be much easier to implement as the CLR supported multiple versions of dll to be loaded.
And even with that ALC work around, implementing it on .NET framework would be more manageable sine AppDomain unloading is supported.

Fundamentally it all comes down to a one single constraint that does not allow implementing it nicely: The assemblies that are loaded with ALC (collectable assemblies) cannot be referenced when compiling with Roslyn.

TBH I see not logical explanation fro this Roslyn limitation. It serves no obvious purpose. It's just there.