Global lifecycle hooks

[aleperaltabazas]

Versions

Godot 4+

The Feature

GUT offers the possibility for each test class that inherits GutTest to redefine before_each, before_all, after_each and after_all hooks to execute certain code before or after each event. However, this is limited to each test suite and does not allow to register common behaviour for each event and thus is necessary for state to reset to duplicate code all throughout.

One simple solution to this problem exists, which is editing the GutTest class (or extending it with a custom script) to redefine the hook methods, but this also requires the explicit call of super in every other test, which may be error prone.

I suggest for GUT to have a mechanism to register global hooks in any point and for them to affect all tests, similar to Kotest's extensions system.

The following is an example of what the extension could be like:

# defines methods for the hooks with default, null behaviour for every one of them, user is expected to override whichever method/s they need
class GutExtension
extends RefCounted


# called before every test in every suite
func before_each():
  pass


# called once for every suite, before every suite
func before_suite():
  pass


# called only once before any test is run
func before_any():
  pass


# called after every test in every suite
func after_each():
  pass


# called once for every suite, after every suite
func after_suite():
  pass


# called only once after every test is run
func after_every(): # bad name, expressive but doesn't match with before_any
  pass

As for using it, the following is an example use from a pre-run hook script.

extend GutHookScript


class CustomHook extends GutHook:
  func before_each():
    reset_my_state()

  func after_all():
    tear_down()


func run():
  GutMain.register(CustomHook.new())

I'm working on a little PoC on the project I'm working right now, as I commonly need to reset state to defaults, and I end up having a lot of before and afters with the same code. Would it be fine if I extract it into a fork to show the way it could be used?

[bitwes]

similar to Kotest's extensions system.

Interesting. If nothing else, I think we should lobby Godot to allow us to use fun instead of func. override suspend fun...does that mean you are preventing fun from being suspended, as to ensure the good times continue?

Would it be fine if I extract it into a fork to show the way it could be used?

Absolutely. I can see the need for more hooks/setup/teardown. I haven't run into any cases yet myself, but I think I created an issue about adding more hooks (at least a pre-script and post-script hook).

or extending it with a custom script

This is what I do, and it is expected that people would do this. I would not recommend editing GutTest directly as that is a maintenance nightmare. In most of my projects I end up making a new base script that extends GutTest that all my tests for that project extend. I've settled on this as my standard:

# res://test/resources/project_test.gd
class_name ProjectTest
extends GutTest

As you mentioned, if you need specific setup/teardown in a script then you do have to remember to call super.

Terminology

Does "suite" == "script" in GUT's case? Basically an external before_all/after_all?

before_any sounds like the current pre-run hook script to me.

after_every sounds like a post-run hook script.

[aleperaltabazas]

override suspend fun...does that mean you are preventing fun from being suspended, as to ensure the good times continue?

That's specific to Kotlin. Kotlin also has coroutines but they're managed quite differently from Godot, so I don't think that bit applies to this.

Does "suite" == "script" in GUT's case? Basically an external before_all/after_all?

Correct, maybe before_script or before_test (or after) is a better name. I'd mainly avoid the before_all/after_all names because it's a bit confusing whether it's before or after all tests in all scripts are executed or once per script.

before_any sounds like the current pre-run hook script to me.

after_every sounds like a post-run hook script.

Yeah, you're right, there's probably no need for those hooks then.

Absolutely. I can see the need for more hooks/setup/teardown. I haven't run into any cases yet myself, but I think I created an issue about adding more hooks (at least a pre-script and post-script hook).

Great! I'll come back after a few days of tinkering.

[bitwes]

I thought I should jot this down while I was thinking about it. I checked the issues and there is one issue about adding hooks but it wouldn't help with this. What do you think about adding two new hooks; one for pre-script and one for post-script? The run method for these would be passed the GutTest instance. I think that would get you before_suite and after_suite.

What I don't like about this approach is that you have to have four separate scripts to do what you want. It might be better to introduce a new type of HookScript, similar to what you originally posted that has a method for each "event".

In GUT terms it might be:

func pre_run()
func pre_script(script_instance)
func post_script(script_instance)
func post_run()

Side note, these two new hooks could allow people to implement a categorizing system to run subsets of tests if the pre_script method allowed you to skip a script somehow.

We might be able to add in all the setup/teardown methods too (essentially implementing everything you are proposing).

func before_all(script_instance)
func before_each(test_method_callable)
func after_each(test_method_callable)
func after_all(script_instance)

Stuff I'm not sure about:

• I think the before/after methods should be called before the script version but maybe it needs to be configurable.
• If the before/after are called after the script methods then there might be some magic we could do that would allow the scripts to invoke the hook script equivalent when it wants, and not call it automatically if that happens.
• Maybe this monstrosity solves the previous two points?

func before_before_all(script_instance)
func after_before_all(script_instance)
func before_before_each(test_method_callable)
func after_before_each(test_method_callable)
# etcetera

• Should before_all be able to skip the script?
• Should before_each be able to skip the test?

[bitwes]

I have no idea why I didn't think of this earlier. gut.gd has signals for all this stuff already, and the pre-run hook has a reference to gut. Also, gut retains a reference to the pre-run hook until the end of the run, so that it can be passed to the post-run hook (though making any signal connections would probably keep it alive, idk). The new hooks scripts might still be a good idea, but I think you do everything you want with the existing ones.

Here's an example of a pre-run hook script that ties into all the signals. This wasn't tested, I just started typing and this is where I got.

extends GutHookScript

const NO_TEST = "__NO_TEST__"
var _current_test_script_object : GutTest = null
var _current_collected_script = null
var _current_test_name = NO_TEST

func run():
	gut.start_run.connect(_on_run_started)
	gut.start_script.connect(_on_script_started)
	gut.start_test.connect(_on_test_started)
	gut.end_test.connect(_on_test_ended)
	gut.end_script.connect(_on_script_ended)
	gut.end_run.connect(_on_run_ended)

	# Do pre-run stuff here

# This might be redundant, and it might have already been emitted byt the time
# this hook is called.  I wanted it in here for illustration purposes.
func _on_run_started():
	pass

# This is passed an instance of res://addons/gut/collected_script.gd.  It is not
# the instance of the script that will be run.  You can get to the script object
# using `load_script`, but you can't get to the actual instance that will be
# run.
func _on_script_started(collected_script):
	_current_collected_script = collected_script
	# The GutTest script, not the instance.
	_current_test_script_object = collected_script.load_script()


# This is just the name of the test.
func _on_test_started(test_name):
	_current_test_name = test_name


func _on_test_ended():
	# example of inspecing the test that ended if you wanted to.
	if(_current_collected_script.get_test_named(_current_test_name).is_failing()):
		pass

	_current_test_name = NO_TEST


func _on_script_ended():
	#example of inspecing the script that ended if you wanted to
	if(!_current_collected_script.was_skipped):
		if(_current_collected_script.get_fail_count() > 0):
			print("The script ", _current_collected_script.get_full_name(), ' failed.')
		else:
			print("cool.")
			
	_current_collected_script = null
	_current_test_script_object = null


# I'm not sure if this is called before or after the post-run hook.  This can't
# do everything a post-run hook can do, but it might be enough for this.
func _on_run_ended():
	pass
	# Do "after_every" things here.

Here's a link to the collected script class https://github.com/bitwes/Gut/blob/main/addons/gut/collected_script.gd

Good luck!