-
Notifications
You must be signed in to change notification settings - Fork 35
Description
Migrated from: goodfire-ai/spd-gf#51
Original author: @leesharkey
SPD produces a set of subcomponents which need to be clustered together in order to identify parameter components. Some of the models in our evals suite will have parameter components that must be clustered together from subcomponents. We will want to evaluate the results of this clustering process in order to test our analysis pipeline end to end. If either SPD or clustering are broken, then we will end up with poor results. So we need to evaluate our clustering method.
It will probably suffice to use the clustering methods on our toy models where we have ground truth. We can usually identify if SPD has succeeded without evaluating the subsequent clustering. But the clustering should still work! So we should test the clustering separately from our SPD metrics.