[spike] Better support for pre-existing codebases

[spencatro-pub]

agentblame should better support pre-existing codebases via optional initialization configurations. Marked as a spike to kick off the discussion, but below are some ideas.

Currently, the only initialization option is to assume an entire codebase is human-generated. This is a sane default, but certainly one with an opinion. Different teams with stricter requirements may want a different initialization strategy. (Personally, in an existing codebase that has used agentic generation before integrating agentblame, I would prefer to default all existing code to an "unknown" state, and thus force a human to take attribution over the lines they really care about.) Marking a line of code as unknown attribution would be a signal for humans that is a bit stronger than "line was human generated," but a softer signal than "line was AI generated," when it comes to the concern of "does this code require additional scrutiny?"

This feature might fit hand-in-hand with a mechanism similar to CODEOWNERS, where a repo can set up more fine-grained attribution rules, similar to how they might assign rules about test coverage or ownership. (Perhaps the source files in src/lib/auth/*.rs should be stricter about allowing non-human attributed code than say src/view/styles/*.css .) I'm imagining a CI pipeline that simply calls something like agentblame verify-attribution-rules to automatically reject PR's that don't meet some criteria. (The value here maybe being more about codifying an org's posture into actionable code that can generate actionable data, instead of e.g. a confluence / wiki page with no real enforcement or method of measurement.)

[warrenbhw]

Yes, I like this concept a lot, and totally agree that there should be some concept of "unknown" provenance.

Btw, any context on your particular motivation? General interest, or specifically looking to solve an issue with low-quality AI-generated contributions for an OSS repo?

[spencatro-pub]

General interest I suppose. My experience has taught me that it is really, really difficult to prove the quality of code. No approach is ever 100% perfect, but our best bet is still always discipline and observable metrics that can be reasoned about. I don't believe the genAI is going back in the bottle; organizations are going to have to reckon with contributions that were not written by hand, that maybe haven't even been scrutinized beyond "it seems to work." I'm guilty of it myself in my own projects, and even in that micro, I want a way to know where the risks are hiding. I haven't had a ton of experience dealing with "low quality AI contributions," but yeah, that's a pretty obvious outcome of wider genAI adoption.

My interest is mostly coming from the belief that orgs / projects that identify the risks, assign metrics to them, and seek to optimize (or at least understand) those metrics based on their needs are going to have a better time than those that simply don't. (Murali said it well, you can't fly blind.)