Conversation
A simpler AI policy document to start
Fix some typos and formatting, and clarify some points.
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
joelaforet
left a comment
joelaforet
left a comment
There was a problem hiding this comment.
Made some comments + suggestions.
|
|
||
| AI models often hallucinate comments or reinvent existing utilities. | ||
|
|
||
| - You should existing `mupt` utilities and never reinvent the wheel, except for when the utility is not available. Creating new helper functions when a MuPT equivalent exists will be rejected |
There was a problem hiding this comment.
| - You should existing `mupt` utilities and never reinvent the wheel, except for when the utility is not available. Creating new helper functions when a MuPT equivalent exists will be rejected | |
| - You should use existing `mupt` utilities and never reinvent the wheel, except for when the utility is not available. Creating new helper functions when a MuPT equivalent exists will be rejected |
| - An AI agent (e.g., Cursor, GitHub Copilot, ChatGPT, etc.) generated the code or commit messages | ||
| - You cannot explain the logic without referring to the AI's output | ||
| - The PR was created primarily by an agent with minimal human review or modification | ||
| - This should not currently submitted to MuPT |
There was a problem hiding this comment.
This should not currently submitted to MuPT
Is this really the policy? Seems like we're blocking agents writing code + generating commits even if the author has full ownership of the code that is produced. I read this as contradicting the previously mentioned "Explain It" standard.
|
|
||
| All PRs must accurately describe the actual use of AI tools. | ||
|
|
||
| **When to mark as 🔴 AI-generated:** |
There was a problem hiding this comment.
I personally don't like the implied requirement that we need to tag all of our code contributions with emojis. I've seen people in the OS community have a short line at the bottom of their commits/PRs that read "🤖 Generated with Claude Code". See patterns in PyLabRobot by @rickwierenga
dwhswenson
left a comment
dwhswenson
left a comment
There was a problem hiding this comment.
Overall take is (as I often say with AI): good patterns for AI are also good patterns for humans. Making restrictions AI-specific isn't needed if you make it a core requirement of all PRs (as it should be).
The one point I'd argue especially strongly for is, in the section about "don't use AI to write your words," adding exceptions for translation/grammar. Unless you want to take the opposite direction (which some projects do!) of saying "write in your native language; the reader will translate it." Other suggestions are less firm.
I'll add a link in Slack to (a pinned version of) a document I'm working on related to this for OMSF. That doc isn't quite ready for public sharing yet, and the linked version is a copy in the MuPT drive that won't be updated as the main doc gets updated. However, I spent some time last month exploring what I thought would be the core principles for this, and that doc is what came out.
| AI tools frequently write code that looks correct but fails execution. Therefore, "vibe checks" are insufficient. | ||
|
|
||
| - Every PR introducing functional changes must pass all tests that are currently extant. This is mandatory for all contributors and is particularly important for first-time contributors. | ||
| - All code with AI contributions should include tests to demonstrate their validity. Intially, as the testing framework develops, this may be relaxes, but will become mandatory in the near future. |
There was a problem hiding this comment.
Theme that I'll repeat a few times: Many rules for agents are equally good for humans. I see no reason to call these out as "AI contributions" -- the same standard should apply to human contributions. Tests should be required regardless of the level of AI contribution.
|
|
||
| ### The "Explain It" Standard | ||
|
|
||
| - All pull requests messages and descriptions must be entirely authored by human. Individual commit methods may use AI tools to fully summarize the changes. If the PR is suspected to be AI, the submitters will be asked to rewrite the PR message from scratch. |
There was a problem hiding this comment.
Generally recommend exceptions for translation or grammar improvement (thinking of use by non-native speakers).
| **When to mark as 🔴 AI-generated:** | ||
| - An AI agent (e.g., Cursor, GitHub Copilot, ChatGPT, etc.) generated the code or commit messages | ||
| - You cannot explain the logic without referring to the AI's output | ||
| - The PR was created primarily by an agent with minimal human review or modification | ||
| - This should not currently submitted to MuPT |
There was a problem hiding this comment.
I have a couple of concerns about this section:
- "Mark it this way, but don't submit it" -- this is contradictory. Let's not talk about how to mark it if it isn't allowed.
- It is unclear to me whether the bullet points are joined by "AND" operators or "OR" operators. If "AND" operators, sure, don't accept this. If "OR" operators, IMO, the real gatekeeping is "can you explain it."
- Following on 2, I'm not sure that the other points should be gatekeepers. For example, I recently had a number of "
pkg_resourcesis longer available by default in my environment, update toimportlib" issues, where basically I said that to an agent, skimmed the changes, ensured it passed tests, and called it good. That's "created primarily by an agent with minimal human review or modification", but also a good use of AI to cut my time to solve the problem by a factor of 5 or more.
|
|
||
| AI models often hallucinate comments or reinvent existing utilities. | ||
|
|
||
| - You should existing `mupt` utilities and never reinvent the wheel, except for when the utility is not available. Creating new helper functions when a MuPT equivalent exists will be rejected |
There was a problem hiding this comment.
This is another "applies to humans as well as AI." Calling a for hard reject of a PR based on that may be extreme.
3 participants
A simpler AI policy document to start with
Description
This provides a simpler description of the current AI policy
Todos
Notable points that this PR has either accomplished or will accomplish.