Add option to include untracked files in git and mercurial fetchers

[jfroche]

It can be confusing (especially for new users) that the git and mercurial fetchers do not include untracked files. This commit adds an option for the fetchers to include untracked files.

The option is disabled by default and still take into account the repository ignored files.

Priorities

Add 👍 to pull requests you find important.

This subject has been referenced before in
#8389 #7107 #3231 #6858

[sellout]

I agree that the current behavior is confusing and odd. If there is a line to be drawn, it should be between committed/uncommitted or maybe staged/unstaged. But the current split includes some changes that aren't expected to be committed, but not others. Untracked is odd on its own because there's no way to make it unstaged. The closest you can get is intent-to-add, which stages an empty file, leaving the staged changes in a state that you probably don't want to commit.

It's also odd because the current behavior will fetch the whole tree if there is no repo, but as soon as you git init, the fetcher fetches nothing.

This PR at least gives the option to avoid this oddness, without change the behavior for any existing usage.

[roberth]

I'm in favor of this, but unfortunately it seems that the git part of the implementation will have to be redone after #9240, which is a priority because it is a (long awaited) fix for reproducibility issues with git.

Mercurial might suffer from similar issues, but we haven't made a priority out of stabilizing that fetcher yet.

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/the-garn-devlog/36297/2

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/use-nix-file-excluded-from-git/37196/7

[NobbZ]

What is the difference between this flag and using path-type explicitely?

[v-kat]

Is this still planned for release? It'd be helpful with some agenix issues I was facing

[jfroche]

@roberth I realize I may not have been clear earlier. Three days after your comment, I've pushed an implementation that should align with the new approach based on libgit2. I'm curious about the likelihood of it being reviewed and eventually merged. Any insights on this?

[jfroche]

What is the difference between this flag and using path-type explicitely?

New users are most of time unaware of the distinctions between running it within or outside a Git repository, often leading to confusion. This option explicitly clarifies its intended functionality.

This implementation also overlooks files that are specified in the .gitignore.

[Jonas-Sander]

This implementation also overlooks files that are specified in the .gitignore.

Just bumped into this, would love for this restriction to be removed.

[NobbZ]

This implementation also overlooks files that are specified in the .gitignore.

Given the various ignoremechanisms in git (global [XDG_CONFIG_HOME/git/ignore], shared/in repo [REPO_ROOT/.gitignore], local [GITDIR/info/exclude]) we have to decide between confusing 2 groups of poeple…

1. When I run the same command with the same manual changes, why does it produce a different result on my 2 computers
2. Why is my entry in the global/local ignore file not respected and the not-meant-to-appear file copied into the store?

---

edit add missing `

[jfroche]

This implementation also overlooks files that are specified in the .gitignore.

Given the various ignoremechanisms in git (global [XDG_CONFIG_HOME/git/ignore], shared/in repo [REPO_ROOT/.gitignore], local [GITDIR/info/exclude]) we have to decide between confusing 2 groups of poeple…

1. When I run the same command with the same manual changes, why does it produce a different result on my 2 computers
2. Why is my entry in the global/local ignore file not respected and the not-meant-to-appear file copied into the store?

edit add missing `

Ah maybe I wasn't clear in my last message, we keep using libgit2 so all the ignore files should be taken into account.

[NobbZ]

@jfroche As said, that confuses one group of people, as they might get different results on different computers, with seemingly same settings because there is something in the global ignore, that makes a file not being available to nix.

[sellout]

How about something more like

    Setting<std::string> includeAdditionalFiles{this, "none", "include-additional-files",
        "Whether to include untracked files in Git/Mercurial trees."};

and the values can be "none", "untracked", or "untracked and ignored"?

The "untracked" option still has the uncertainty around locally-ignored files, but I doubt libgit2 provides any finer distinction where one could include some ignored files.

[jfroche]

Maybe #12884 is good enough ?

[tomberek]

Partially addressed in 12884. Potential to improve the message. Not planning to change the default at the moment. Will need another PR to improve the message for files other than flake.nix. (maybe #12915)

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2025-05-04-nix-team-meeting-minutes-230/65206/1