Don't ignore unstaged files in local flakes

[SuperSandro2000]

The current behavior to just ignore unstaged files is unintuitive and super annoying. Also the warning that the worktree is dirty makes you expect that unstaged files would be included.

I've been using this patch since a month and it made my development workflow so much easier and I no longer commit nix files by accident because flakes made me do a git add.

[ElvishJerricco]

Maybe this could be a config or command line option or something. I think the current behavior is the right default. It's far too easy to add tons of superfluous files like nixos.qcow2 images and the like if untracked files are included, which really slows down copying the flake to the nix store and wastes a ton of space.

[edolstra]

Yeah this behaviour should be configurable. Note that in this mode, if any untracked but unignored file exists in the worktree, then the repo should be considered dirty. So the dirtiness check (i.e. where we call git diff) needs to be modified as well.

[edolstra]

Note that in the lazy-trees branch (#6530), you get a warning if you try to access an untracked file, e.g.

error: access to path '/home/eelco/Dev/nix/.version2' is forbidden because it is not under Git control; maybe you should 'git add' it to the repository '/home/eelco/Dev/nix'?

I no longer commit nix files by accident because flakes made me do a git add.

I mean, you're supposed to add Nix files and anything else used by the evaluator to ensure reproducible evaluation. The alternative is that evaluation works for you but fails for somebody who tries to build the same Git revision.

[SuperSandro2000]

Note that in the lazy-trees branch (#6530), you get a warning if you try to access an untracked file, e.g.

Its good that it is no longer silent.

you're supposed to add Nix files and anything else used by the evaluator to ensure reproducible evaluation

Completely on your side but when you are still developing stuff locally and maybe even work on two changes at once then adding files is not an intuitive workflow. At that point it also does not need to be reproducible and you are still getting the dirty tree warning which also indicates that the tree is not reproducible.

Yeah this behaviour should be configurable.

thats above my c++ knowledge

[SuperSandro2000]

I am not sure what is wrong with the tests. I wouldn't expect my changes to break flake.lock's.

[ilkecan]

... I no longer commit nix files by accident because flakes made me do a git add.

-N (--intent-to-add) git option can be used to avoid this.

[mightyiam]

Sorry for going against the grain, here. I feel against ignoring untracked files by default. I may not be the most experienced Nixer, but relying on whether a file is tracked or not seems like an anti-pattern to me.
Git is for managing code. It shouldn't be used for the behavior of it. I realize that Nix is in the domains of building software and therefore figuring out "what is included in this software?" is part of the domain, but relying on whether a file is tracked or not seems bonkers to me. I shouldn't have to git add a file in order to make it part of the software. It's part of the software because it exists.

End of thoughts from an inexperienced Nixer who doesn't have all the context.

[ElvishJerricco]

It's part of the software because it exists.

This seems like a bad assumption. My example above was nixos.qcow2; a large file created by doing VM based testing with nixpkgs. There are plenty of files that you wouldn't want to have copied to the world readable store just because you're messing with them in you flake cwd.

[mightyiam]

This seems like a bad assumption. My example above was nixos.qcow2; a large file created by doing VM based testing with nixpkgs. There are plenty of files that you wouldn't want to have copied to the world readable store just because you're messing with them in you flake cwd.

Sorry. I don't know the details. This "feature" (no offence) is to prevent some otherwise implicit copying to the store?

I don't remember what error brought me here, but is that error necessarily one that occurs after that copying to the store?

[Ericson2314]

Triaged in the Nix Team meeting this morning (n.b. I could not make it):

• We should wait for lazy trees before changing the behavior.

• Improvements to the error message would be great though in the meantime if it's not too costly to implement (@thufschmitt had a PR doing something like that)

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2023-08-04-nix-team-meeting-minutes-77/31487/1

[nrdxp]

Aren't there security implications to this change? What if you have untracked secrets in repo that are not world readable, but now Nix just copies them to a world readable location in the /nix/store? I suppose you could alleviate that by respecting .gitignore rules, but just thought I'd mention it since I didn't see it brought up here yet.

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/why-cant-nixos-rebuild-see-files-that-arent-in-the-git-repository/49053/5

[tomberek]

@Ericson2314 Presumably with the new SourcePath and Accessor infrastructure, this should be done as a new GitSourcePath?

[thislooksfun]

Since this output is being parsed in a script it would probably be better to use git status --porcelain, as that output is guaranteed to be stable (whereas git status --short is not).

[thislooksfun]

FYI the else { and } lines use tabs to indent, while the rest of the file uses spaces.

[thislooksfun]

Would it be possible to have an option to not run the standard excludes? I would like to be able to have .gitignored local files that still get applied in my nix config. I don't care about other users/programs on the computer reading them, they just don't make sense to be in the upstream repo.

(Also see #12291 for an alternative approach.)

[MagicRB]

This has bit me in the behind so many times I don't even get why that's a thing in the first place this is the dumbest design decision. I don't get why doesn't just copy only files that are referenced by modules in the store and that only, this makes no sense whatsoever, there's like a billion better ways to do this

The thing about your idea is that youre interleaving evaluation and the the initial setup. Its certainly possible, but very difficult to do. You'd have to essentially start evaluation and then copy files one by one as they're being accessed by Nix. The current evaluator also cannot "yield and go do soemthing else" so while copying you'd block evaluation. It would slow down evaluation a lot.

[thislooksfun]

You'd have to essentially start evaluation and then copy files one by one as they're being accessed by Nix.

That's in the works already: #11367 / #6530

[mawkler]

I found this issue after an hour of troubleshooting because Nix suddenly couldn't find hardware-configuration.nix when doing sudo nixos-rebuild switch after I tried switching my configuration to using flakes.

As a new Nix user this seems like absolutely crazy behaviour. However, if Nix really has to be coupled to Git, it should at least give me a hint in the error message stating that the reason it can't find the file might be that Git isn't tracking it.

I would have prefered having hardware-configuration.nix ignored by Git, but I guess that's not possible.

[XSAM]

I thought the point of nix is to make deployment reproducible and scalable. With the requirement to bring hardware-configuration.nix in the git where it contains generated hardware info that only binds to a certain machine, how could it scalable?