Add option to ignore VCSError

[rochacbruno]

While in development, I want o be able to run flit build before commiting and adding files to my github repo.

if I do that I get

$ flit build
flit.common.VCSError: Untracked or deleted files in the source directory. Commit, undo or ignore these files in your VCS. (.)

I want to be able to do:

$ flit build dev
#or
$ flit build --novcs

So I can test my builds before commiting

[takluyver]

I've deliberately started off with this strict with this to see how it works out before relaxing it. To help me understand the need:

• What kind of files do you have that aren't tracked by the VCS? Are they new code (which should be in the sdist), or odds and ends for testing which you're not going to commit?
• Why are you calling flit build? Do you want to check that it succeeds? Or do you want to use one of the built files? And if so, what do you want to use it for?

The idea is that you need to make a positive decision on which files belong in the sdist. Since I'm trying to avoid something like MANIFEST.in, the VCS is how we decide that. The error message isn't quite accurate: you don't actually need to commit new files, it's enough to git add them (or ignore them).

Depending on what you want this for, you can run flit wheel during development - that doesn't rely on the VCS, so it's not affected by the check.

I want to avoid adding a 'eh, whatever' flag which just bypasses the check, because developers are lazy, and before long people will just use that flag all the time.

[rochacbruno]

maybe add a test mode?

I want to test my builds before commiting and releasing, so flit test-build would be nice.

I see your motivation and I agree, but you can keep the constraints on publish command and allow one to test the build without publishing it.

[takluyver]

I want to test my builds before commiting and releasing

Do you mean you want to test that building works? Or you want to build it so that you can run tests? And if so, how are you doing that?

[rochacbruno]

There are some cases

1. I want to assure the build happens without error
2. I want to test the builded package is installed with no errors
3. I want to pack the builded package as other packages (.deb or .rpm for example) I use a tool which takes the .tar.gz and repacks it for distribution on other formats, so I want to test it.
4. I want to ensure the assets are there, static files for example and also assure the description generated by my Markdown to Rst is good.

I know it is not a common case, not everyone tests the builds like this.

[Carreau]

Why not commit everything, and test without pushing ? If all works great, than all is good, otherwise just --amend your commit ?

[takluyver]

Reiterating, but you don't actually need to make a commit - just git add any new files which you want to go into the sdist. You can make more changes before committing them, or unstage if you change your mind.

I'm going to spin a background thread on how we could make this easier - in particular, if there's something better to do than making an option to skip the check.

For the check-the-build-works part, one possibility is to add a command like flit check which does all the checks involved in making a build, but doesn't produce a result (or produces one and discards it). So you'd still see a message about untracked files, but it wouldn't prevent the other checks.

[carlbordum]

I think there's just a bit confusion here, because the exception message doesn't make it clear that you just need to stage all files. Maybe replacing "Commit" with "Stage" will help users that run into this in the future?

"Untracked or deleted files in the source directory. Stage, undo or ignore these files in your VCS."

If mercurial has different terminology we can have different messages based on the vcs. Then we could add the helpful but specific git add command to the message to help users even faster.

[richieadler]

I didn't understand the method flit used to find out what files would be part of the package... apparently somebody needs to create a module for Fossil. Until then, I'm out of luck, apparently.

[takluyver]

If you want to have a go at writing that, it shouldn't be much work - depending on how tricky it is to do the required operations with fossil. It needs:

• A way to detect that a directory is a fossil checkout (the equivalent of checking for the .git directory).
• A function to list the files in a given directory which are tracked by fossil.
• A function to list the files added and deleted in a given directory but not updated in fossil.

[richieadler]

You were right, it wasn't much work.
I'm not comfortable with pull requests yet, but I created the following and it worked OK in my system and I propose them for inclusion if you like to have Fossil support added:

• fsl.py to detect fossil and list the files as indicated;
• a modified __init__.py so you can use fsl.py

I'm attaching them in this ZIP file: fsl.zip

[takluyver]

Thanks, I've turned your files into pull request #224.

[BuongiornoTexas]

A brief comment from someone learning python - I'm putting my first package together and I've just hit the brick wall that flit appears to require that package code is in VCS (at least 'flit build' crashes if you try to run it on a package not in VCS). This requirement isn't in the documentation.

This requirement is hard on newcomers - I'm struggling to learn the mechanics of packaging based on assembling and distributing a small package I'm developing locally. I'm happy enough with my local source code control in the short term. So I'm deferring getting to grips with git until I have a proper handle on packaging (avoiding "trying-to-learn-too-many-things-at-once" syndrome).

It would be nice if flit could provide a sand box/testing mode that novices like myself could use for local packages. Particularly so given that all of the pypa examples and tutorials are based on local packages rather than using a VCS.

Could I suggest either updating the document to specify that flit only works on packages in VCS, or request that flits implements support for local, non-VCS packages?

Finally, my apologies - not quite as brief a comment as I planned!

[richieadler]

I think if Git proves to be too complex for you and you are happy with local SCC, you may want to try Fossil <fossil-scm.org>. There is a branch (still unmerged, I think) where there are some additions I created to use flit with Fossil.

[BuongiornoTexas]

I think if Git proves to be too complex for you and you are happy with local SCC, you may want to try Fossil <fossil-scm.org>.

Thanks for the steer Richie - if I was sticking with local SCC, I'd definitely look into Fossil.

The problem isn't complexity. It's just that I had a lot of balls in the air trying to understand how python packaging works - especially given that from an outsiders perspective, I'm trying to do this in the middle of a packaging war! Adding learning how VCS or local SCC works and integrates with my IDE did not add to the fun.

In the end I've bitten the bullet, read an awful lot of sometimes contradictory documentation, and got github, setuptools, twine and pypi working for me. Quite a lot red wine followed.

I'll amend my previous comment though: the requirement for VCS in flit isn't newbie friendly. That said neither is the pypa recommended approach!

While I'd prefer flit for simplicity, I'm going to hold off for now until there is some sort of resolution on #236.

[skullydazed]

I just came across this issue because I'm setting up flit for the first time on my first project. I thought I wanted a bypass flag when I got here, but now I've changed my mind.

Perhaps you could avoid the user confusion by using the word "Add" instead of "Commit" in the error message? If the error message hinted that a git add . was all I needed to be able to continue it would have made the lazy thing also the right thing.

[miccoli]

Maybe I'm a VERY sloppy programmer, but usually I have test files lying in my working tree, that I do not want to add to the index. They are just there for future work, testing, todos, etc. I know how to manage a correct build/test/VCS process: the extra-security that makes impossible to shoot myself in the foot is just annoying.

In other words, what's the point in forcing me to add to the index a todo.txt file that is in a place that will not be included in the source dist or the python wheel?

(Just my opinion, so please feel free to ignore my comment, If you think that this is an important flit feature.)

[takluyver]

By default, the source dist includes everything that's tracked in your VCS under the folder containing pyproject.toml. So it gives this error because it doesn't know whether the files should be included in the sdist or not.

Since Flit 2, you can include & exclude files in the config file (https://flit.readthedocs.io/en/latest/pyproject_toml.html#sdist-section ). So you could have a scratch/ subfolder and put it in the exclude list. But you could also add it to .gitignore, which should have the same result.

[miccoli]

I understand the flit approach this way:

• use the index to decide what has to be included in sdist, but build against the working tree.

With respect to this approach we have three different cases

1. files in the index but missing in the working tree. This is an error, since I'm instructing flit to add to the sdist a non-existing file.

2. files in the working tree but not in the index. This should be a warning, and those files should not be included in the sdist

3. files in the working tree modified with respect to the index. This should be a warning, since I'm building (and possibly publishing) a version which is not yet consolidated under VCS.

Currently for flit, 1. and 2. are an error and 3. is ignored.

My suggestion: make 2, and 3 warnings, and add a way for silencing them, something like export FLITVCSUNTRACKED=ignore and export FLITVCSMODIFIED=ignore, on the line of PYTHONWARNINGS

Finally, regarding your suggestion about a scratch directory: I think that it is a bit intrusive from flit to dictate how I should organise my working tree, and where and when I have to put/commit my files. If my VCS hygiene is not up to the flit standard, I would kindly accept a suggestion (i.e. a warning) but the fact that flit refuses to work, without a compelling reason, in my personal setup, is a great annoyance.

But again, this is just my personal view, so thank you for your comments.

[merwok]

Great to see the new exclude option! So far I have added entries to .hgignore, without committing the changes to that file, which felt like flit making me jump through hoops and not really helping me.

To answer your use case question:

• I have notes, snippets, experiments (i.e. adding an npm build) that I don’t want to commit yet or at all
• when adding a new module, I like to have tests pass before I add, and tox needs flit build to work

The first case seems to be covered by exclude/include (by the way, moving to flit 2 was painless), or I can keep the .hgignore trick. For the second case, I am slowly being converted to flit’s viewpoint. I still have the options to run my app without build (thanks to develop mode / flit install --symlink), I can run pytest that doesn’t need a build, and when I run tox before committing, an error from flit about adding the file is actually better than a mysterious import error later during tests.

[erinn]

I believe I have a reasonably valid use case for this.

While building an RPM we don't wish the .git directory to be part of the sources thus we have a clean root without VCS but I want the ability to flit build the wheel then install the wheel where needed. As it stands now without the ability to ignore the VCS requirement, it doesn't appear flit build can be used in an RPM spec file (from what I have found).

-Erinn

[takluyver]

@erinn - this should no longer be a requirement with Flit 2. Now, if it can't find a VCS, it will build the sdist with just the same files that would go into the wheel.

Edit: the same files give or take inclusions and exclusions specified in pyproject.toml : https://flit.readthedocs.io/en/latest/pyproject_toml.html#sdist-section