[Beginner] How to reduce the number of false positives?

[duboism]

Hello,

We are beginners with lychee and we would need a bit of advice.

We started to add the GitHub action in our repository which mostly contains jupyter notebooks (in case you're not familiar, it's basically a list of "cells", each cell containing python code and the result of its evaluation, stored in a JSON document) and Markdown files. It helped us to spot a few dead links and some that could be upgraded to HTTPS. However, there are also some false positives (see gw-odw/odw#26) and I would like to know the best way to silence them.

We have 3 main categories:

1. URLs of the form https://github.com/gw-odw/odw/raw/main/Tutorials/Day_2/Data/%7B%7D/ is in fact the python string https://github.com/gw-odw/odw/raw/main/Tutorials/Day_2/Data/{}/ used to format URL. I guess the best option is to use some regex in the .lycheeignore file but I'm not sure of the format given that it contains special characters.
2. URLs ending with /n (like https://github.com/gw-odw/odw/raw/main/Tutorials/Day_2/Data/PyCBC_T2_1.gwf/n): I think this comes from using the aforementioned string to produce an actual URL and printing it (which adds a \n); this is stored in JSON so newlines are written explicitly and are then escaped. Not sure what to do here.
3. Last but not least, there are a bunch of file:// URLs that I think are not worth testing. Any advice on how to exclude the file:// scheme ?

[mre]

Welcome to lychee! 👋

I took a look at the repo and can see the false positives.

At the moment, we extract links from Jupyter Notebook files using a simple plaintext extractor, which explains the false-positives. For deeper inspection, we'd need to parse the file format. There is a crate for that! I've opened #1659 for that.

In the meantime, you can exclude the broken links with a custom .lycheeignore:

# Issues with placeholder links in Jupyter Notebook files.
# Remove once https://github.com/lycheeverse/lychee/issues/1659 got resolved.
https://dcc.ligo.org/public/0182/T2200137/001/O3bPE_downsampled.tar.gz/n
https://dcc.ligo.org/public/0157/P1800370/005/GW150914_GWTC-1.hdf5/n
https://raw.githubusercontent.com/gw-odw/odw/main/Tutorials/Day_3/toy_model.csv/n
https://github.com/gw-odw/odw/raw/main/Tutorials/Day_2/Data/PyCBC_T2_1.gwf/n
https://github.com/gw-odw/odw/raw/main/Tutorials/Day_2/Data/PyCBC_T2_2.gwf/n
https://github.com/gw-odw/odw/raw/main/Tutorials/Day_2/Data/PyCBC_T3_0.gwf/n

# Legitimate exclude; not a bug.
https://lscsoft.docs.ligo.org/bilby/examples.html

# In the document, `%7B%7D` this is actually `{}`. This is hard to fix with lychee since it can't understand placeholders.
# So this is another valid exclude.
https://github.com/gw-odw/odw/raw/main/Tutorials/Day_2/Data/%7B%7D/

# Exclude all file-links. Workaround until https://github.com/lycheeverse/lychee/issues/1646 is resolved
file://

A few things I like to do:

• I don't exclude links using regular expressions if possible. Instead, I list every link "verbatim." That's because a catch-all could lead to false negatives -- e.g. you exclude too much and now you don't check those links anymore. Better to be explicit.
• I add commments about why I excluded a link. Makes it easier to follow up later and remove the excludes when an issue gets fixed as is the case for excluding file links, which we tackle in Check external links only #1646. Your input would be helpful over there, since we are currently looking for a good name for that option. Something like --exclude-local-files.
• From time to time, I start over with an empty .lycheeignore to see which exclusions I really need. Maybe we fixed a bug or the link is no longer there.

Hope that helps.

[duboism]

Oh no it doesn't help, it brilliantly solves the current problem and provide both great advice and a path for the future. This is one of the best answer I have read on a forum (and one of the kindest I might add)! Thank you very much.

I am just not sure why you excluded https://lscsoft.docs.ligo.org/bilby/examples.html. If I'm correct, this is a variation of the second problem with links in markdown cell (actually, I missed this case) so should be covered by #1659. Does it relate to your second point?

[mre]

Oh, yes, you're correct. It would be resolved with #1659. The full link is https://lscsoft.docs.ligo.org/bilby/examples.html\n. I misread it as example.html, which is a common placeholder link, but it turns out that it's a legit website. That was my mistake. It's probably just a side-effect of staring at links for too long. 😆