Daily notable summary - good idea?

[mhdawson]

I was talking with @joesepi about how it's hard to keep up with everything going on in the project. People are still being surprised by some relatively important PRs late in the process and sometimes even after they have landed, even though they follow all of our guidelines for waiting time, approvals etc.

Would it make any sense to try and tag PR/issues as notable and send out a daily summary? I suspect we could build a GitHub action to run a query and send out an email. I'm thinking an email as I don't think GitHub actions can be configured to call out something specific like that in a way that would be visible (at least I have no idea how to do that).

The existing notable-change label would not be the right thing as it may be added later on if something should be called out in the changelog (based on the description for it) versus a tag added up front.

The main ask on collaborators would be to add the notable label to appropriate PRs that they open or that they review.

[Trott]

If we do this, please don't use notable as it is guaranteed that people will (understandably) not know the difference between notable and notable-change. Let's set people up for success.

Who would get the email? TSC? (Then maybe tsc-alert?) People who subscribe? (email-digest maybe?)

I think a daily email is going to be something that people will ignore but I'll leave that up to the people that would actually sign up for such a thing.

Before we do all this, though, what are some examples of the problem we're trying to solve? What are the important PRs you are referring to and who exactly was surprised by them?

[ljharb]

I was very surprised by the test runner PR. It was only open a week, and clearly the maintainers of the well-known test runners in the ecosystem weren’t all asked for feedback (given that i'm one of them) - before the PR nor during it (some were, ofc).

For a notable PR to get attention, it has to sit open for months, not days.

[gireeshpunathil]

definitely a great idea. I have signed up for multiple repos and many times I miss out important ones because sometimes I cannot open every notification mails and examine each PRs, so I opt-in based on the title that comes as the subject line of the mail. Often, this approach leads to failure - partly due to my poor judgement and partly the PR title does not reflect the content enough.

the challenge may be how to qualify notable PRs consistently. Some may be super-interesting for one and less for another?

[RaisinTen]

People are still being surprised by some relatively important PRs late in the process and sometimes even after they have landed, even though they follow all of our guidelines for waiting time, approvals etc.

Is there anything wrong with using the notable-changes label for both of these sorts of PRs:

• the ones that need to be highlighted on the changelog
• the important ones referred to in this issue

The existing notable-change label would not be the right thing as it may be added later on if something should be called out in the changelog (based on the description for it) versus a tag added up front.

We could change the description to fit both cases.

Also, if there are PRs that are important but they don't deserve to be highlighted on the changelog, we can introduce a separate label for them, perhaps named "changelog-highlights"?

It was only open a week, and clearly the maintainers of the well-known test runners in the ecosystem weren’t all asked for feedback (given that i'm one of them) - before the PR nor during it (some were, ofc).

Collaborators do have the ability to block (clicking on "request changes") a certain PR if they want someone else to take a look before it lands.

[ljharb]

@RaisinTen yes, but I’m not a collaborator, and either way, ecosystem experts probably shouldn’t have to be a collaborator to be consulted prior to landing something in their domain.

[RaisinTen]

@ljharb in that case, I think you can post a comment asking a collaborator to block the PR with the reasoning behind it?

[ljharb]

First, that’s pretty aggressive to ask to block a PR solely because i want time to review it, and second, the PR landed before i was aware if it.

Dramatically slowing down velocity is the only way to increase the chances of notification and feedback. You can choose to prefer velocity, but the tradeoff is worsening those, and vice versa.

[RaisinTen]

that’s pretty aggressive to ask to block a PR solely because i want time to review it

Isn't that the main ask here - that we want to make sure that a PR, that people think to be important, doesn't land until some folks get time to review it?

[ljharb]

Yes, but I’m also suggesting that talking to domain experts should happen before a single line of code is written - because the api design might change drastically given the right information.

[aduh95]

The conversation is getting a bit off rails, discussion regarding changing the minimal wait time, or the review threshold for a PR to land should go on its own thread.

---

I'm not convinced that daily notable summary suggestion would solve the original issue (some PRs would still slip through that notable filter, and we would be in the same situation as today), and would give us more triaging work. I'm not opposed to it if other folks are supportive, count me as -0.

[bnb]

heh, funnily enough I decided to start working on something literally exactly like this recently.

I used to go through the repository every week on Monday and read through all the activity and put them into a blog post, but haven't had the ability to do that as often recently. I worked a bit ago on encites for my team at Microsoft to report what we're doing in Open Source, and figured the concepts wouldn't be too hard to transfer over to filtering data from an org.

It turns out there's a lot of opacity in what GitHub counts as "updated" and it's a bit harder to filter just for that. I paired with a friend at GitHub and built out a GraphQL query that gets most public "activity" in the org, just need to convert it to Markdown (should be pretty easy) and I can share. My plan was to share this code and propose it run weekly and auto-generate a blog post that can go on nodejs.org, but I happened across this issue and figured I could share that I was working on this now. If folks do want to do a label, that's totally fine too - I can set issues/PRs/discussions with those labels apart in the markdown for easier visibility, too.

[mhdawson]

I can set issues/PRs/discussions with those labels apart in the markdown for easier visibility, too.
+1 as I think that might make the list more manageable.

[mmarchini]

I don't know how they do it, but LLVM (a much bigger project) has a weekly curated digest: https://llvmweekly.org/, which is handy if you want to follow developments in the project but can't keep an eye on everything that is happening. Something similar in design (where users can subscribe to instead of just being available to TSC or collaborators) would be ideal IMO.

[tniessen]

Years ago, I was thinking of an RSS feed or something similar that would highlight all semver-minor (and semver-major) changes. Of course, RSS feeds aren't exactly the most popular technology and semver-minor and semver-major PRs are not the only relevant activity.

There have been numerous cases in which I wished I'd had time to weigh in on a PR but either wasn't aware of it or didn't have time before it landed. Some PRs that make significant changes have been rushed, and others that make small but important technical changes are neglected for days or weeks.