Formalizing the fastverse?

[SebKrantz]

Dear All,

Given that Morgan Jacob from kit has just transferred the repo to this organization because he is concerned about his ability to maintain it, and given that my application with collapse as a NumFOCUS Affiliated Project (see details below) has been rejected on the basis of this being essentially a 'Solo project', but also given the increased popularity and value to the community, there is, I think, some impetus to think about greater formalism. Concretely, I am thinking of transferring collapse to this organization, and building a community of maintainers/people with write permissions that have at least in theory the capacity to make changes to these repos.

For a start, I was thinking of @2005m and @davidbudzynski (contributors to kit), and @NicChr (built two packages on top of collapse), @alinacherkas (contributions to issue handling and code of conduct) as well as @aitap, @eddelbuettel, and perhaps @MichaelChirico as experienced programmers though, I understand, largely with other commitments. Happy to consider others.

Overall, I do not in the least plan to abandon my curent role as the main active maintainer of these projects. I am just thinking about a few steps to maximize the benefits and visibility of these projects to the community. Creating an more formal looking organization and adding a bunch of maintainers seems to be a minimum requirement for something like NumFOCUS affiliation, and could be a basis to apply for NumFOCUS sponsorship (which I currently don't see need for, but it may arise in the future).

In any case, feedback is welcome before I do anything or invite anyone else to the org.

Best,

Sebastian

---

Numfocus Application as an Affiliated Project – Rejection Letter from October 8th

We are sorry to tell you that collapse was not accepted as an affiliated project. The NumFOCUS Affiliated Project Selection Committee felt that collapse was not yet ready for affiliation.

Here is some feedback from the committee:

"The documentation looks really professional, and I think you express what it does well. I do not think the time is right to become an affiliate project, you need more developers as we do not accept single person projects. Also, you need to add a code of conduct page and contact point. Preferably an institutional address created specifically for that purpose"

You're welcome to re-apply later once you've addressed the feedback provided.

---

My Numfocus Application as an Affiliated Project From July 3rd

Does your project have a contributor Code of Conduct?
Yes

What is the name of your project?
collapse

Please provide the url of your project's (primary) repo:
https://github.com/SebKrantz/collapse

Your project's website:
https://sebkrantz.github.io/collapse/

Please provide a summary description of your project in a few sentences:
collapse is a large C/C++-based R package for advanced data transformation and statistical computing in R. It aims to: (1) Facilitate complex data transformation, exploration and computing tasks in R (including vectorized grouped and weighted statistical operations on matrix-like objects, advanced time series and panel data operations, recursive tools, and highly performant data exploration and manipulation – supporting missing values, variable labels, and complex scientific data). (2) Help make R code fast, flexible, parsimonious and programmer friendly (its novel class-agnostic architecture supports all basic R objects and their popular extensions, including units, integer64, xts/zoo, tibble, grouped_df, data.table, sf, pseries and pdata.frame). Please see the article (https://arxiv.org/abs/2403.05038), forthcoming in Journal of Statistical Software, for an overview and discussion of how collapse adds to the scientific computing ecosystem.

Does your project have a logo?
Yes

Please upload a .svg file of your project's logo.
collapse_logo.svg

Your project's Twitter handle or other social media handles/urls:
https://x.com/collapse_R, https://bsky.app/profile/rcollapse.bsky.social

Why do you want your project to join NumFOCUS?
I would like it to have greater visibility as I believe it adds advanced and unique statistical capabilities and class-agnostic flexibility to the scientific computing ecosystem. It is also firmly among the fastest statistical packages in the world. I expect the publication of the article later this year to promote its usage. I am also thinking about expanding the number of maintainers and creating a community. Toward this end, I have founded the fastverse organization (https://github.com/fastverse) and meta-package to promote high-quality high-performance statistical software for R. There is a growing number of such packages in R, of which collapse is the largest and most comprehensive and mature effort. I plan to soon shift the collapse repo to the fastverse organization, add some maintainers to it (several people have shown significant interest and willingness to contribute). I expect numFOCUS sponsorship to increase its visibility in the scientific community, further increase the stack of contributors.

Are you applying for Fiscal Sponsorship or Affiliation?
affiliation

How does your project relate to or integrate with the existing ecosystem of NumFOCUS tools?
It adds advanced statistical and data manipulation capabilities to R in a way that seamlessly integrates with the language, is flexible, parsimonious, and computationally very powerful. In this sense it is complimentary to tools like numpy, pandas, dplyr and data.table, or rather, it blurrs the boundaries between these different tools while adding advanced statistical capabilities not found elsewhere. As an academic/applied economist, I saw great need for a toolset that facilitates more complex statistical workflows and provides greater flexibility than existing matrix or data frame oriented libraries. For a more detailed overview of collapse's relation to existing tools please consult the introduction section of the article (https://arxiv.org/abs/2403.05038).

Describe how your project furthers the NumFOCUS mission:
There are many commercial packages heavily used by researchers/economists/statisticians such as STATA, SPSS, MATLAB, and SAS. Key reasons are that they (1) make it rather easy to perform advanced statistical operations, such as those involving weights or advances time series or panel data; and (2) make it easier to deal with complex scientific data involving missing values and/or variable labels. collapse provides complex (grouped, weighted, indexed) statistical operations in R, with much greater flexibility and performance, and including operations (such as various weighted mode and quantile estimators) that no commercial package provides. It also seamlessly supports vector, matrix, and data frame-like objects in an attribute preserving manner, including special support for variable labels and factors (categorical variables) in R. It has been adopted by many professsionals, and is used to build production-grade research and data pipelines in organizations including the World Bank, Google, Amazon, etc. It thus contributes to your mission of promoting open practices in research, data, and scientific computing.

Does your project have any significant contributors who are not maintainers?
Not currently, but a number of people have expressed interest, and it is my objective to build a community.

What is your project doing to attract and/or mentor new contributors and maintainers?
I added some contributing guidelines. As mentioned, I am planning to shift it to the fastverse GitHub organization and add a few people that have been active filing issues, sending PRs, or building packages with collapse.

Where do you host conversations about project development and governance (e.g. mailing lists, forums, etc.), and how many participants do you have?
I recently started a discussion section: https://github.com/SebKrantz/collapse/discussions, also for the fastverse: https://github.com/fastverse/fastverse/discussions. 3 people have opened discussions so far.

What license(s) does your project currently use?
Most of it is GPL-2.0. Some source files have a more permissive MPL-2.0 license.

Projects must adopt the NumFOCUS Code of Conduct or one similar in spirit. Please tell us how you plan to meet this requirement:
I can add notes to the issue or contributing guidelines or make a separate code of conduct file. So far, I have had no issues with users/contributors.

[NicChr]

This looks great! Happy to be involved and contribute where I can😃

[alinacherkas]

Hi @SebKrantz, I think it's a great move and would help collapse gain more traction in the community. I can definitely support the project by managing issues, introducing additional community standards and, whenever I can, contributing code. So count me in!

Once you transfer the repo and add other maintainers, we should probably start by creating a roadmap to map out priorities.

Thanks!

[davidbudzynski]

Hi! Thanks for tagging me in. Happy to get involved in my free time!

[SebKrantz]

Ok, great! I've sent invites to join the organization to the three that responded so far. Note that by default your membership will be private but you can make it public. I'll have to wait a bit before the collapse transfer because I need to sort it out how to make the website forward to the new one – will probably require recreating a collapse repo on my account after the transfer and putting some forwarding code into GitHub pages.

One thing that can be done immediately – if someone wants to do it (I can also do it end of next week), is to create a fastverse-style website for kit. I.e., standard pkgdown setup, with this theme configuration and adding this CSS stylesheet in pkgdown/extra.css.

[aitap]

Thank you very much for the invitation! Always glad to help with issue diagnosis.

[eddelbuettel]

(Total aside: Very slick CSS for the pkgdown variant. That's a pretty elegant look, and step up from the defaults.)

Happy to help, but not really able to commit sizeable chunks of time.

(And before I go, a micro nag. The block that mentions Rcpp also mentions "that other C++ wrapper" and repeats its line about offering faster compile time. We added include entry point alternatives Rcpp/Light and Rcpp/Lighter and Rcpp/Lightest four or so years agos so if one cares one can also drive that down in Rcpp. Anyway... )

[SebKrantz]

Thanks Dirk!

And I wasn't aware of those headers. I tried

#include <Rcpp/Lighter>
using namespace Rcpp;

Instead of #include <Rcpp.h>. Seems a bit faster. Wondering: is using namespace Rcpp; still correct then?

[eddelbuettel]

Hi Seb,

As with many things of the usual complexity of a mid-sized package it is of course underdocumented :) I made the change four years ago (and have been using them in many example since) because the key criticism is valid: Rcpp Modules has cost for template resolution, not everybody / every build needs it. Next, Rcpp Sugar has cost, not everybody needs it and so. With that I arrived at four layers, with Rcpp/Rcpp as before as the other four 'faster'. RcppArmadillo has this now too.

What you do with using namespace Rcpp; is between you and the compiler, so orthogonal to this 'trick' of essentially showing fewer headers + capabilities to the compiler. Per Stroustrup's famous C++ motto: "you don't pay for what you don't use". So here cheaper / faster compilations. So keep it if you like, myself and many others recommend against. Namespace protection is good.

Changing / propagating existing APIs is hard. I just spent the last few minutes chasing which of nearly 200 packages another package depended upon was importFrom(Rcpp, xxxx) an xxxx we deprecated years ago and which I try to remove, Found it, and it is an unused (!!) import in that package's NAMESPACE. Alas....

[SebKrantz]

I now created a source package with <Rcpp/Lighter> and get a ~2 second compile time improvement, from 33.384 to 31.752 seconds. I noticed that the generated RcppExports.cpp still has #include <Rcpp.h>. Anyway, that's a worthwhile improvement, but most of the compile time is still due to Rcpp.

[SebKrantz]

kit website now up and running, with a vignette generated by Claude Opus 4.5.

[eddelbuettel]

Badge for CI is dead. You have https://github.com/fastverse/kit/workflows/R-CMD-check/badge.svg, the proposed snippet has https://github.com/fastverse/kit/actions/workflows/R-CMD-check.yaml/badge.svg with the full snippet I just got being

[![R-CMD-check.yaml](https://github.com/fastverse/kit/actions/workflows/R-CMD-check.yaml/badge.svg)](https://github.com/fastverse/kit/actions/workflows/R-CMD-check.yaml)

[SebKrantz]

Thanks, fixed!

[SebKrantz]

Also just one idea while we're at it: the parallel functions in kit are very fast and useful – significantly faster than pmin() or pmax(), and especially than do.call(pmax, df). So I was thinking of also implementing pmin() and pmax() again in kit, e.g., as fpmin() and fpmax(), so that one can do e.g. fpmax(df) or in general just get the kit-level of performance when needed. However, that would break with the naming convention. Another idea I thus had was create prange(..., na.rm = TRUE, how = c("range", "min", "max")) adding useful range functionality and including pmin() and pmax() using how argument. Anyway, thoughts are welcome, and of course feel free to implement (e.g., @NicChr or @davidbudzynski).

[NicChr]

You could potentially rename them with an underscore, e.g. p_min()?

I'll try look into it when I can! Probably just before or after the Christmas break.

[NicChr]

One advantage of using the p_ format is that Rstudio should autosuggest the kit functions when you start typing p_

[SebKrantz]

Thanks Nick, I agree, in principle a good idea with p_. And I think building it into a prange() function is not a good idea. I note, though, that the package so far has not used '_' — fpmin() or pMin() would be more consistent with the naming conventions applied. Anyway, would be good to hear some further opinions.

And thanks if you want to implement! I think this one can probably be done easily using any AI editor.

[SebKrantz]

So collapse has just moved here and the redirects are also working. Everybody should now be able to create branches and submit PRs. I've restricted the master and development branch (which is my development branch), so PR's need to be sent there (ideally to development).

Next steps according to NumFOCUS points would be "you need to add a code of conduct page and contact point. Preferably an institutional address created specifically for that purpose."

I can create the email address. I was wondering if potentially @alinacherkas may be good at code of conduct (there is also some info here).

[SebKrantz]

Also, would be great if a few more could make their organization membership public for NumFOCUS resubmission.

[alinacherkas]

Hi @SebKrantz, sure, I will add it over the weekend.

On a related note, perhaps it's a good idea to buy a fastverse.org domain (there is a reasonable deal on Cloudflare). The domain can be used for GitHub Pages and/or email. What do you think?

[SebKrantz]

Thanks!

[eddelbuettel]

For what it is worth I used to host a handful of domains at gandi.net (but they turned bad/evil after a sale) and now use porkbun.com.
I can help if you need it. But at the end of the day fastverse.github.io already works and is free...

[SebKrantz]

Thanks @eddelbuettel and @alinacherkas! In general, I like the idea of a domain – ultimate proof that you are serious about something (e.g. for NumFOCUS resubmission). It of course throws up questions about long-term financing, but I have a GitHub sponsor now which can cover it so it doesn't strictly need to be a personal expense. In the long-run it could also be a reason to apply for NumFOCUS fiscal sponsoring, but I guess for that we need much clearer development ambitions and guidelines so it is better to start with Affiliation.

[SebKrantz]

We are live at fastverse.org. Just got a 10-year registration for $87 at namecheap. Paid for by @yihui. Many thanks!

[yihui]

You are welcome! Happy to cover this expense for you indefinitely!

[SebKrantz]

It's perfect timing BTW, tomorrow I'm going to Porto in Portugal to give an invited talk about collapse and the fastverse: https://www.bportugal.pt/en/evento/workshop-speeding-empirical-research-tools-and-techniques-fast-computing-bplim. Also, if @tony-aw wants to join the org, happy to send an invite.