[Website] Website deployment workflow (deploy.yml) is failing due to Node.js 18 version bump in ubuntu-latest GitHub Actions runner image and Webpack usage of md4 hashing algorithm

[kevingurney]

Describe the bug, including details regarding any error messages, version, and platform.

See the following comment on apache/arrow#322.

A few weeks ago, the apache/arrow-site deployment workflow (.github/workflows/deploy.yml) started failing with the following output:

.
.
.
npm ci
npm WARN deprecated [email protected]: You can find the new Popper v2 at @popperjs/core, this package is dedicated to the legacy v1

added [13](https://github.com/apache/arrow-site/actions/runs/4257427943/jobs/7407542742#step:9:14)2 packages, and audited 133 packages in 2s

17 packages are looking for funding
  run `npm fund` for details

found 0 vulnerabilities
npx webpack --mode=production
rm -f javascript/main.js
node:internal/crypto/hash:71
  this[kHandle] = new _Hash(algorithm, xofLen);
                  ^

Error: error:0308010C:digital envelope routines::unsupported
    at new Hash (node:internal/crypto/hash:71:19)
    at Object.createHash (node:crypto:133:10)
    at BulkUpdateDecorator.hashFactory (/home/runner/work/arrow-site/arrow-site/node_modules/webpack/lib/util/createHash.js:[14](https://github.com/apache/arrow-site/actions/runs/4257427943/jobs/7407542742#step:9:15)4:18)
    at BulkUpdateDecorator.update (/home/runner/work/arrow-site/arrow-site/node_modules/webpack/lib/util/createHash.js:46:50)
    at RawSource.updateHash (/home/runner/work/arrow-site/arrow-site/node_modules/webpack-sources/lib/RawSource.js:64:8)
    at NormalModule._initBuildHash (/home/runner/work/arrow-site/arrow-site/node_modules/webpack/lib/NormalModule.js:753:[17](https://github.com/apache/arrow-site/actions/runs/4257427943/jobs/7407542742#step:9:18))
    at handleParseResult (/home/runner/work/arrow-site/arrow-site/node_modules/webpack/lib/NormalModule.js:817:10)
    at /home/runner/work/arrow-site/arrow-site/node_modules/webpack/lib/NormalModule.js:908:4
    at processResult (/home/runner/work/arrow-site/arrow-site/node_modules/webpack/lib/NormalModule.js:640:11)
    at /home/runner/work/arrow-site/arrow-site/node_modules/webpack/lib/NormalModule.js:692:5 {
  opensslErrorStack: [ 'error:03000086:digital envelope routines::initialization error' ],
  library: 'digital envelope routines',
  reason: 'unsupported',
  code: 'ERR_OSSL_EVP_UNSUPPORTED'
}

Node.js v18.14.1
rake aborted!
Command failed with status (1): [npx webpack --mode=production...]
/home/runner/work/arrow-site/arrow-site/Rakefile:37:in `block in <top (required)>'
/home/runner/work/arrow-site/arrow-site/vendor/bundle/ruby/3.0.0/gems/rake-13.0.6/exe/rake:27:in `<top (required)>'
/opt/hostedtoolcache/Ruby/3.0.2/x64/bin/bundle:[23](https://github.com/apache/arrow-site/actions/runs/4257427943/jobs/7407542742#step:9:24):in `load'
/opt/hostedtoolcache/Ruby/3.0.2/x64/bin/bundle:23:in `<main>'
Tasks: TOP => generate => javascript/main.js
(See full trace by running task with --trace)
Error: Process completed with exit code 1.

This appears to be related to the use of Webpack by apache/arrow-site and the following issues:

1. nodejs 17: digital envelope routines::unsupported webpack/webpack#14532 (comment)
2. Webpack Hash is not FIPS-Compliant webpack/webpack#13572
3. allow to configure all hash functions used webpack/webpack#14306

My high level understanding is that in Node 18 (the build output above shows Node.js v18.14.1 is being used), the md4 hashing algorithm is deprecated (more specifically, it seems that Node 18 uses OpenSSL 3.0, which has deprecated md4) and the version of Webpack used by apache/arrow-site (v5.21.2) seems to default to using md4.

Webpack v5.61.0 added a WASM md4 implementation as a fallback. However, the advice in webpack/webpack#14532 (comment) recommends setting output.hasFunction in the Webpack config to use an alternative hashing algorithm instead. Specifically, it recommends using xxhash64 (which is planned to be the default hashing algorithm when Webpack 6 is released).

It seems that the version of Node.js in the ubuntu-latest GitHub Actions runner image (used by deploy.yml) was bumped to v18 on Februrary 13, 2023. This would explain why this issue started appearing a few weeks ago.

Workarounds

There are a few different approaches we could pursue to address this issue:

1. We could choose to pin the version of Node.js used by the GitHub Actions runner to v16 for the actions/setup-node action to work around this issue. Of course, this would mean we would be continuing to rely on an outdated version of Node.js, which doesn't seem ideal in the long term.

2. We could follow the advice in nodejs 17: digital envelope routines::unsupported webpack/webpack#14532 (comment) and set output.hashFunction in the Webpack config to use an alternative hashing algorithm, like xxhash64.

3. We could follow the advice of @avantgardnerio in DataFusion Substrait blog post #322 (comment) and move away from relying on the proprietary ubuntu-latest image, which is subject to sudden updates like the Node.js one that caused this issue. Instead, we can use the official ubuntu:latest container image (this is the approach followed by arrow-ballista). ubuntu:latest wouldn't have unexpected library updates, and it would also be possible to run the container image locally for debugging purposes.

Component(s)

Website

[avantgardnerio]

FWIW, I'm in favor of both 2 & 3: we should control our upgrades and not have them unexpectedly change, but we should also follow a reasonable upgrade cycle and not use outdated cryptographic functions.

That being said, the arrow project in general lacks a surplus of front-end development skills, so it option 3 makes the most sense to me if we only have the resources for 1 of the above.

[kevingurney]

@avantgardnerio - thanks for sharing your thoughts.

I agree - 2 + 3 seems like the optimal solution. As you mentioned, if we are concerned with resourcing, starting with option 3 feels like a good first step which would unblock things in the short term. We could always choose to address 2 in a separate PR when resourcing permits.

[kevingurney]

take

[kevingurney]

A quick status update:

I've been actively working on implementing workaround 3. (run the website deployment workflow inside of an ubuntu:latest container) in the mathworks/arrow-site fork.

However, getting the entire website deployment workflow to run successfully inside of an ubuntu:latest container has turned out to be more complicated than expected due to a variety of differences between the ubuntu-latest and ubuntu:latest environments.

I managed to get the workflow to succeed on the mathworks/arrow-site fork once. However, I had to change a significant portion of the workflow to get this working. Also, the use of apt-get update appears to be slowing the workflow down quite significantly (this might be resolvable through caching, but I have yet to get a chance to explore this).

You can see the history of attempts at getting the workflow running here:

https://github.com/mathworks/arrow-site/actions

Two more important issues I ran into while trying to get the container approach working:

1. I had to manually create a gh-pages branch before running the workflow.
2. I had to manually switch the deployment branch to gh-pages in the settings of the mathworks/arrow-site repository.

I suspect that 1. shouldn't be required to get the workflow running. It isn't clear to me at this point why this code was failing to create the gh-pages programmatically.

Also, it seems like README.md for arrow-site could be improved by adding a note that 2. is required when testing on a fork. I can follow up with a PR to address this.

Given the issues / tradeoffs here, it will take some more time to get this working effectively.

To unblock the website deployment for other PRs, we could consider pursuing workaround 1. or 2. described in this issue as short term solutions. I'm open to whatever approach the community prefers.

[kevingurney]

Update:

By reverting back a debugging change I accidentally left behind in the Rakefile (I had temporarily changed npm ci to npm install when I was debugging some npm related errors early on), the workflow passed successfully again. I can see that the changes were successfully deployed to https://mathworks.github.io/arrow-site/.

This appears to have also resolved the issue where the workflow was unable to create a gh-pages branch automatically. After deleting the gh-pages branch on mathworks/arrow-site and re-running the workflow, the workflow successfully created a gh-pages branch automatically. That being said, I still had to manually switch the deployment branch to gh-pages in the repository settings.

At this point, it seems like the main issue is reducing the time of the workflow (currently, it takes around 10 minutes) - possibly, through the use of caching.

[kevingurney]

Update:

I've managed to reduce the time of the workflow to around 5 minutes by reverting back to using the setup-node and setup-ruby actions, which have support for caching. I had previously removed these while debugging issues with the container workflow early on.

It would still be great to reduce the workflow time further if possible.

[kevingurney]

Update:

At this point, the only part of the workflow that is adding any significant additional time (around 1 minute) is the call to apt-get update and the installing of dependencies into the container.

[kevingurney]

Update: I've opened pull request #326 to address this issue.