Test production gateways

[laurentsenta]

Contributes to #8

• pick one or more gateways (https://ipfs.github.io/public-gateway-checker/ - ipfs.runfission.com might be a good candidate)
• automatically pin our fixtures.car file (using web3.storage) on merge to main
• configure the test suite to run tests there (probably not the one related to dnslink and ipns)
• generate dashboards

[laurentsenta]

The follow-up task would be to create a dashboard when we run these tests:

Phase 1:

1. run the tests for every gateway we care about
2. aggregate test results by specs involved (gateway-car, gateway-trustless, etc).
   ideally we end up with a matrix of (gateway x specs x PASS/FAIL)
3. generate an HTML, similar to https://ipfs.github.io/public-gateway-checker/

Phase 2:

add support for dnslink and ipns record, these will require sync'ing with the gateway providers

[BigLep]

@laurentsenta : cool. I can see how this can be done for gateways with a public URL like the ones in https://github.com/ipfs/public-gateway-checker . How would we do this against various gateway implementations (Boxo, Kubo, bifrost-gateway)? Would we be needing to recreate their setups in the gateway-conformance pipeline so we can get an instance of them running?

[laurentsenta]

@BigLep

Would we be needing to recreate their setups in the gateway-conformance pipeline so we can get an instance of them running?

That would be a "worst case" scenario: since all the projects you listed have gateway conformance in their CI, we can definitely find a collaborative approach where we reuse their workflows or the artifacts they produced.

• https://github.com/ipfs/kubo/actions/workflows/gateway-conformance.yml
• https://github.com/ipfs/bifrost-gateway/actions/workflows/gateway-conformance.yml
• https://github.com/ipfs/boxo/actions/workflows/gateway-conformance.yml
• https://github.com/filecoin-saturn/L1-node/actions/workflows/gateway-conformance.yml

[laurentsenta]

@lidel @BigLep
fwiw I drafted a solution in #116

example dashboard output: https://github.com/singulargarden/gateway-conformance/actions/runs/5681118160/attempts/1#summary-15396709726

[laurentsenta]

Closing this, it's running. Follow-up / related tasks live in #120