Running validate on HPC

[jblalockusgs]

We've been experimenting with running validate on our USGS HPC (https://hpcportal.cr.usgs.gov/hpc-user-docs/supercomputers/hovenweep.html). This seems to have a lot of potential for handling some of our larger archives with many products/labels.

We've noticed some intermittent errors like error.label.unresolvable_resource that are happening because of timeouts. Is there some sort of rough upper limit to the number of simultaneous validations we can/should run? Or is this a situation where we should try and use local schemas and schematrons? I'm concerned that we might be overwhelming something if we are running several thousand instances of validate simultaneously. The HPC can handle that...but I'm not sure validate can.

[jordanpadams]

@jblalockusgs a few questions/comments:

• could you provide the output from the logs? that may help us understand what resource it is unable to access.
• to get even more insights, increasing the verbosity on the logs may help us dig deeper into exactly where it is failing
• what version of validate are you running? we just had a bug fix to fix some possible scalability issues that has yet to be released.

[jblalockusgs]

Here's an example report with --verbose 1 and --debug: validation_report_I92610043BSNU_11944001_7975.log

The only other DEBUG information was the various execution times that don't get routed to the report file:

DEBUG  [debug.execution.time]  1770854024262 :: THEMIS/PDS4/USA_NASA_PDS_ODTGEO_200XX/browse/odtbws2_0084/i926xxbws2/I92610043BSNU.xml :: gov.nasa.pds.tools.validate.rule.pds4.LabelValidationRule:validateLabel in 999480 ms

DEBUG  [debug.execution.time]  1770854024450 :: Validation complete (1 targets completed in 1002237 ms)

DEBUG  [debug.execution.time]  1770854024464 :: processMain() in 1002397 ms

Completed execution in 1002719 ms

In that particular case, it was having problems with the pds schema, but I've seen similar messages for geom, cart, and msn.

I've also had a few cases where validate seems just gets stuck and I never see any DEBUG info because the HPC task hits the wall time limit and gets cancelled. This run had a 90 minute wall time per task.

Out of 10,000 validate tasks I ran, 5 had the error.label.unresolvable_resource and 10 timed out. If I just re-run those tasks, they complete normally without issues.

I'm running validate 4.0.8

[jordanpadams]

thanks @jblalockusgs.

@al-niessner any thoughts on above? I am thinking this is possibly an issue with trying to download the schemas too often, but I may be wrong. @jblalockusgs it may be work testing with local schemas/schematrons

[al-niessner]

@jordanpadams

I do not know about too often, but I do know that the PDS site does sputter occasionally and it is load related. More than too many accesses, since I was doing dozens not thousands of downloads, it seemed to be collisions. I never tracked down the actual cause or problem so the cause is all speculation. I can, with certainty, confirm the sputtering.

@jblalockusgs

Yes, this is the perfect situation for local schema. You may already know all of the schema you need. If you do not, then you will need to collect them from all your labels. You will need all of them because it is an override all or none situation. Make a catalog file like:
https://github.com/NASA-PDS/validate/blob/main/src/test/resources/github1358/catalog.xml

When you run validate, point it at the catalog file with -C /path/catalog.xml. Adjust to your OS. They should run faster in general because you will not be going out to the web as often.

[jblalockusgs]

Just to follow up:
Using the local schema catalog worked very well. We got to the point where we could run validate on ~1000 labels simultaneously without any problems.

We were just able to do full validation on 753,384 products in ~19 hours, so this suggestion was a huge help for us...once we got everything working smoothly.