Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update output data validations for frames and ctf #1528

Open
Tracked by #1508
manasaV3 opened this issue Jan 24, 2025 · 0 comments
Open
Tracked by #1508

Update output data validations for frames and ctf #1528

manasaV3 opened this issue Jan 24, 2025 · 0 comments
Assignees
Labels
backend P1 Should have

Comments

@manasaV3
Copy link
Contributor

manasaV3 commented Jan 24, 2025

Relates to: #1418

Rationale

This feature expects the source data to have the required related entities before it can be ingested to surface the frames and ctf metadata. To ensure that the source data is in a valid state, we need to add a validation step that can determine if the data is accurate before the ingestion.

Tasks

Validations for MDOC/Frames

  1. number of mdoc sections == number of frame files on s3 == number of items in frames table/metadata
  2. frame file names match file names in mdoc section
  3. tilt axis angle in mdoc file matches that in tilt series metadata (+/- 10 deg)
  4. tilt axis angle in mdoc file matches that in the alignment metadata [PSAP.rotation_angle] (+/- 10 deg)
  5. number of frames >= # of per section parameters

Validations for Frames Metadata

  1. When isGainCorrected == False, a Gains entity exists for the run
  2. max(acquisitionOrder) < number of frames - 1
  3. Sum of exposureDose of all frames associated with a tilt series == totalFlux of tilt series
  4. Sorting acquisitionOrder low-to-high and accumulatedDose low-to-high results in the same order

Validations for PerSectionParameter Metadata

  1. -180 <= astigmaticAngle <= 180
  2. majorDefocus > minorDefocus
  3. 0 <= phaseShift <= 2*pi
  4. maxResolution > 0
  5. rawAngle matches mdoc angle as defined above
  6. 0 <= zIndex <= (z-Dimension of tilt series - 1)

What Done Looks Like?

The existing validation tests include the new tests. We are able to run validation for any standardized data from command line or in the batch environment.

@manasaV3 manasaV3 added backend P0 Must have P1 Should have and removed P0 Must have P1 Should have labels Jan 24, 2025
@junxini junxini added this to the sprint 1 (phase 5) milestone Jan 28, 2025
@manasaV3 manasaV3 added P1 Should have and removed P0 Must have labels Feb 4, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backend P1 Should have
Projects
None yet
Development

No branches or pull requests

2 participants