Get as much info from Loculus into ENA as possible

[theosanderson]

For now there are some unmapped fields that don't make it through (Theo will update this with more info).

[claude]

This issue may be related to #2983 (Improve assembly field mapping), which also tracks improving the field data that gets passed through to ENA during deposition. Please check if there is overlap.

[theosanderson-agent]

Unmapped Pathoplexus fields in ENA deposition

Based on the current field mappings in ena-submission/config/defaults.yaml and the Pathoplexus metadata schema in loculus_values/values.yaml, the following user-submittable fields are not currently mapped to ENA during deposition:

Sample collection details:

• geoLocSite, sampleType, presamplingActivity, collectionDevice, collectionMethod, cultureId

Specimen details:

• foodProduct, foodProductProperties, specimenProcessing, specimenProcessingDetails, experimentalSpecimenRoleType

Host details:

• hostOriginCountry, hostDisease, hostRole, hostAgeBin, hostTaxonId, isLabHost, hostVaccinationStatus

Exposure & travel:

• travelHistory, exposureSetting, exposureDetails

Infection history:

• previousInfectionDisease, previousInfectionOrganism

Sequencing details:

• sequencingProtocol, sequencingAssayType, sequencingDate, ampliconPcrPrimerScheme, ampliconSize, purposeOfSequencing

Sequencing contact:

• sequencedByContactName, sequencedByContactEmail

Bioinformatics:

• rawSequenceDataProcessingMethod, dehostingMethod, assemblyReferenceGenomeAccession, breadthOfCoverage

Quality control:

• qualityControlMethodName, qualityControlMethodVersion, qualityControlDetermination, qualityControlIssues, qualityControlDetails

Diagnostics:

• diagnosticMeasurementMethod, diagnosticTargetPresence, diagnosticTargetGeneName, diagnosticMeasurementValue, diagnosticMeasurementUnit

Culture/passage:

• cellLine, passageNumber, passageMethod

Other:

• comment

Note: The collector name ENA field is currently mapped to an empty list with a default of "not provided" — sequencedByContactName could potentially fill this. Also, sequencingProtocol is available but only sequencingInstrument is mapped (to PLATFORM in the assembly manifest). The config also has a comment noting #ena_checklist: ERC000033 - do not use until all fields are mapped to ENA accepted options.

Some of these fields (e.g. hostDisease, travelHistory, passageNumber/passageMethod) have direct counterparts in the ENA virus pathogen reporting checklist (ERC000033) and could be mapped relatively straightforwardly.

[theosanderson-agent]

Detail on unmapped fields with ENA ERC000033 equivalents

Cross-referencing the unmapped Pathoplexus fields against the ENA virus pathogen reporting checklist (ERC000033):

Fields with direct or near-direct ERC000033 equivalents

Unmapped Pathoplexus field	ERC000033 field	ENA status	Notes
sequencedByContactName	collector name	mandatory	Currently mapped to [] with default "not provided". This is a mandatory ENA field that we're defaulting away — mapping sequencedByContactName here would improve data quality.
geoLocSite	geographic location (region and locality)	recommended	Currently only geoLocAdmin1, geoLocAdmin2, geoLocCity are mapped. geoLocSite could be appended for more specific location info.
hostDisease	host_description	optional	ERC000033's host_description accepts "other descriptive information relating to the host" — hostDisease fits here.
hostRole	host_description	optional	Could also contribute to host_description.
isLabHost / cellLine	lab_host	optional	ERC000033 has lab_host for "scientific name of laboratory host used for propagation". isLabHost + cellLine could inform this.
exposureSetting	host_habitat	recommended	ERC000033's host_habitat covers "domestic farm types, live market, wild migratory/resident" — exposureSetting overlaps with this.
exposureDetails	subject_exposure	optional	Currently only exposureEvent maps to subject exposure. exposureDetails could be appended for richer data.
travelHistory	subject_exposure	optional	Travel-related exposure context could also contribute to the subject exposure field.
passageNumber / passageMethod	(no exact field)	—	Not in ERC000033, but commonly included as custom sample attributes in ENA viral submissions. Could be passed as free-text attributes.

Fields that could improve existing mappings

Current mapping	Improvement
PLATFORM (assembly manifest) ← sequencingInstrument	sequencingProtocol is also available and the docs list it as mapping to PLATFORM, but the actual config uses sequencingInstrument. Consider whether both should contribute, or if the mapping/docs need reconciling.
PROGRAM (assembly manifest) ← consensusSequenceSoftwareName + version	sequencingAssayType could add context here.
host subject id ← [] (default "not provided")	This is mandatory in ERC000033. No obvious Pathoplexus field maps here, but worth noting as a gap.

Unmapped fields with NO ERC000033 equivalent

These PHA4GE-origin fields don't have a standard ERC000033 counterpart but could be passed as custom ENA sample attributes if desired:

• Sample collection: sampleType, presamplingActivity, collectionDevice, collectionMethod
• Specimen: foodProduct, foodProductProperties, specimenProcessing, specimenProcessingDetails, experimentalSpecimenRoleType
• Host: hostOriginCountry, hostAgeBin, hostTaxonId, hostVaccinationStatus
• Infection history: previousInfectionDisease, previousInfectionOrganism
• Sequencing: sequencingDate, ampliconPcrPrimerScheme, ampliconSize, purposeOfSequencing
• Contact: sequencedByContactEmail
• Bioinformatics: rawSequenceDataProcessingMethod, dehostingMethod, assemblyReferenceGenomeAccession, breadthOfCoverage
• QC: all qualityControl* fields
• Diagnostics: all diagnostic* fields
• Other: comment

Summary of priorities

The biggest wins for ENA data quality would likely be:

1. sequencedByContactName → collector name (filling a mandatory field that's currently defaulted)
2. geoLocSite → geographic location (region and locality) (enriching an existing recommended mapping)
3. exposureSetting → host_habitat (filling a recommended field)
4. hostDisease / hostRole → host_description (adding useful optional context)
5. isLabHost / cellLine → lab_host (adding useful optional context)

Also worth noting: the config has #ena_checklist: ERC000033 - do not use until all fields are mapped to ENA accepted options — enabling the checklist would enforce ENA validation on these fields, which would catch format mismatches but requires ensuring all mapped values conform to ENA's controlled vocabularies.

[theosanderson]

Audit of unmapped fields & ENA checklist gaps

I went through the ena-submission mapping code (config/defaults.yaml, create_sample.py, create_assembly.py, ena_types.py) and cross-referenced it against ENA's virus pathogen checklist ERC000033. Findings below.

The core blocker

The checklist is deliberately disabled — config/defaults.yaml:30:

#ena_checklist: ERC000033 - do not use until all fields are mapped to ENA accepted options

So submissions validate against ENA's generic default, not the virus-pathogen standard, and nothing is checked against ENA controlled vocabularies. The issue is less "missing fields" than (a) a few recommended fields not mapped, and (b) several mapped fields sending free text where ENA requires a closed vocabulary — which is what's keeping the checklist off.

Mapped, but CV-noncompliant (would fail validation if we enabled ERC000033)

ENA field	Loculus source	Required vocabulary
host_health_state (mandatory)	hostHealthState	diseased / healthy / not provided / …
host_sex (mandatory)	hostGender	female / male / hermaphrodite / neuter / other / not provided
hospitalisation	regex on hostHealthState	ENA wants yes/no; we emit true/false
sample_capture_status (recommended)	purposeOfSampling	active surveillance in response to outbreak, market sample, …
host_disease_outcome (recommended)	hostHealthOutcome	dead / recovered / recovered with sequelae

Mapped, but always hardcoded to "not provided"

Two mandatory ERC000033 fields have empty loculus_fields and a "not provided" default — no real data flows:

• collector_name (defaults.yaml ~L91) — could use sequencedByContactName
• host_subject_id (~L100)

ERC000033 fields ENA recommends that we don't map at all

ENA field	Requirement	Candidate Loculus source
lab_host	optional	isLabHost
virus_identifier	recommended	cultureId / isolate id
host_behaviour	recommended	hostRole? (CV: captive-wild / domestic / wild / other)
host_habitat	recommended	— (CV: domestic:live market, wild:migratory, …)
serotype	recommended	— (no schema field)
definition_for_seropositive_sample	recommended	—
strain, illness_duration, sample_storage_conditions	optional	—

Larger categories never sent to ENA at all (~60 schema fields)

No ERC000033 slot exists for most of these, so they'd need the assembly manifest/EMBL flatfile or custom attributes rather than the sample checklist:

• Quality control: qualityControl*, totalSnps, totalFrameShifts, breadthOfCoverage, depthOfCoverage(✅ via COVERAGE), …
• Diagnostics: diagnosticTarget*, diagnosticMeasurement*
• Sequencing detail: sequencingProtocol, sequencingAssayType, purposeOfSequencing, passageNumber, passageMethod
• Other: lineage, travelHistory, hostVaccinationStatus, hostDisease, sampleType

Suggested path

1. Convert the ~5 CV-noncompliant fields to emit ENA-accepted values.
2. Wire up the empty-default mandatory fields (collector_name, host_subject_id).
3. Add the recommended fields with an existing source (isLabHost→lab_host, virus_identifier, host_behaviour).
4. Un-comment ena_checklist: ERC000033 and let it validate.

Related: #2983 (assembly field mapping).