Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PB-1405: Split bod_migrate command #69

Merged
merged 8 commits into from
Feb 11, 2025
Merged

Conversation

benschs
Copy link
Contributor

@benschs benschs commented Jan 30, 2025

Model changes

Add field slug to providers and attributions. This field is non-null and unique. This allows it to be used in the url path to identify resources. It is not null and unique, which is why db migration is done in 3 steps to populate the column with default values.

Command changes

Add flags to management command bod_migrate to control which data should be migrated.
Added flags are --providers, --attributions, --datasets. They can also be combined to migrate them at the same time. Beware that --clear will still clear all the migrated data.

Currently there are 4 providers that can't be migrated:

name_de attribution reason
ASTRA, BPUK ch.astra Two entries with attribution ch.astra
SED / ETH Zürich sss Attribution does not contain .
swisstopo + ENSI NULL No Attribution
swisstopo + Erlebnis Geologie NULL No Attribution

Currently cannot migrate the following attribution as there is no provider ch.kanton.

name_de attribution reason
AV + Kantone ch.kanton.av Missing provider ch.kantone

The following datasets could not be migrated:

id_dataset reason
ch.kantone.cadastralwebmap-farbe Missing provider ch.kantone
ch.bafu.gefahren-historische_erdbeben Provider is sss
ch.swisstopo.geologie-dosisleistung-terrestrisch Missing provider for swisstopo + ENSI
ch.swisstopo.images-spotimage Missing foreign key to attribution/provider
ch.swisstopo.geologie-bodenunruhe Provider is sss
ch.astra.baulinien-nationalstrassen_aenderung_v2_0.oereb Missing provider ASTRA, BPUK
ch.astra.baulinien-nationalstrassen_v2_0.oereb Missing provider ASTRA, BPUK
ch.astra.projektierungszonen-nationalstrassen_v2_0.oereb Missing provider ASTRA, BPUK

Actions

  • Model changes documented in the business entity model
  • Resolve issues with data that could not be migrated (TODOs in code)

Add field external_id to provider that is populated by the attribution field in
BOD. Field can be used as external id e.g. in url paths. It is not null and
unique, which is why db migration is done in 3 steps to populate the column
with default values.
Add field external_id to attribution that is populated by the attribution
field in BOD. Field can be used as external id e.g. in url paths. It is not
null and unique, which is why db migration is done in 3 steps to populate the
column with default values.
Use slug for provider and attribution to be consistent with dataset.
Add slug in json api response of provider and attribution. Adapt test cases for
slug.
Add flags to bod_migration command to migrate providers, attributes and
datasets independently. Add slug and only migrate "top-level" providers.
Link multiple attributions to single provider based on slug.
@benschs benschs changed the title PB-1405: Add external id to provider PB-1405: Split bod_migrate command Feb 3, 2025
@benschs benschs requested review from msom, schtibe and boecklic and removed request for schtibe February 3, 2025 15:47
@benschs benschs marked this pull request as ready for review February 3, 2025 15:47
@msom
Copy link
Contributor

msom commented Feb 4, 2025

Nice! Looks good in general, some thoughts:

  • Would be nice to have some tests for this new functionality (bod_migrate with only some of the flags and maybe edge cases)
  • Should we use the slugs in the API instead of the ID's now that we have them?
  • Should we rename bod_migrate to bod_sync?

Add more test cases for bod_migrate command with only some of the flags set.
@benschs
Copy link
Contributor Author

benschs commented Feb 4, 2025

Nice! Looks good in general, some thoughts:

* Would be nice to have some tests for this new functionality (bod_migrate with only some of the flags and maybe edge cases)

* Should we use the slugs in the API instead of the ID's now that we have them?

* Should we rename `bod_migrate` to `bod_sync`?

Thanks for the input. I added more test cases. I also think the name bod_sync is better and renamed it.
I also think we should use the slug in the API, but would do this in a separate PR.

Data has been cleaned in bod DB, so no longer need special check.
@benschs
Copy link
Contributor Author

benschs commented Feb 10, 2025

As mentioned in https://jira.swisstopo.ch/browse/PB-1405, BOD database has been cleaned, now all providers, attributions and datasets can be synced as expected.

@benschs benschs merged commit 4ae866d into develop Feb 11, 2025
3 checks passed
@benschs benschs deleted the feat-PB-1405-sync-bod-cron branch February 11, 2025 08:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants