Skip to content

feat(dwc): add schema extensions for DwC mapping and export#7937

Closed
foozleface wants to merge 1 commit intospecify:mainfrom
calacademy-research:cas/dwc-schema
Closed

feat(dwc): add schema extensions for DwC mapping and export#7937
foozleface wants to merge 1 commit intospecify:mainfrom
calacademy-research:cas/dwc-schema

Conversation

@foozleface
Copy link
Copy Markdown
Collaborator

Contributed by @foozleface

Adds the data model foundation for the DwC export pipeline: four new Django models in the export app and three new nullable columns on spqueryfield. This is the schema layer that the backend and frontend PRs build on top of.

Implementation

  • SchemaMapping — links a human-readable name, mapping type (Core/Extension), and vocabulary key to an SpQuery. Each mapping owns one query whose fields carry DwC term assignments.
  • ExportDataSet — represents a publishable DwC archive. Holds the archive filename, RSS toggle, EML metadata reference (via SpAppResource), and a foreign key to the core SchemaMapping.
  • ExportDataSetExtension — join table attaching additional extension mappings to an ExportDataSet, with a sort order for deterministic archive layout.
  • CacheTableMeta — tracks pre-built cache tables (table name, row count, build status, last-built timestamp) so the backend can skip re-execution of expensive queries.
  • Three new nullable columns on spqueryfield: term (DwC IRI, VARCHAR 500), isstatic (BOOLEAN), staticvalue (TEXT). These let a query field carry its DwC term assignment and optional static/override values. All nullable, so existing queries are completely unaffected.
  • pre_delete signal on SchemaMapping drops the associated cache table before CASCADE removes CacheTableMeta.
  • Migration 0047 (specify app) adds the three spqueryfield columns; depends on 0046 (tree indexes).
  • Migrations 0001 and 0002 (export app) create the four new tables; 0001 depends on 0047.

Note: Migration numbering assumes upstream PR #7878 (0045) and the tree indexes PR (0046) merge first.

This is part of the DwC export pipeline addressing issues #7709-#7748 (40 GitHub issues for Darwin Core Archive support).

Testing instructions

  • Run python manage.py migrate — verify migrations specify.0047, export.0001, and export.0002 apply cleanly
  • Confirm the four new tables exist: schemamapping, exportdataset, exportdatasetextension, cachetablemeta
  • Confirm spqueryfield has the new columns: Term, IsStatic, StaticValue
  • Verify existing queries still load and execute normally (all new columns are nullable with safe defaults)
  • Run python manage.py makemigrations --check — should report no new migrations needed

…umns

New models: SchemaMapping, ExportDataSet, ExportDataSetExtension, CacheTableMeta
New columns on spqueryfield: term, isstatic, staticvalue

These enable DwC term assignment on query fields and structured export
dataset configuration. The spqueryfield columns are nullable — existing
queries and fields are unaffected.

Note: Migration 0047 depends on 0046 (tree indexes). Export migrations
depend on 0047 (needs spquery FK target to exist).
@foozleface
Copy link
Copy Markdown
Collaborator Author

Superseded by the 7-PR atomic stack on issue-cas-* branches:

  1. issue-cas-7746: extensions join table + vocabulary on Schemamapping #8032 — issue-cas-7746: extensions table + vocabulary
  2. issue-cas-7737: CacheTableMeta model + cache table infrastructure #8033 — issue-cas-7737: CacheTableMeta + cache infrastructure
  3. issue-cas-7714: DwC schema terms vocabulary + permissions #8034 — issue-cas-7714: DwC schema terms vocabulary + permissions
  4. issue-cas-7709: Schema Mapper UI shell + Schema Config DwC section #8035 — issue-cas-7709: Schema Mapper UI shell + Schema Config DwC section
  5. issue-cas-7712: Clone endpoint + list APIs + Export Packages shell #8036 — issue-cas-7712: Clone endpoint + list APIs + Export Packages shell
  6. issue-cas-7710: Mapping UI features (NewMappingDialog, autoMap, toolbar) #8037 — issue-cas-7710: Mapping UI features (NewMappingDialog, autoMap, toolbar)
  7. issue-cas-7733: DwC export pipeline Phases 4-6 (DwCA, EML, GBIF, RSS) #8038 — issue-cas-7733: DwC export pipeline Phases 4-6 (DwCA, EML, GBIF, RSS)

The new stack:

@foozleface foozleface closed this Apr 27, 2026
@github-project-automation github-project-automation Bot moved this from 📋Back Log to ✅Done in General Tester Board Apr 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: ✅Done

Development

Successfully merging this pull request may close these issues.

1 participant