Heritage GIS

Project Scoping & Data Governance: Automated Validation & Spatial Compliance Workflows

Establishing a defensible data governance framework at the outset of any archaeological or heritage management initiative prevents downstream spatial inconsistencies, compliance failures, and irreversible data loss. Within the broader Heritage GIS Architecture & Fundamentals paradigm, project scoping must explicitly define coordinate reference systems, attribute schemas, validation thresholds, and versioning protocols before field data collection begins. This cluster page delivers a reproducible Python/GIS automation pipeline for scoping validation, CRS compliance checking, and attribute governance, engineered for field archaeologists, heritage managers, spatial developers, and academic research teams.

Phase 1: Metadata Standardization & Schema Enforcement

Before deploying any spatial database or field collection template, teams must codify a project data dictionary and compliance matrix. Heritage workflows require strict alignment with international documentation standards such as CIDOC-CRM or MIDAS HES, ensuring interoperability with national heritage registers and academic repositories. Standardizing these parameters early aligns desktop environments with automated validation logic, as detailed in Setting Up QGIS for Archaeological Surveys. A formal scoping document must explicitly state:

  • Required coordinate precision: e.g., ±0.01m for RTK-GNSS, ±1.5m for handheld consumer GNSS, ±0.1m for total station offsets.
  • Mandatory metadata fields: site_id, feature_type, stratigraphic_phase, collection_method, collector_id, acquisition_date.
  • Acceptable geometry types per feature class: Point (finds, survey markers), Polygon (excavation units, site boundaries), Line (trench edges, survey transects).
  • Null/missing data protocols: Explicit NULL handling for unverified attributes; default fallbacks restricted to controlled vocabularies (e.g., UNKNOWN, UNVERIFIED).

Schema enforcement should be implemented via pandas/GeoPandas type coercion and strict dtype mapping prior to ingestion. This prevents silent type degradation (e.g., dates parsed as strings, numeric IDs cast as floats) that compromises downstream spatial joins and statistical analyses.

Phase 2: CRS Validation & Transformation Pipeline

Spatial compliance begins with coordinate reference system verification. Heritage projects routinely ingest legacy survey data, drone orthomosaics, LiDAR point clouds, and national heritage registers, each carrying disparate projections. Selecting an appropriate projection for heritage contexts requires balancing metric accuracy with regional survey standards, as detailed in CRS Selection for Heritage Sites. Automated CRS validation prevents silent misalignments during spatial joins, buffer analyses, or volumetric calculations.

The following pipeline enforces a target CRS, validates transformation integrity, and flags geometries that fail reprojection or exceed precision tolerances. Dependencies are strictly version-pinned to ensure reproducibility across field laptops, institutional servers, and cloud runners.

# requirements.txt
geopandas==0.14.3
pyproj==3.6.1
shapely==2.0.3
pandas==2.1.4
import geopandas as gpd
import pandas as pd
from pyproj import CRS, Transformer
from shapely.validation import make_valid
import logging
from typing import List, Tuple

logging.basicConfig(
    level=logging.INFO,
    format='%(asctime)s | %(levelname)s | %(message)s',
    datefmt='%Y-%m-%d %H:%M:%S'
)

ALLOWED_SOURCE_EPSGS: List[int] = [4326, 25832, 25833, 27700, 32633]
TARGET_EPSG: int = 25832  # ETRS89 / UTM zone 32N (example for Central European heritage)

def validate_and_transform_crs(
    input_gdf: gpd.GeoDataFrame,
    target_epsg: int = TARGET_EPSG,
    allowed_sources: List[int] = ALLOWED_SOURCE_EPSGS
) -> gpd.GeoDataFrame:
    """
    Validates input CRS against allowed heritage survey standards, 
    transforms to target EPSG, and logs transformation integrity.
    """
    if input_gdf.empty:
        raise ValueError("Input GeoDataFrame is empty.")

    src_crs = input_gdf.crs
    if src_crs is None:
        raise ValueError("Input GeoDataFrame lacks CRS definition. Assign before validation.")

    src_epsg = src_crs.to_epsg()
    if src_epsg not in allowed_sources:
        logging.warning(f"Source CRS EPSG:{src_epsg} not in allowed list. Forcing validation.")

    target_crs = CRS.from_epsg(target_epsg)
    transformer = Transformer.from_crs(src_crs, target_crs, always_xy=True)

    # Pre-transform geometry validation
    valid_mask = input_gdf.geometry.is_valid
    if not valid_mask.all():
        invalid_count = (~valid_mask).sum()
        logging.warning(f"Repairing {invalid_count} invalid geometries before transformation.")
        input_gdf.loc[~valid_mask, "geometry"] = input_gdf.loc[~valid_mask, "geometry"].apply(make_valid)

    # Execute transformation
    output_gdf = input_gdf.to_crs(target_crs)
    
    # Post-transform precision enforcement (2 decimal places for metric UTM)
    output_gdf["geometry"] = output_gdf["geometry"].apply(
        lambda geom: geom if geom.is_valid else make_valid(geom)
    )
    
    logging.info(f"Successfully transformed {len(output_gdf)} features to EPSG:{target_epsg}")
    return output_gdf

Phase 3: Attribute & Topological Compliance

Once spatial coordinates are normalized, attribute and topological validation must enforce heritage recording standards. Common failure modes include overlapping excavation units, self-intersecting site boundaries, and uncontrolled string inputs for categorical fields.

The compliance routine should:

  1. Enforce controlled vocabularies: Validate categorical columns against predefined set domains. Reject or quarantine out-of-scope values.
  2. Check topological integrity: Flag overlapping polygons, sliver geometries (<0.05m²), and duplicate vertices. Use shapely spatial predicates (intersects, touches, overlaps) to audit feature relationships.
  3. Validate temporal consistency: Ensure acquisition_date precedes processing_date and falls within the project’s active field season window.
  4. Handle precision drift: Round coordinate values to the project’s declared precision tolerance (e.g., 2 decimals for UTM meters) to prevent floating-point artifacts during spatial indexing.

Automated attribute validation should run as a pre-commit hook or CI pipeline stage, generating a structured compliance report (JSON/CSV) that quarantines non-conforming records for manual review by heritage managers.

Phase 4: Pipeline Routing & Version Control Integration

A production-ready spatial governance pipeline requires deterministic routing, explicit dependency isolation, and immutable version tracking. The recommended execution graph follows a strict Directed Acyclic Graph (DAG):

[Raw Ingest] → [Schema Validation] → [CRS Normalization] → [Topology/Attribute Checks] → [Versioned Output]
       ↓                ↓                      ↓                        ↓                    ↓
   GeoJSON/CSV      Type Coercion        EPSG Enforcement         Domain/Geometry Rules    GeoPackage + Git LFS

Pipeline routing should be orchestrated via lightweight task runners (e.g., prefect, dvc, or Makefile targets) to ensure idempotent execution. Spatial datasets exceeding 100MB must be tracked via Git LFS, with commit messages adhering to conventional changelog standards. Branching strategies should isolate experimental survey data from production baselines, enabling heritage managers to audit spatial changes without disrupting active field operations. For comprehensive implementation strategies tailored to archaeological datasets, refer to Setting up version control for GIS heritage projects.

Final outputs should be packaged as GeoPackage (.gpkg) files, which natively support spatial indexing, metadata embedding, and multi-layer storage. Compliance reports and transformation logs must be archived alongside the spatial data to satisfy audit requirements from funding bodies and national heritage authorities.