Batch Processing Photogrammetry Datasets #

Scaling photogrammetric reconstruction across multiple excavation units, structural ruins, or landscape transects requires deterministic, audit-ready automation. This workflow addresses the batch processing stage within the broader Photogrammetry & 3D Site Mapping Pipelines framework, providing field-ready validation, coordinate system integrity checks, and reproducible Python automation for archaeologists, heritage managers, and academic research teams.

Pipeline Architecture & Routing #

The automation routine follows a strictly sequential, idempotent DAG (Directed Acyclic Graph) to ensure reproducibility across multi-season campaigns. Each processing node is isolated with explicit rollback capabilities and structured logging to prevent cascading failures when orchestrating hundreds of datasets. Field teams deploying Automated Drone Image Processing Workflows must standardize directory layouts, camera calibration profiles, and GCP/RTK metadata prior to execution.

The pipeline routes datasets through five deterministic stages:

1. Ingestion & EXIF Parsing: Validates image headers, extracts focal lengths, and maps GPS/IMU metadata to project coordinates.
2. Sparse Alignment: Detects tie points, estimates camera poses, and performs bundle adjustment with fixed interior orientation parameters.
3. Dense Point Cloud Generation: Executes multi-view stereo matching with GPU acceleration, applying depth filtering thresholds appropriate for stratigraphic surfaces.
4. Mesh Reconstruction: Builds watertight geometry from dense points, applying decimation and topology cleanup for archival storage.
5. Georeferenced Export: Projects assets to the target CRS, generates orthomosaics, and writes spatial database-ready metadata (GeoTIFF, OBJ, LAS).

Each stage writes a checkpoint manifest. If a node fails, the pipeline halts, quarantines the dataset, and logs the exact failure signature without corrupting downstream outputs.

Coordinate Reference System (CRS) Validation & Transformation #

Spatial integrity is non-negotiable in heritage recording. Before batch alignment, the automation routine must validate that all input imagery shares a consistent coordinate reference system or contains valid RTK/GPS metadata for automatic projection assignment. A pre-processing validation routine should:

1. Parse EXIF GPSLatitude, GPSLongitude, GPSAltitude, and camera orientation tags.
2. Cross-reference against project EPSG definitions (e.g., EPSG:27700 for UK Ordnance Survey National Grid, EPSG:32633 for WGS 84 / UTM zone 33N, or EPSG:4326 for raw WGS 84 lat/lon).
3. Flag datasets with mixed datums, missing altitude values, or inconsistent focal length metadata.
4. Apply a deterministic transformation matrix via pyproj if local archaeological grid coordinates are required for stratigraphic mapping.

Validation rules must enforce a maximum horizontal/vertical residual threshold (typically ≤0.05 m for RTK, ≤0.15 m for PPK) before proceeding to dense reconstruction. Any dataset exceeding these tolerances triggers a quarantine state for manual review.

Reproducible Python Implementation #

The following implementation uses the Agisoft Metashape Python API to orchestrate batch processing. It includes explicit try/except blocks, structured logging, and CRS validation hooks. Metashape’s Python module (Metashape) is provided by Agisoft as part of the licensed Metashape installation; it is not available on PyPI. Refer to python-scripts-for-batch-processing-agisoft-metashape-projects for detailed configuration.

Version-Pinned Python Dependencies (requirements.txt)

# pip-installable packages
pyproj==3.6.1
exifread==3.0.0
numpy==1.26.4
# Requires Python >= 3.10, < 3.12
# Metashape Python API is NOT on PyPI — install from Agisoft's licensed wheel:
# https://www.agisoft.com/downloads/installer/

Core Batch Orchestrator

import os
import logging
import pyproj
import Metashape  # Requires Agisoft Metashape installation

# Configure structured logging
logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s | %(levelname)s | %(message)s",
    handlers=[logging.FileHandler("batch_processing.log"), logging.StreamHandler()]
)

def validate_crs_and_geotags(image_path: str, target_epsg: int) -> bool:
    """Validate EXIF GPS tags and check coordinate transformability for target CRS."""
    try:
        doc = Metashape.Document()
        chunk = doc.addChunk()
        chunk.addPhotos([image_path])
        chunk.matchPhotos(match_generic=False)
        chunk.alignCameras()

        cam = chunk.cameras[0]
        if not cam.reference.location:
            logging.warning(f"Missing GPS metadata: {image_path}")
            return False

        lat, lon, alt = cam.reference.location
        # Transform to target CRS to confirm coordinate validity
        transformer = pyproj.Transformer.from_crs(
            "EPSG:4326", f"EPSG:{target_epsg}", always_xy=True
        )
        x, y = transformer.transform(lon, lat)
        logging.info(f"CRS validation passed for {image_path} -> EPSG:{target_epsg} ({x:.2f}, {y:.2f})")
        return True
    except Exception as e:
        logging.error(f"CRS/EXIF validation failed: {e}")
        return False

def process_dataset_batch(image_dir: str, output_dir: str, target_epsg: int) -> None:
    doc = Metashape.Document()
    chunk = doc.addChunk()
    # Correct Metashape CRS syntax uses single colon: "EPSG:<code>"
    chunk.crs = Metashape.CoordinateSystem(f"EPSG:{target_epsg}")

    for img in sorted(os.listdir(image_dir)):
        if not img.lower().endswith(('.tif', '.jpg', '.png')):
            continue

        img_path = os.path.join(image_dir, img)
        if not validate_crs_and_geotags(img_path, target_epsg):
            logging.warning(f"Quarantining dataset due to CRS/GPS failure: {img}")
            quarantine_dir = os.path.join(output_dir, "quarantine")
            os.makedirs(quarantine_dir, exist_ok=True)
            os.rename(img_path, os.path.join(quarantine_dir, img))
            continue

        chunk.addPhotos([img_path])

    try:
        chunk.matchPhotos()
        chunk.alignCameras()
        chunk.buildDepthMaps()
        chunk.buildPointCloud()
        chunk.buildModel()
        chunk.exportModel(
            os.path.join(output_dir, "reconstructed_mesh.obj"),
            format=Metashape.ModelFormatOBJ
        )
        logging.info("Batch processing completed successfully.")
    except Exception as e:
        logging.critical(f"Pipeline failure during reconstruction: {e}")
        raise

Execution, Monitoring & Spatial Database Ingestion #

Headless execution should be deployed on a Linux CI runner or high-performance workstation with dedicated CUDA-capable GPUs. Upon successful completion, the pipeline generates SHA-256 checksums for all exported assets, ensuring chain-of-custody integrity for heritage archives:

sha256sum output/*.obj output/*.tif > output/checksums.sha256

Processed meshes and point clouds should undergo topology validation and LOD generation before ingestion into spatial databases or web viewers. For complex ruin geometries requiring retopology or artifact removal, teams should route outputs to Mesh Generation & Optimization for Ruins for downstream refinement. Automated reporting should aggregate alignment residuals, dense cloud point counts, and CRS transformation matrices into a machine-readable JSON manifest for academic publication and long-term repository compliance.