Skip to main content

SWMM Feature Publisher Function

Azure Functions-based service that processes SWMM (Stormwater Management Model) INP files and publishes feature layers to ArcGIS Online.

Overview​

This Azure Function App automatically publishes SWMM projects to ArcGIS Online when INP files are uploaded to blob storage. The blob trigger detects uploads and processes them automatically.

Architecture​

End-to-End Flow​

Automatic Publishing Flow (Blob Trigger - Primary)​

  1. INP Upload: ASP.NET API uploads INP file to swmm-input/{projectId}/{filename}.inp
  2. Blob Trigger Activates: Function automatically detects upload
  3. Status Update: "Uploaded input file" → "Processing"
  4. Publish: Parse INP, generate GeoJSON, publish to ArcGIS with prefix SD1_{projectId}
  5. Complete: Update status to "Complete", store feature metadata, and merge map center onto the project row (see below)

Projects table: map center (WGS84)​

After a successful INP feature publish, the function computes a default view center from all vertices in the published GeoJSON (mean latitude / longitude in EPSG:4326) and upserts these optional columns on the same Projects row (PartitionKey = projects, RowKey = project id):

ColumnTypeDescription
MapCenterLatitudeEdm.DoubleMean latitude of network geometry
MapCenterLongitudeEdm.DoubleMean longitude of network geometry

Clients (e.g. SWMM web maps) can use this pair as { lat, lng } for the default 2D map home and 3D scene framing when a project is selected. If geometry yields no coordinates, these fields are not updated on that run (any previous values remain). SLPK-only publish does not recompute center; the value from the INP publish remains.

Hybrid GeoJSON publishing​

When USE_HYBRID_GEOJSON is enabled (default), Junction, Conduit, Outfall, Subcatchment, and Raingage layers are built using a hybrid INP + engine approach:

  • INP parser (publisher.pyswmm_feat.utils.inp_parser) supplies structure: nodes, coordinates, tags, inflows, treatment, conduits, outfalls.
  • Engine inspection (publisher.pyswmm_feat.utils.inspect_network) supplies hydraulic attributes from pyswmm (invert elevation, depths, offsets, losses, etc.).

The resulting GeoJSON uses the same field names and schema as the demo notebooks (pyswmm/examples/junction_geojson_full_demo.ipynb, tests/junction_geojson_full_demo_feat.ipynb) so that the API and frontend see stable property names (e.g. Name, Node_Type, Link_Type, Invert_Elev). Geometry is reprojected to WGS84 for ArcGIS Online.

  • Schema: publisher/geojson_schema.py maps conceptual keys to ArcGIS field names using an externally supplied ordered field map (loaded from Azure AttributeMap).
  • Builders: publisher/geojson_builders.py implements the hybrid builders, including:
    • build_subcatchment_polygon_features(...) -> Subcatchment (Polygon)
    • build_subcatchment_centroid_features(...) -> Subcatchment_Centroid (Point)
  • Subcatchment rule: Subcatchment is strict polygon geometry from [Polygons] in the INP (no fallback to centroid geometry).
  • Fallback: If engine initialization or inspection fails, publishing falls back to INP-only mode for that run and logs a warning.
  • Toggle: Set USE_HYBRID_GEOJSON=false (or 0/no) to use the legacy INP-only GeoJSON generator for all layers.

Developer note: The schema (publisher/geojson_schema.py) and builders (publisher/geojson_builders.py) are the canonical implementation shared with the pyswmm demo notebook. When adding or changing published fields for Junction/Conduit/Outfall, update the conceptual dicts in the builders and ensure the corresponding ordered field maps in Azure AttributeMap cover the new conceptual keys.

Components​

  • Blob Trigger: Monitors swmm-input container for INP uploads
  • Blob Storage: Stores INP input files in {projectId}/{filename}.inp format
  • Table Storage:
    • Projects table: Project processing status, EPSG, and optional MapCenterLatitude / MapCenterLongitude after successful feature publish
    • ProjectFeatureMetaData table: Published layer metadata
  • ArcGIS Online: Target platform for feature layers
  • HTTP Health Check: /api/health endpoint for monitoring

Required Azure Resources​

Storage Account​

  • Container: swmm-input (or configured name)
  • Tables: Projects, ProjectFeatureMetaData

Application Insights​

  • For monitoring, logging, and telemetry

Azure Function App​

  • Python 3.11+ runtime
  • Linux hosting (recommended)

Environment Variables​

VariableRequiredDefaultDescription
AzureWebJobsStorageYes-Connection string for Functions runtime
0b61f1_STORAGEYes-Connection string for blob trigger
AZURE_STORAGE_ACCOUNT_NAMEYes*-Storage account name (if using SAS)
AZURE_STORAGE_SAS_TOKENYes*-SAS token (if not using connection string)
AZURE_STORAGE_CONNECTION_STRINGYes*-Full connection string (alternative to SAS)
SWMM_INP_CONTAINERYes-Blob container name for INP files (e.g., swmm-input)
PROJECT_TABLE_NAMEYes-Table name for project status
FEATURES_TABLE_NAMENoProjectFeatureMetaDataTable name for feature metadata
ENABLE_INDEX_BUILDINGNotrueWhether to build NodeIdIndex and LinkIdIndex tables during publish
ARCGIS_URLNohttps://www.arcgis.comArcGIS Online URL
ARCGIS_API_KEYYes-ArcGIS API key for authentication
MAX_ATTEMPTSNo3Maximum retry attempts (1-10)
USE_HYBRID_GEOJSONNotrueUse hybrid INP + engine builders for Junction/Conduit/Outfall/Subcatchment/Raingage layers (see Hybrid GeoJSON publishing)

Notes:

  • *Either connection string OR (account name + SAS token) is required
  • 0b61f1_STORAGE is used for blob trigger bindings
  • For local development, use full connection string with AccountKey

Blob Trigger Workflow (Automatic Publishing)​

The blob trigger automatically publishes projects when INP files are uploaded:

Trigger Pattern​

  • Path: swmm-input/{projectId}/{filename}.inp
  • Any .inp file in a project folder triggers publishing
  • Example: swmm-input/rivershore/rivershore.inp → Auto-publishes

Status Lifecycle​

  1. "Uploaded input file" - Blob detected, acknowledged
  2. "Processing" - Publishing in progress
  3. "Complete" or "Failed" - Final state

Automatic Behavior​

  • projectId extracted from folder name
  • titlePrefix automatically set to SD1_{projectId}
  • Idempotency: Re-uploading won't reprocess if already "Complete"
  • Error handling: Automatic retries up to MAX_ATTEMPTS

Testing​

Quick test:

# Upload test INP file to trigger automatic publishing
python scripts/test_blob_upload.py --project-id testproject --inp-file scripts/rivershore.inp

Expected behavior:

  • Blob trigger activates automatically
  • Status transitions: "Uploaded input file" → "Processing" → "Complete"
  • Layers published to ArcGIS with prefix SD1_testproject

Verification:

  1. Check function logs for processing messages
  2. Query Projects table to verify status
  3. Check ArcGIS Online for published layers
  4. Verify ProjectFeatureMetaData table has entries

Testing idempotency: Re-uploading the same file should skip processing if status is already "Complete".

Testing error handling:

  • Upload empty INP file → Should fail with error message
  • Upload invalid projectId → Should fail validation
  • Test max attempts by setting MAX_ATTEMPTS=1 and uploading invalid file

Project Status Tracking​

The function maintains comprehensive status in the Projects table:

FieldTypeDescription
PartitionKeystringAlways "projects"
RowKeystringProject ID
ProjectStatusstring"Uploaded input file", "Processing", "Complete", "Failed"
AttemptCountintNumber of processing attempts
LastAttemptAtdatetimeTimestamp of last attempt
ProcessingStartedAtdatetimeFirst processing start time
ProcessingCompletedAtdatetimeCompletion timestamp
LastErrorMessagestringError details (if failed, max 2000 chars)
NeedsInterventionboolTrue if manual intervention required

Local Development​

Prerequisites​

  • Python 3.11+
  • Azure Functions Core Tools v4
  • Azure Storage Account (or Azurite)

Setup​

  1. Clone repository

    cd c:\Git\gqc\hydrotrek-ssc-suite\swmm-feature-publisher-function

  2. Create virtual environment

    python -m venv .venv
    .venv\Scripts\activate  # Windows
    source .venv/bin/activate  # Linux/Mac

  3. Install dependencies

    pip install -r requirements.txt

  4. Configure local settings

    Copy local.settings.json.example to local.settings.json and fill in values:

    {
      "IsEncrypted": false,
      "Values": {
        "AzureWebJobsStorage": "DefaultEndpointsProtocol=https;AccountName=...;AccountKey=...;EndpointSuffix=core.windows.net",
        "FUNCTIONS_WORKER_RUNTIME": "python",
        "0b61f1_STORAGE": "DefaultEndpointsProtocol=https;AccountName=...;AccountKey=...;EndpointSuffix=core.windows.net",
        "AZURE_STORAGE_ACCOUNT_NAME": "your-account",
        "AZURE_STORAGE_SAS_TOKEN": "?sv=2025-07-05&...",
        "SWMM_INP_CONTAINER": "swmm-input",
        "PROJECT_TABLE_NAME": "Projects",
        "FEATURES_TABLE_NAME": "ProjectFeatureMetaData",
        "ARCGIS_URL": "https://www.arcgis.com",
        "ARCGIS_API_KEY": "your-api-key",
        "MAX_ATTEMPTS": "3"
      }
    }

  5. Run locally

    func start

Deployment​

For detailed deployment instructions, see DEPLOYMENT.md (same folder as this file).

The function can be deployed to Azure using:

  • Azure Portal (recommended for first-time setup)
  • Azure CLI (automated script)
  • Managed Identity (production best practice)
  • CI/CD (GitHub Actions)

Quick deployment:

func azure functionapp publish <your-function-app-name>

Monitoring​

Application Insights​

Application Insights is automatically configured via the APPLICATIONINSIGHTS_CONNECTION_STRING environment variable.

View in Azure Portal:

  1. Navigate to your Function App
  2. Click "Application Insights" in left menu
  3. Or go directly to Application Insights resource

Health Check​

curl https://<function-app-name>.azurewebsites.net/api/health

Response:

{
  "status": "healthy",
  "service": "swmm-feature-publisher"
}

Key Metrics​

Success Rate​

requests
| where cloud_RoleName == "swmm-feature-publisher"
| where name == "swmmBlobTriggerPublisher"
| summarize
    Total = count(),
    Success = countif(success == true),
    Failed = countif(success == false),
    SuccessRate = 100.0 * countif(success == true) / count()
| project Total, Success, Failed, SuccessRate

Execution Duration​

requests
| where cloud_RoleName == "swmm-feature-publisher"
| where name == "swmmBlobTriggerPublisher"
| summarize
    avg(duration),
    percentiles(duration, 50, 90, 95, 99),
    max(duration)
by bin(timestamp, 1h)
| render timechart

Publishing Success Rate​

traces
| where message contains "Published" and message contains "layer(s) to ArcGIS"
| extend
    project_id = extract("project_id=([^\\s]+)", 1, message),
    layer_count = toint(extract("Published ([0-9]+) layer", 1, message))
| summarize
    count(),
    avg(layer_count),
    sum(layer_count)
by bin(timestamp, 1h)
| render timechart

Log Queries​

Recent Function Executions​

traces
| where cloud_RoleName == "swmm-feature-publisher"
| where message contains "Function execution started"
   or message contains "Successfully completed processing"
| project timestamp, message, severityLevel
| order by timestamp desc
| take 100

Failed Executions​

traces
| where cloud_RoleName == "swmm-feature-publisher"
| where severityLevel >= 3  // Warning and above
| where message contains "project_id"
| extend project_id = extract("project_id=([^,\\s]+)", 1, message)
| project timestamp, project_id, message, severityLevel
| order by timestamp desc

Projects Needing Intervention​

traces
| where message contains "Max attempts reached"
   or message contains "needs_intervention=True"
| extend project_id = extract("project_id=([^,\\s]+)", 1, message)
| summarize count() by project_id, bin(timestamp, 1d)
| order by timestamp desc

Alerts​

Critical Alerts​

1. Function Failures Alert

requests
| where name == "swmmBlobTriggerPublisher"
| where success == false
| summarize count() by bin(timestamp, 5m)
| where count_ > 5

2. Projects Needing Intervention

traces
| where message contains "Max attempts reached"
| summarize count() by bin(timestamp, 1h)
| where count_ > 0

3. High Processing Duration

requests
| where name == "swmmBlobTriggerPublisher"
| where duration > 300000  // 5 minutes
| summarize count() by bin(timestamp, 15m)

Dashboards​

Create custom dashboard in Azure Portal with these tiles:

Success Rate (Last 24h):

requests
| where timestamp > ago(24h)
| where name == "swmmBlobTriggerPublisher"
| summarize
    SuccessRate = 100.0 * countif(success == true) / count()
| project SuccessRate

Executions Over Time:

requests
| where timestamp > ago(24h)
| summarize
    Successful = countif(success == true),
    Failed = countif(success == false)
by bin(timestamp, 1h)
| render timechart

Average Processing Time:

requests
| where timestamp > ago(24h)
| where name == "swmmBlobTriggerPublisher"
| summarize avg(duration) by bin(timestamp, 1h)
| render timechart

Availability Test​

Create availability test in Application Insights:

  • URL: https://<function-app>.azurewebsites.net/api/health
  • Frequency: Every 5 minutes
  • Locations: Multiple regions
  • Alert if 2+ locations fail

Error Handling & Recovery​

Retry Logic​

The function automatically retries failed executions based on the retry policy configured in host.json:

  • Exponential backoff strategy
  • Maximum retry count: 2 (configurable)
  • Retry intervals: 2-15 seconds

Manual Intervention​

Projects with NeedsIntervention = true require manual review:

  1. Check LastErrorMessage in Projects table
  2. Verify INP file exists and is valid
  3. Check ArcGIS permissions
  4. Reset status if needed:
    python scripts/reset_project_status.py --project-id <id>

Performance Tuning​

Concurrency​

Blob triggers process files automatically as they're uploaded. The function runtime handles concurrency automatically based on available resources.

Timeouts​

  • Blob download: 300s (5 min)
  • Function timeout: 600s (10 min, configurable in Azure Portal)

Security​

  • Secrets: Never commit local.settings.json
  • Production: Use Managed Identity instead of connection strings
  • SAS Tokens: Use least-privilege permissions, short expiration
  • Input Validation: All blob paths and project IDs validated for security

Troubleshooting​

Common Issues​

Blob trigger doesn't activate​

Check:

  • Function is running (func start)
  • 0b61f1_STORAGE connection string is correct in local.settings.json
  • Blob was uploaded to correct container (swmm-input)
  • File has .inp extension

Solution: Check function logs for any startup errors

Function activates but fails immediately​

Check:

  • All environment variables are set in local.settings.json
  • ArcGIS API key is valid
  • Storage credentials have correct permissions

Solution: Review function logs for specific error messages

Status stuck at "Processing"​

Check:

  • Function didn't crash (check logs for exceptions)
  • ArcGIS publishing didn't timeout or fail
  • Network connectivity to ArcGIS Online

Solution: Check LastErrorMessage field in Projects table

Multiple triggers for same upload​

Possible causes:

  • Blob being modified/overwritten multiple times
  • Function runtime creating multiple instances

Solution: Check idempotency logic is working (should skip if "Complete")

Debugging Steps​

  1. Check Function Logs: Review Application Insights or function logs for errors
  2. Verify Storage: Ensure blob exists and is accessible
  3. Check Table Status: Query Projects table for current status
  4. Verify ArcGIS: Check ArcGIS Online for published layers
  5. Review Metadata: Check ProjectFeatureMetaData table for stored URLs
Last updated on by jacobgqc