Skip to main content

Network Model and Shapefile Processing

The following diagram shows the high-level flow of how EPANET Models (inp/input files) and Shapefiles (purely shape-based representations of networks) will be used by GQC to support network modeling, the use of DWD software, and network model modifications (adding, removing, modifying EPANET networks).

Diagram displaying the flow of data related to EPANET model files and Shapefile

Some of the pieces of the flow are already complete, such as the flow from EPANET Model inp file to frontend display pipeline.

Most other pieces of the flow, namely the Shapefile standardization, storage, and modification flows, need to be designed and implemented.

Data Flow Description

I'm writing up the Data Flow description for Cursor. This will be a textual representation of the flow diagram PNG. Eventually, I would like Cursor to generate a mermaid diagram to facilitate the design-to-LLM communications.

The flow always starts with clients providing a network file to GQC. This network file can be either an EPANET Model input file or a generic Shapefile containing lines (and a separate file containing points).

Network Model and Shapefile Processing Flow

The following mermaid diagram illustrates the complete data flow for processing both EPANET models and shapefiles within the GQC system:

Flow Description

The system supports two primary input paths:

EPANET Model Path:

  1. Client provides EPANET Model input file
  2. NetworkManager processes and imports to PostgreSQL database
  3. Data flows to DwdApiAspnet for API serving
  4. JSON geometries are generated for 2D and 3D map visualization
  5. Optional: PSQL data can be exported to standardized shapefile format

Shapefile Path:

  1. Client provides generic shapefile
  2. Shapefile is standardized to GQC EPANET conventions
  3. Standardized shapefile is uploaded to ArcGIS Online
  4. Network Feature Layer is created and branches into:
    • 2D Network Editor for frontend editing
    • EPANET Model generation for blob storage

Original Flow Descriptions

First Draft (Goal-oriented):

  1. Client provides EPANET Model input file
    1. Input to NetworkManager.csproj process block to import the network into output circle PSQL DB.
    2. (Developer-TODO-3) Process block - GQC will write a Shapefile from the PSQL circle (outputs to "STANDARDIZED Network Shapefile" then DEAD END for this flow branch).
    3. The PSQL is input to a process block "DwdApiAspnet", representing the API serving the geometry data
    4. JSON Geometries are output and input to blocks for 2D and 3D (attributes-only) maps
  2. Client provides a generic Shapefile
    1. (Developer-TODO-1) Shapefile is input to process block "Standardize to GQC EPANET Conventions"
    2. Standardized shapefile is output.
    3. Shapefile input to process block "Upload to arcgis online" resulting in output "network feature layer"
    4. Network feature layer is input into two flows:
      1. 2D Network Editor process block
      2. (Developer-TODO-2) Write EPANET Model process block into blob storage

Second Draft (I/O focused):

  1. (Process block) with "Client provides network file", branches in two directions with outputs: a. (Resource circle) EPANET Model Input File
    1. (Process block) EpanetConsoleCore.NetworkManager
    2. (Resource circle) EPANET Network Geometries and Attributes (PSQL)
    3. (Process block) DwdApiAspnet
    4. (Resource circle) EPANET Network Geometries and Attributes (JSON) a. (Process block) 2D Maps b. (Process block) 3D Maps (attributes only) b. (Resource circle) RAW Network Shapefile
    5. (Process block) Standardize to GQC EPANET Conventions
    6. (Resource circle) STANDARDIZED Network Shapefile
    7. (Process block) Upload to ArcGIS Online
    8. (Resource circle) Network Feature Layer, splits into two branches: a. (Process block) Write EPANET
      1. (Resource circle) Input File (in Blob Storage) b. (Process block) 2D Network Editor page in Frontend GUI

Implementation Steps

1. Standardize Shapefile for EPANET Compatibility

  • The shapefile geometries should have properties that are consistent with GQC's standard EPANET naming conventions.
  • EPANET Model Attributes Analysis:

Based on EPANET 2.2 specifications and GQC's implementation, the following attributes are required for network elements:

Node Classes and Attributes

Junction (Required Attributes):

  • ID (string) - REQUIRED - Unique identifier
  • Elevation (double) - REQUIRED - Ground elevation in feet/meters
  • Demand (double) - REQUIRED - Base demand in flow units
  • Pattern (string) - Optional - Demand pattern ID
  • Comment (string) - Optional - Descriptive text

Tank (Required Attributes):

  • ID (string) - REQUIRED - Unique identifier
  • Elevation (double) - REQUIRED - Bottom elevation in feet/meters
  • InitLevel (double) - REQUIRED - Initial water level in feet/meters
  • MinLevel (double) - REQUIRED - Minimum water level in feet/meters
  • MaxLevel (double) - REQUIRED - Maximum water level in feet/meters
  • Diameter (double) - REQUIRED - Tank diameter in feet/meters
  • MinVol (double) - Optional - Minimum volume in cubic feet/cubic meters
  • VolCurve (string) - Optional - Volume curve ID
  • Overflow (string) - Optional - Overflow condition (YES/NO)

Reservoir (Required Attributes):

  • ID (string) - REQUIRED - Unique identifier
  • Head (double) - REQUIRED - Total head in feet/meters
  • Pattern (string) - Optional - Head pattern ID
  • Comment (string) - Optional - Descriptive text

Pipe (Required Attributes):

  • ID (string) - REQUIRED - Unique identifier
  • Node1 (string) - REQUIRED - Start node ID
  • Node2 (string) - REQUIRED - End node ID
  • Length (double) - REQUIRED - Pipe length in feet/meters
  • Diameter (double) - REQUIRED - Pipe diameter in inches/mm
  • Roughness (double) - REQUIRED - Roughness coefficient
  • MinorLoss (double) - Optional - Minor loss coefficient
  • Status (string) - Optional - Status (OPEN/CLOSED/CV)

Pump (Required Attributes):

  • ID (string) - REQUIRED - Unique identifier
  • Node1 (string) - REQUIRED - Start node ID
  • Node2 (string) - REQUIRED - End node ID
  • Power (double) - REQUIRED - Pump power in horsepower/kW
  • Speed (double) - Optional - Relative speed setting
  • Pattern (string) - Optional - Speed pattern ID
  • Curve (string) - Optional - Pump curve ID

Valve (Required Attributes):

  • ID (string) - REQUIRED - Unique identifier
  • Node1 (string) - REQUIRED - Start node ID
  • Node2 (string) - REQUIRED - End node ID
  • Diameter (double) - REQUIRED - Valve diameter in inches/mm
  • Type (string) - REQUIRED - Valve type (PRV/PSV/FCV/TCV/GPV)
  • Setting (double) - REQUIRED - Valve setting value
  • MinorLoss (double) - Optional - Minor loss coefficient

Coordinate Attributes (All Elements)

  • X (double) - REQUIRED - X coordinate in WGS84 for visualization
  • Y (double) - REQUIRED - Y coordinate in WGS84 for visualization

Shapefile to EPANET Mapping Configuration

To support various shapefile formats and naming standards, I recommend implementing a flexible configuration-based mapping system:

1. Element Type Detection:

{
"elementTypeDetection": {
"nodeTypes": {
"junction": ["junction", "jct", "node", "point"],
"tank": ["tank", "storage", "reservoir_tank"],
"reservoir": ["reservoir", "source", "supply"]
},
"linkTypes": {
"pipe": ["pipe", "main", "line", "conduit"],
"pump": ["pump", "booster", "lift_station"],
"valve": ["valve", "control_valve", "prv", "psv"]
}
}
}

2. Attribute Mapping Configuration:

{
"attributeMapping": {
"nodes": {
"junction": {
"required": {
"ID": ["id", "node_id", "name", "label"],
"Elevation": ["elevation", "elev", "ground_elev", "z"],
"Demand": ["demand", "flow", "base_demand", "usage"]
},
"optional": {
"Pattern": ["pattern", "demand_pattern", "flow_pattern"],
"Comment": ["comment", "description", "notes"]
}
},
"tank": {
"required": {
"ID": ["id", "tank_id", "name", "label"],
"Elevation": ["elevation", "elev", "bottom_elev", "z"],
"InitLevel": ["init_level", "initial_level", "start_level"],
"MinLevel": ["min_level", "minimum_level", "low_level"],
"MaxLevel": ["max_level", "maximum_level", "high_level"],
"Diameter": ["diameter", "diam", "tank_diameter", "size"]
}
}
},
"links": {
"pipe": {
"required": {
"ID": ["id", "pipe_id", "name", "label"],
"Node1": ["from_node", "start_node", "node1", "upstream"],
"Node2": ["to_node", "end_node", "node2", "downstream"],
"Length": ["length", "len", "pipe_length", "distance"],
"Diameter": ["diameter", "diam", "pipe_diameter", "size"],
"Roughness": ["roughness", "rough", "hazen_williams", "c_factor"]
}
}
}
}
}

3. Coordinate System Configuration:

{
"coordinateMapping": {
"xField": ["x", "longitude", "lon", "easting", "x_coord"],
"yField": ["y", "latitude", "lat", "northing", "y_coord"],
"coordinateSystem": {
"source": "EPSG:xxxx",
"target": "EPSG:4326",
"units": "meters"
}
}
}

4. Validation Rules: I'm not interested in managing validation rules at this time...

Implementation Benefits

  1. Flexibility: Supports multiple naming conventions without code changes
  2. Maintainability: Easy to update mappings for new client formats
  3. Extensibility: Can add new element types and attributes easily
  4. Documentation: Self-documenting configuration structure
  5. Python Integration: Leverages arcpy and robust geospatial libraries for reliable geometry processing

Alternative Approaches Considered

  • Database-driven mapping: More complex but allows runtime configuration
  • Hard-coded mapping: Simple but inflexible for different clients
  • Machine learning approach: Overkill for this use case

Recommendation: Start with JSON configuration files as they provide the best balance of flexibility, maintainability, and simplicity.

Shapefile Storage Strategy

Since your NetworkManager already handles WGS84 conversion for database storage, the question is how to handle shapefile storage and processing. Here are the two approaches:

Why This Is Better:

  1. Preserves Original Accuracy: No precision loss from coordinate transformations
  2. Maintains Client Context: Shapefiles remain in the projection the client expects
  3. Reduces Processing Overhead: No unnecessary conversions during shapefile operations
  4. Supports Regional Workflows: Clients can work in their familiar coordinate systems
  5. Flexible Export: Can export to any projection without intermediate conversions

Implementation Strategy:

class ShapefileManager:
def __init__(self):
self.shapefile_metadata = {}

def store_shapefile(self, shapefile_path, original_crs):
"""Store shapefile in original projection with CRS metadata"""
# Store the shapefile as-is
# Record CRS information in metadata
self.shapefile_metadata[shapefile_path] = {
'crs': original_crs,
'epsg_code': original_crs.to_epsg(),
'units': original_crs.axis_info[0].unit_name,
'stored_date': datetime.now()
}

def convert_for_processing(self, shapefile_path, target_crs='EPSG:4326'):
"""Convert shapefile to target CRS only when needed for processing"""
if target_crs == 'EPSG:4326':
# Convert to WGS84 for database operations
return self._convert_to_wgs84(shapefile_path)
else:
# Convert to other CRS as needed
return self._convert_to_crs(shapefile_path, target_crs)

def export_to_original_crs(self, processed_data, original_shapefile_path):
"""Export processed data back to original CRS"""
original_crs = self.shapefile_metadata[original_shapefile_path]['crs']
return self._convert_to_crs(processed_data, original_crs)
# 1. Import shapefile (keep in original projection)
original_shapefile = "client_network.shp" # e.g., UTM Zone 15N
store_shapefile(original_shapefile, original_crs="EPSG:26915")

# 2. Convert to WGS84 only when needed for database operations
wgs84_data = convert_for_processing(original_shapefile, "EPSG:4326")
store_in_database(wgs84_data) # Your NetworkManager handles this

# 3. Process shapefile operations in original projection
process_shapefile_edits(original_shapefile) # Keep in UTM Zone 15N

# 4. Convert to WGS84 only when updating database
updated_wgs84 = convert_for_processing(original_shapefile, "EPSG:4326")
update_database(updated_wgs84)

# 5. Export back to original projection for client
export_to_client(original_shapefile) # Still in UTM Zone 15N

Benefits of Keeping Original Projection

  • Accuracy Preservation: No precision loss from unnecessary transformations
  • Client Satisfaction: Shapefiles remain in familiar coordinate systems
  • Processing Efficiency: Only convert when actually needed
  • Flexibility: Can work in any projection without constraints
  • Regional Optimization: UTM zones provide better local accuracy than WGS84

Recommendation: Keep shapefiles in their original projection and only convert to WGS84 when needed for database operations or EPANET export. This preserves accuracy and maintains client context while leveraging your existing NetworkManager for database conversions.

2. Convert ESRI Feature Layer to EPANET Model Input File

info

Users will NOT be editing EPANET models directly. Instead, GQC has decided to rely on the Feature Layer editing functionality provided by ESRI out of the box. The only way to modify a network will be against the Feature Layer. Then, the modified Feature Layer will need to be exported to an EPANET Model (inp) file for use in the EPANET engine.

  • When the user modifies a Feature Layer, we need functionality to convert that Feature Layer into a new EPANET Input File.
  • This will require programmatic access to the asset hosted on ArcGIS Online
  • We will need to consider how to 1. retrieve the asset and 2. process it (in memory or on local drive?)

Preliminary Design for Feature Layer to EPANET Conversion

Architecture Overview

Implementation Components

1. Python Geospatial Processing Service

import arcpy
import geopandas as gpd
import pandas as pd
from shapely.geometry import Point, LineString
import requests
import json

class ArcGISFeatureProcessor:
def __init__(self, service_url, token):
self.service_url = service_url
self.token = token
self.arcpy.env.overwriteOutput = True

def get_network_features(self, layer_name, geometry_type="ALL"):
"""Extract network features using arcpy for robust geometry handling"""
# Use arcpy to access feature service
feature_layer = arcpy.MakeFeatureLayer_management(
f"{self.service_url}/{layer_name}",
f"temp_{layer_name}"
)

# Convert to GeoDataFrame for processing
with arcpy.da.SearchCursor(feature_layer, "*") as cursor:
fields = [field.name for field in arcpy.ListFields(feature_layer)]
data = [row for row in cursor]

return gpd.GeoDataFrame(data, columns=fields)

def get_modified_features(self, layer_name, last_sync_date):
"""Get features modified since last sync using arcpy date queries"""
where_clause = f"EditDate > '{last_sync_date}'"
return self.get_network_features(layer_name, where_clause)

2. Network Topology Processing

import networkx as nx
from shapely.ops import snap

class NetworkTopologyProcessor:
def __init__(self):
self.network_graph = nx.Graph()

def build_network_graph(self, nodes_gdf, links_gdf):
"""Build network connectivity graph from nodes and links"""
# Add nodes to graph
for idx, node in nodes_gdf.iterrows():
self.network_graph.add_node(
node['ID'],
pos=(node.geometry.x, node.geometry.y),
attributes=node.to_dict()
)

# Add edges (links) to graph
for idx, link in links_gdf.iterrows():
if link['Node1'] in self.network_graph and link['Node2'] in self.network_graph:
self.network_graph.add_edge(
link['Node1'],
link['Node2'],
attributes=link.to_dict()
)

def fix_connectivity_issues(self, nodes_gdf, links_gdf, snap_tolerance=1.0):
"""Fix common topology issues using shapely operations"""
# Snap nodes to nearby links
for idx, node in nodes_gdf.iterrows():
node_point = node.geometry
for link_idx, link in links_gdf.iterrows():
if node_point.distance(link.geometry) <= snap_tolerance:
# Snap node to link geometry
snapped_point = link.geometry.interpolate(
link.geometry.project(node_point)
)
nodes_gdf.at[idx, 'geometry'] = snapped_point

return nodes_gdf, links_gdf

3. EPANET Model Builder

class EpanetModelBuilder:
def __init__(self, config):
self.config = config
self.inp_sections = {}

def build_model(self, nodes_gdf, links_gdf):
"""Convert GeoDataFrames to EPANET INP format"""
# Generate each INP section
self._generate_title_section()
self._generate_junctions_section(nodes_gdf)
self._generate_tanks_section(nodes_gdf)
self._generate_reservoirs_section(nodes_gdf)
self._generate_pipes_section(links_gdf)
self._generate_pumps_section(links_gdf)
self._generate_valves_section(links_gdf)
self._generate_coordinates_section(nodes_gdf)
self._generate_vertices_section(links_gdf)

return self._compile_inp_file()

def _generate_junctions_section(self, nodes_gdf):
"""Generate [JUNCTIONS] section from junction nodes"""
junctions = nodes_gdf[nodes_gdf['Type'] == 'JUNCTION']
section_lines = ["[JUNCTIONS]"]

for idx, junction in junctions.iterrows():
line = f"{junction['ID']:<20} {junction['Elevation']:<10} {junction['Demand']:<10}"
if 'Pattern' in junction and pd.notna(junction['Pattern']):
line += f" {junction['Pattern']}"
section_lines.append(line)

self.inp_sections['JUNCTIONS'] = section_lines

def _generate_coordinates_section(self, nodes_gdf):
"""Generate [COORDINATES] section"""
section_lines = ["[COORDINATES]"]

for idx, node in nodes_gdf.iterrows():
# Transform coordinates if needed
x, y = self._transform_coordinates(node.geometry.x, node.geometry.y)
section_lines.append(f"{node['ID']:<20} {x:<15} {y:<15}")

self.inp_sections['COORDINATES'] = section_lines

def _transform_coordinates(self, x, y):
"""Apply coordinate system transformation"""
# Use arcpy for coordinate transformation
point = arcpy.Point(x, y)
# Apply transformation based on config
return x, y # Simplified for now

def _compile_inp_file(self):
"""Compile all sections into final INP file"""
inp_content = []

# Add sections in EPANET order
section_order = ['TITLE', 'JUNCTIONS', 'TANKS', 'RESERVOIRS',
'PIPES', 'PUMPS', 'VALVES', 'COORDINATES', 'VERTICES']

for section in section_order:
if section in self.inp_sections:
inp_content.extend(self.inp_sections[section])
inp_content.append("") # Empty line between sections

return "\n".join(inp_content)

Data Processing Flow

Step 1: Feature Layer Access

  • Use arcpy to authenticate with ArcGIS Online feature services
  • Query feature service for network elements (nodes and links) using arcpy cursors
  • Handle large datasets with arcpy pagination capabilities
  • Support incremental updates based on modification timestamps using arcpy date queries

Step 2: Geospatial Data Processing

  • Extract geometry and attributes using arcpy for robust geometry handling
  • Convert to GeoPandas DataFrames for efficient processing
  • Apply coordinate system transformations using arcpy projection tools
  • Map feature attributes to EPANET standard format using configuration files

Step 3: Network Topology Processing

  • Build network connectivity graph using NetworkX
  • Use Shapely operations for geometry processing and topology fixes
  • Snap nodes to nearby links using Shapely snap operations
  • Handle duplicate elements and connectivity issues

Step 4: EPANET Model Generation

  • Create INP file structure following EPANET format
  • Generate all required sections with proper formatting
  • Apply unit conversions and scaling factors
  • Include metadata and comments

Step 5: Storage and Versioning

  • Store generated INP file in Azure Blob Storage
  • Maintain version history with timestamps
  • Link to original feature layer for traceability
  • Provide download/access endpoints

Technical Considerations

Performance Optimization

  • Use arcpy's native performance optimizations for large datasets
  • Implement GeoPandas chunking for memory-efficient processing of large networks
  • Use NetworkX for efficient graph operations
  • Leverage arcpy's built-in spatial indexing for fast geometry queries
  • Implement incremental processing for updates using arcpy date queries

Python Library Integration

  • arcpy: Primary tool for ArcGIS Online integration and geometry operations
  • GeoPandas: Efficient handling of geospatial data in DataFrame format
  • Shapely: Advanced geometry operations and topology fixes
  • NetworkX: Network connectivity analysis and graph operations
  • Pandas: Data manipulation and attribute processing

Error Handling

  • Graceful handling of missing or invalid geometry data
  • Robust error handling for arcpy operations
  • Logging and monitoring for troubleshooting
  • Fallback mechanisms for failed geometry operations

Security

  • Secure token management for ArcGIS Online access
  • Encrypted storage of generated INP files
  • Access control for different user roles
  • Audit logging for all conversion operations

Configuration Requirements

{
"arcgisOnline": {
"serviceUrl": "https://services.arcgis.com/...",
"authentication": {
"type": "oauth2",
"clientId": "your_client_id",
"clientSecret": "your_client_secret"
}
},
"processing": {
"batchSize": 1000,
"coordinateSystem": {
"source": "EPSG:xxxx",
"target": "EPSG:4326"
},
"topology": {
"snapTolerance": 1.0,
"maxOrphanedNodes": 10
}
},
"pythonLibraries": {
"arcpy": "ArcGIS Pro installation required",
"geopandas": ">=0.12.0",
"shapely": ">=1.8.0",
"networkx": ">=2.8.0",
"pandas": ">=1.5.0"
},
"storage": {
"containerName": "epanet-models",
"retentionDays": 365
}
}

Python Environment Setup

# requirements.txt
geopandas>=0.12.0
shapely>=1.8.0
networkx>=2.8.0
pandas>=1.5.0
requests>=2.28.0
azure-storage-blob>=12.14.0

# Note: arcpy comes with ArcGIS Pro installation
# Ensure ArcGIS Pro is installed and licensed for arcpy access

This design provides a robust, Python-based solution for converting modified Feature Layers to EPANET models, leveraging the power of arcpy and other geospatial libraries for reliable geometry processing and network analysis.

3. Convert GQC EPANET Model (from PSQL) to Standardized Shapefile

  • In cases where the client provides an EPANET Model input file, or in cases where we have existing networks (such as MTW), we will need to be able to convert from the EPANET Model to Shapefile. What makes the most sense is building the Shapefile from the PSQL model representation.

Python-based PSQL to Shapefile Conversion

import psycopg2
import geopandas as gpd
import pandas as pd
from shapely.geometry import Point, LineString
import arcpy

class PSQLToShapefileConverter:
def __init__(self, connection_string):
self.conn = psycopg2.connect(connection_string)

def extract_network_data(self):
"""Extract network data from PostgreSQL database"""
# Query nodes from database
nodes_query = """
SELECT id, elevation, demand, pattern, comment, x, y, type
FROM network_nodes
"""

# Query links from database
links_query = """
SELECT id, node1, node2, length, diameter, roughness,
minor_loss, status, type, x1, y1, x2, y2
FROM network_links
"""

nodes_df = pd.read_sql(nodes_query, self.conn)
links_df = pd.read_sql(links_query, self.conn)

return nodes_df, links_df

def create_geodataframes(self, nodes_df, links_df):
"""Convert database data to GeoDataFrames"""
# Create node geometries
node_geometries = [Point(x, y) for x, y in zip(nodes_df['x'], nodes_df['y'])]
nodes_gdf = gpd.GeoDataFrame(nodes_df, geometry=node_geometries)

# Create link geometries
link_geometries = [
LineString([(x1, y1), (x2, y2)])
for x1, y1, x2, y2 in zip(links_df['x1'], links_df['y1'],
links_df['x2'], links_df['y2'])
]
links_gdf = gpd.GeoDataFrame(links_df, geometry=link_geometries)

return nodes_gdf, links_gdf

def export_to_shapefile(self, nodes_gdf, links_gdf, output_path):
"""Export GeoDataFrames to shapefiles using arcpy for best compatibility"""
# Use arcpy to ensure proper shapefile creation
arcpy.env.workspace = output_path

# Export nodes
nodes_gdf.to_file(f"{output_path}/network_nodes.shp")

# Export links
links_gdf.to_file(f"{output_path}/network_links.shp")

# Create feature class in geodatabase for better performance
arcpy.CreateFeatureclass_management(
output_path, "network_nodes", "POINT",
nodes_gdf, spatial_reference=arcpy.SpatialReference(4326)
)

arcpy.CreateFeatureclass_management(
output_path, "network_links", "POLYLINE",
links_gdf, spatial_reference=arcpy.SpatialReference(4326)
)

This approach leverages Python's database connectivity and geospatial libraries while using arcpy for reliable shapefile creation and geodatabase management.