River Flows System - Functional Specifications
Overview
The River Flows System is a Django-based application that processes, stores, and serves river flow data. It handles HDF5 files containing river flow measurements and processes them into either a PostgreSQL database with TimescaleDB extension (Vanilla) or a SQL Server database (RSMS). The system provides REST API endpoints for data access and supports different processing pipelines based on the implementation.
Core Functionality
1. Data Ingestion
Common Features
- Downloads HDF5 files from an SFTP server
- Processes HDF5 files containing river flow data
- Extracts flow measurements
- Converts timestamps to Unix epoch format
- Tracks processed files to avoid duplicate processing
Vanilla Implementation
- Extracts velocity measurements
- Processes stage measurements
- Uses Django models for file tracking
RSMS Implementation
- Calculates area from flow/velocity ratio
- Uses SQL Server for file tracking
- Implements RivermileIndex mapping
2. Data Storage
Vanilla Implementation
- Stores raw flow data in
RiverFlowtable with:- Timestamp
- Station reference
- Flow measurements
- Velocity measurements
- Stage measurements
- Maintains daily aggregated data in
DailyRiverFlowtable with:- Daily timestamps
- Station reference
- Statistical measures (avg, max, min, stddev) for:
- Flow
- Velocity
- Stage
RSMS Implementation
- Stores flow data in
HEC_RASFlowtable with:- Timestamp
- UDID (Unique Data Identifier)
- Flow measurements
- Area measurements
- Stage measurements
- River mile information
- Tributary mile information
- Tracks processed files in
HEC_RASFilesProcessedtable - Maintains station mapping in
RivermileIndextable
3. Data Access API
Common Endpoints
- Raw flow data retrieval
- River, reach, and station information
- Filtering by:
- Time ranges
- Specific stations
- Rivers and reaches
Vanilla-Specific Endpoints
- Daily aggregated data retrieval
- Statistical analysis endpoints
- Time-series specific queries
RSMS-Specific Endpoints
- Area-based measurements
- River mile-based queries
- Tributary-specific data
4. Data Models
Vanilla Implementation
- River: Represents a river system
- Reach: Represents a section of a river
- Station: Represents a measurement station with:
- HDF index mapping
- Geographic coordinates
- River and reach associations
- FlowFileProcessed: Tracks processed HDF files
RSMS Implementation
- HEC_RASFlow: Stores flow measurements with:
- Area calculations
- River mile data
- Tributary mile data
- HEC_RASFilesProcessed: Tracks processed files
- RivermileIndex: Maps HDF indices to physical locations
System Requirements
Common Requirements
- Python 3.x
- Django
- Pandas
- SQLAlchemy
- Fastcore
- TQDM
- Nginx
Vanilla-Specific
- PostgreSQL with TimescaleDB extension
- Django ORM
RSMS-Specific
- SQL Server
- Windows compatibility
- ODBC drivers
Configuration
Common Settings
- SFTP server settings
- Local file storage paths
- Processing parameters
Vanilla-Specific
- TimescaleDB configuration
- Django database settings
- ORM configuration
RSMS-Specific
- SQL Server connection parameters
- Windows-specific paths
- ODBC configuration
Error Handling
Common Features
- Tracks successful and failed file processing
- Maintains processing history
- Provides error logging
- Handles SFTP connection issues
- Manages database transaction integrity
Implementation-Specific
- Vanilla: Django ORM error handling
- RSMS: SQL Server-specific error handling
- Windows: PowerShell error handling
Security
Common Features
- Secure SFTP connections
- Database credential management
- API endpoint access control
- File system access restrictions
Implementation-Specific
- Vanilla: Django authentication
- RSMS: SQL Server authentication
- Windows: PowerShell security policies