Skip to main content

River Flows System - Functional Specifications

Overview​

The River Flows System is a Django-based application that processes, stores, and serves river flow data. It handles HDF5 files containing river flow measurements, processes them into a PostgreSQL database with TimescaleDB extension, and provides REST API endpoints for data access.

Core Functionality​

1. Data Ingestion​

  • Downloads HDF5 files from an FTP server (configurable)
  • Processes HDF5 files containing river flow data
  • Extracts flow, velocity, and stage measurements
  • Converts timestamps to Unix epoch format
  • Tracks processed files to avoid duplicate processing

2. Data Storage​

  • Stores raw flow data in RiverFlow table with:
    • Timestamp
    • Station reference
    • Flow measurements
    • Velocity measurements
    • Stage measurements
  • Maintains daily aggregated data in DailyRiverFlow table with:
    • Daily timestamps
    • Station reference
    • Statistical measures (avg, max, min, stddev) for:
      • Flow
      • Velocity
      • Stage

3. Data Access API​

Provides REST API endpoints for:

  • Raw flow data retrieval
  • Daily aggregated data retrieval
  • River, reach, and station information
  • Filtering by:
    • Time ranges
    • Specific stations
    • Rivers and reaches

4. Data Models​

  • River: Represents a river system
  • Reach: Represents a section of a river
  • Station: Represents a measurement station with:
    • HDF index mapping
    • Geographic coordinates
    • River and reach associations
  • FlowFileProcessed: Tracks processed HDF files

System Requirements​

  • Python 3.x
  • Django
  • PostgreSQL with TimescaleDB extension
  • Paramiko (for FTP operations)
  • Pandas
  • SQLAlchemy
  • Fastcore
  • TQDM
  • Nginx

Configuration​

  • FTP server settings (optional)
  • Database connection parameters
  • Local file storage paths
  • Processing parameters

Error Handling​

  • Tracks successful and failed file processing
  • Maintains processing history
  • Provides error logging
  • Handles FTP connection issues
  • Manages database transaction integrity

Security​

  • Secure FTP connections
  • Database credential management
  • API endpoint access control
  • File system access restrictions
Last updated on by Jake