Skip to main content

HDF File Processing Flow

Overview​

This document details the processing flow for HDF5 files in both the Vanilla (ORSANCO) and RSMS implementations of the River Flows System. The system processes HDF5 files containing river flow data, with different processing paths for each implementation.

Vanilla Implementation Flow​

RSMS Implementation Flow​

NOTE The RSMS implementation uses Task Scheduler instead of Cron, since it's on Windows.

Key Processing Steps​

1. File Download and Extraction​

Common Steps​

  • SFTP connection establishment
  • Remote file listing
  • Local file comparison
  • New file download
  • .tgz archive extraction

Implementation Differences​

  • Vanilla: Uses Linux shell scripts
  • RSMS: Supports both Linux and Windows environments

2. HDF5 Processing​

Common Steps​

  • File navigation
  • Dataset extraction
  • Timestamp conversion
  • Data normalization

Implementation Differences​

  • Vanilla: Processes velocity directly
  • RSMS: Calculates area from flow/velocity

3. Data Transformation​

Common Steps​

  • DataFrame creation
  • Dataset merging
  • Station mapping
  • Data validation

Implementation Differences​

  • Vanilla: Uses Django ORM models
  • RSMS: Uses SQL Server tables
  • Different station mapping approaches

4. Database Operations​

Common Steps​

  • Connection establishment
  • Chunked processing
  • Bulk operations
  • Transaction management

Implementation Differences​

  • Vanilla: TimescaleDB bulk copy
  • RSMS: SQL Server bulk upsert
  • Different chunk sizes
  • Different error handling

5. Post-Processing​

Vanilla-Specific​

  • Daily aggregation
  • Statistical calculations
  • Time-series optimization

RSMS-Specific​

  • Area calculations
  • River mile processing
  • Tributary mile handling

Error Handling​

Common Error Types​

  • SFTP connection failures
  • File extraction errors
  • Data processing errors
  • Database operation failures

Implementation-Specific Handling​

  • Vanilla: Django ORM error handling
  • RSMS: SQL Server error handling
  • Different retry mechanisms
  • Different logging approaches

Performance Considerations​

Common Optimizations​

  • Chunked processing
  • Bulk operations
  • Memory management
  • File system optimization

Implementation-Specific​

  • Vanilla: TimescaleDB optimizations
  • RSMS: SQL Server optimizations
  • Different chunk sizes
  • Different indexing strategies
Last updated on by Jake