HDF File Processing Flow

Key Processing Steps

1. SFTP Download

   • Downloads new .tgz files from NOAA server
   • Stores in /dbdrive/orsanco/
   • Tracks downloaded files

2. File Extraction

   • Extracts .tgz files to temporary directories
   • Processes HDF5 files (Ohio2018a.p02.hdf)
   • Maintains processing history

3. HDF5 Processing

   • Extracts flow, velocity, stage, and time datasets
   • Converts timestamps to Unix epoch
   • Merges datasets into unified format

4. Database Operations

   • Processes data in 25,000 row chunks
   • Uses bulk upsert for efficiency
   • Maintains data integrity

5. Daily Aggregation

   • Calculates daily statistics
   • Stores in separate table
   • Cleans up temporary data

Error Handling

• SFTP connection errors
• File processing errors
• Database transaction failures
• Cleanup failures

Performance Optimizations

• Chunked processing
• Bulk database operations
• Memory cleanup
• Temporary file management