Devices Scripts - Overview
This directory contains Python scripts for processing device and timeseries data in the HydroTrek DWD Suite. The scripts handle device metadata parsing, timeseries data processing, and database operations.
📂 Directory Structure
This directory is organized into two versions:
v1/ - Legacy Implementation (Reference)
Contains the original timeseries processing scripts that use metadata-based frequency detection.
- Single tag per device
- Frequency based on ROUTE_CATEGORY field
- Row-by-row processing
v2/ - New Implementation (Recommended)
Contains the upgraded timeseries processing with dynamic frequency detection.
- Multiple tags per device (one per frequency type)
- Frequency detected from actual data patterns
- Parallel LPS (liters/second) tags for hydraulic modeling
- Pandas-based bulk processing
- Supports devices with mixed frequencies over time
📖 Documentation Structure
- 01_Overview/ - This overview document
- 02_Design/ - Technical design specifications
- 03_Implementation/ - Implementation plan and summary
- 04_Analysis/ - Data analysis results
- 05_User-Guide/ - User documentation and quick start guides
📁 Scripts Overview
v1 - Legacy Scripts (Reference Only)
| Script | Purpose | Input | Output |
|---|---|---|---|
v1/parse_devices_csv.py | Parse device metadata CSV | Device CSV | Database records |
v1/add_bldgtype_to_devices.py | Add building type to existing devices | Device CSV with BLDGTYPE | Updated DeviceGroup & Device records |
v1/parse_timeseries_csv.py | Parse timeseries data (create-as-encountered) | Timeseries CSV | Database records |
v1/parse_timeseries_csv_bulk.py | Parse timeseries data (bulk analysis) | Timeseries CSV | Database records |
v1/analyze_device_joins.py | Analyze device join success/failure | Timeseries CSV | Analysis report + CSV |
v1/preview_timeseries_processing.py | Preview what will be processed | Timeseries CSV | Processing preview |
v1/tag_creation_analysis.py | Analyze DeviceTag creation strategies | Timeseries CSV | Performance analysis |
v1/convert_db_device_coordinates.py | Convert device coordinates | Database | Updated coordinates |
v2 - New Implementation (Current)
| Script | Purpose | Input | Output |
|---|---|---|---|
v2/analyze_frequency_patterns.py | Analyze reading frequency patterns | Timeseries CSV | Analysis report |
v2/parse_timeseries_csv_v2.py | Parse timeseries with dynamic frequency | Timeseries CSV | DeviceTags & Data (consumption + LPS) |
v2/validate_results.py | Validate processing results | Database | Validation report |
v2/compare_device_data_coverage.py | Compare device and data coverage | Two CSVs | Coverage analysis |
Key Improvements in V2
- Dynamic Frequency Detection - Determines frequency from actual reading patterns, not metadata
- Mixed Frequency Support - Handles devices that change frequency over time
- LPS Tag Creation - Parallel liters/second tags for hydraulic modeling
- Max-Period Classification - Uses maximum period number instead of count for accurate classification
- Pandas-Based Processing - Efficient bulk operations
- Comprehensive Validation - Built-in validation and analysis tools
Quick Links
Status
✅ V2 Implementation Complete (November 2025)
- All core features implemented
- Comprehensive testing completed
- Production ready
Last Updated: November 3, 2025