Skip to main content

DNV Meter Device Processing Results

🔧 Device Processing Scripts​

parse_devices_csv.py​

Purpose: Parse device static/metadata CSV files and create device objects in the database.

Usage:

python parse_devices_csv.py devices.csv

Key Features:

  • Handles multiple CSV encodings (UTF-8, CP1252, Latin-1, ISO-8859-1)
  • Uses AssetId as unique identifier
  • Prevents duplicate SerialNumber conflicts
  • Handles empty serial numbers gracefully
  • Maps CSV fields to PascalCase database columns

Sample Output:

2025-10-01 10:30:15,123 - INFO - Starting device CSV processing: devices.csv
2025-10-01 10:30:15,124 - INFO - Successfully opened file with utf-8 encoding
2025-10-01 10:30:15,125 - INFO - Parsed 1,250 device records from devices.csv
2025-10-01 10:30:15,126 - INFO - Connected to database successfully
2025-10-01 10:30:15,130 - INFO - Created new device with ID 1234 for AssetId ASSET_001 with serial 70093784A
2025-10-01 10:30:15,131 - INFO - Device with AssetId ASSET_002 already exists, updating...
2025-10-01 10:30:15,135 - INFO - Processing completed: 1,250 records processed, 0 skipped, 0 errors

📊 Timeseries Processing Scripts​

parse_timeseries_csv_bulk.py (Bulk Analysis Approach) ⭐ RECOMMENDED​

Purpose: Parse meter timeseries CSV files using bulk analysis for optimal performance.

Usage:

python parse_timeseries_csv_bulk.py timeseries.csv

Key Features:

  • Pre-analyzes CSV for unique tag combinations
  • Bulk creates all DeviceTags in one operation
  • Bulk processes DeviceTagDatum records
  • Handles missing devices gracefully
  • Optimized for large datasets

Sample Output:

2025-10-01 11:49:07,258 - INFO - Starting bulk timeseries CSV processing: consumption_2010_to_2023.csv
2025-10-01 11:49:07,285 - INFO - Successfully loaded file with utf-8 encoding
2025-10-01 11:49:07,290 - INFO - Loaded 47,524 valid records from CSV
2025-10-01 11:49:07,331 - INFO - Connected to database successfully
2025-10-01 11:49:07,331 - INFO - Analyzing unique tag combinations...
2025-10-01 11:49:07,337 - INFO - Found 897 unique tag combinations
2025-10-01 11:49:07,351 - INFO - Analyzed 897 unique tag combinations
2025-10-01 11:49:07,351 - INFO - Bulk creating DeviceTag records for 897 combinations...
2025-10-01 11:49:07,360 - INFO - Found 553 devices in database
2025-10-01 11:49:07,360 - INFO - Skipping 343 missing devices
2025-10-01 11:49:07,365 - INFO - Created 553 DeviceTag records
2025-10-01 11:49:07,370 - INFO - Bulk creating DeviceTagDatum records for 47,524 records...
2025-10-01 11:49:07,420 - INFO - Created 42,891 DeviceTagDatum records
2025-10-01 11:49:07,425 - INFO - Skipped 4,633 records due to missing devices
2025-10-01 11:49:07,430 - INFO - All changes committed successfully

🔍 Analysis Scripts​

analyze_device_joins.py​

Purpose: Analyze how many serial numbers from timeseries CSV can be joined with the Devices table.

Usage:

python analyze_device_joins.py timeseries.csv

Output: Comprehensive analysis report + CSV file with failed joins.

================================================================================
DEVICE JOIN ANALYSIS REPORT
================================================================================

📊 CSV FILE ANALYSIS:
  Total records: 49,142
  Unique serial numbers: 897
  Average records per serial: 54.8

🔗 JOIN ANALYSIS:
  Successful joins: 553 (61.6%)
  Failed joins: 344 (38.4%)

📈 RECORD IMPACT:
  Records with successful joins: 31,308 (63.7%)
  Records with failed joins: 16,217 (33.0%)

❌ MISSING DEVICES ANALYSIS:
  Total missing devices: 344
  Total missing records: 16,217
  Average records per missing device: 47.1

💡 RECOMMENDATIONS:
  ⚠️  High failure rate (38.4%) - investigate missing devices
  ⚠️  High impact on records (16,217 records) - consider data cleanup

📄 EXPORTED CSV:
  Failed join serial numbers exported to: C:\git\gqc\hydrotrek-dwd-suite\epanetconsolecore\NetworkManager\python\devices\failed_device_joins.csv
  Total failed serials: 344
2025-10-02 11:27:58,256 - INFO - Analysis completed successfully
Last updated on by Jake