Skip to main content

Device Data AI Predictions - Design Document

Overview​

This document outlines the design for adding AI prediction support for Device (water meter) time-series data to the existing dwd-api-fastapi prediction service and dwd-frontend-react interface.

Date: October 23, 2025 Status: Design Phase

Table of Contents​

  1. Current State Analysis
  2. Device Data Model
  3. Design Goals
  4. Architecture Overview
  5. Backend Implementation (FastAPI)
  6. Frontend Implementation (React)
  7. API Contracts
  8. Implementation Plan
  9. Testing Strategy

Current State Analysis​

Existing AI Predictions Infrastructure​

The system currently supports AI predictions for the following network entities:

  • Tanks - Level, Head, Demand, Pressure metrics
  • Reservoirs - Head, Pressure metrics
  • Junctions - Head, Demand, Pressure metrics
  • Pipes - Flow, Velocity metrics
  • Pumps - Flow, Head metrics
  • Valves - Flow, Pressure metrics

How It Works Currently​

  1. Frontend Component: PredictionButton component in dwd-frontend-react/src/components/PredictionButton.jsx
  2. Integration: Used in QualityHydraulicsDropdowns.jsx for node-based entities
  3. Chart Component: ForecastChart.jsx fetches data from FastAPI and displays predictions
  4. FastAPI Service: dwd-api-fastapi/main.py provides:
    • Generic /{node_type}/{node_id}/graph-data endpoint
    • TSAI-based predictions with scikit-learn fallback
    • Configurable forecast percentage (0.1-2.0)
    • Adds isForecast: true flag to predicted data points

Prediction Algorithm​

  • Primary: TSAI InceptionTimePlus model for time series forecasting
  • Fallback: Scikit-learn Linear Regression with sliding window
  • Final Fallback: Simple moving average
  • Window Size: Configurable (default 10-30% of data)
  • Forecast Points: Configurable percentage of original data (default 50-100%)

Device Data Model​

Database Schema​

// Device - Main water meter entity
Device {
  int Id (PK)
  string Name
  string SerialNumber
  string AccountNumber
  string AssetId
  string MeterType
  int DeviceTypeId (FK)
  int DeviceGroupId (FK)
  string Address
  string Location
  double XCoord
  double YCoord
  string Owner
  string Department
  string ContactName
  string ContactNumber
}

// DeviceType - Device categorization
DeviceType {
  int Id (PK)
  string Name         // e.g., "Water Meter"
  string Description  // e.g., "Water consumption meter"
  string Unit         // e.g., "cubic feet"
}

// DeviceGroup - Grouping and billing
DeviceGroup {
  int Id (PK)
  string Name          // e.g., "METERED - MONTHLY"
  string BillingCycle  // e.g., "Monthly"
  string Frequency     // e.g., "Monthly", "Quarterly", "Yearly"
}

// DeviceTag - Measurement tags
DeviceTag {
  int Id (PK)
  string Name      // e.g., "Consumption", "Water Usage"
  string TagType   // e.g., "monthly_consumption", "quarterly_consumption"
  string Unit      // e.g., "cubic feet"
  int DeviceId (FK)
}

// DeviceTagDatum - Time-series data
DeviceTagDatum {
  int DeviceTagId (PK, FK)
  long UnixTime (PK)
  double Value     // Consumption value
}

Existing ASP.NET API Endpoints​

The dwd-api-aspnet already provides the following Device-related endpoints:

GET /api/Device                                  - Get all devices
GET /api/Device/{id}                            - Get device by ID
GET /api/Device/tagtypes                        - Get unique tag types
GET /api/Device/cluster/data?tagType={}&deviceGroupIds={}&unixtime={}
                                                - Get device cluster data
GET /api/Device/cluster/timestamps?tagType={}&deviceGroupIds={}
                                                - Get available timestamps

Key Observations:

  • Device data is already time-series with DeviceTagDatum.UnixTime and DeviceTagDatum.Value
  • Data is organized by TagType (e.g., "monthly_consumption", "quarterly_consumption")
  • Data can be filtered by DeviceGroup for different billing cycles
  • The structure is similar to SCADA data but simpler (single value per timestamp)

Design Goals​

  1. Reuse Existing Infrastructure: Leverage the current TSAI prediction pipeline
  2. Minimal Changes: Extend, don't rebuild
  3. Consistency: Match the UX/API pattern of existing predictions
  4. Flexibility: Support different tag types and device groups
  5. Performance: Handle potentially large datasets efficiently

Architecture Overview​

┌─────────────────────────────────────────────────────────────────┐
│                     Frontend (React)                            │
│  ┌────────────────────┐  ┌──────────────────────────────────┐  │
│  │ DeviceChart.jsx    │  │ QualityHydraulicsDropdowns.jsx   │  │
│  │ (Existing)         │  │ + PredictionButton Integration   │  │
│  └────────────────────┘  └──────────────────────────────────┘  │
│                  │                    │                          │
│                  └────────┬───────────┘                          │
└───────────────────────────┼──────────────────────────────────────┘
                            │
                            ▼
┌─────────────────────────────────────────────────────────────────┐
│              FastAPI Prediction Service (Python)                │
│  ┌──────────────────────────────────────────────────────────┐  │
│  │ NEW: /device/{device_id}/tag/{tag_id}/graph-data         │  │
│  │      - Fetches DeviceTagDatum data from ASP.NET API      │  │
│  │      - Applies TSAI prediction pipeline                  │  │
│  │      - Returns data with forecast points                 │  │
│  └──────────────────────────────────────────────────────────┘  │
│                            │                                     │
│  ┌──────────────────────────────────────────────────────────┐  │
│  │ Shared Prediction Logic (predict_with_tsai)              │  │
│  │ - Same algorithm used for all entity types               │  │
│  └──────────────────────────────────────────────────────────┘  │
└─────────────────────────────────────────────────────────────────┘
                            │
                            ▼
┌─────────────────────────────────────────────────────────────────┐
│              ASP.NET API (C# .NET Core)                         │
│  ┌──────────────────────────────────────────────────────────┐  │
│  │ NEW: /api/Device/{deviceId}/tag/{tagId}/data            │  │
│  │      - Returns DeviceTagDatum for specific tag           │  │
│  │      - Optional startTime/endTime filtering              │  │
│  │      - Returns: [ { unixTime, value }, ... ]             │  │
│  └──────────────────────────────────────────────────────────┘  │
└─────────────────────────────────────────────────────────────────┘

Backend Implementation (FastAPI)​

1. New Data Models​

Add to dwd-api-fastapi/main.py:

class DeviceTagData(BaseModel):
  """Model for device tag datum."""
  deviceTagId: int
  unixTime: int
  value: float

class DeviceTagInfo(BaseModel):
  """Model for device tag information."""
  id: int
  name: str
  tagType: str
  unit: Optional[str] = None
  deviceId: int

2. New FastAPI Endpoint​

@app.get("/device/{device_id}/tag/{tag_id}/graph-data")
async def get_device_tag_graph_data(
  device_id: int,
  tag_id: int,
  base_url: str = Query(description="Base URL for the ASP.NET API"),
  start_time: Optional[int] = Query(
    default=None, description="Start Unix timestamp"
  ),
  end_time: Optional[int] = Query(
    default=None, description="End Unix timestamp"
  ),
  include_forecast: bool = Query(
    default=True, description="Include forecast data"
  ),
  forecast_percentage: float = Query(
    default=1.0, description="Forecast percentage (0.1-2.0)"
  )
):
  """
  Get device tag data with optional AI forecasting.

  This endpoint fetches time-series data for a specific device tag
  and applies TSAI-based predictions to generate future values.
  """
  try:
    # Validate forecast percentage
    if not validate_forecast_percentage(forecast_percentage):
      raise HTTPException(
        status_code=400,
        detail=get_error_message("invalid_forecast_percentage")
      )

    async with httpx.AsyncClient() as client:
      # Build URL for ASP.NET API
      url = f"{base_url}Device/{device_id}/tag/{tag_id}/data"

      params = {}
      if start_time:
        params["startTime"] = start_time
      if end_time:
        params["endTime"] = end_time

      response = await client.get(url, params=params)

      if response.status_code == 404:
        raise HTTPException(
          status_code=404,
          detail=f"Device {device_id} or Tag {tag_id} not found"
        )
      elif response.status_code != 200:
        raise HTTPException(
          status_code=response.status_code,
          detail="Error fetching device tag data"
        )

      # Get raw data from ASP.NET API
      tag_data = response.json()

      # Transform to format expected by prediction pipeline
      # Input: [{ unixTime: 1234, value: 123.45 }, ...]
      # Transform to: [{ simUnixTime: 1234, value: 123.45 }, ...]
      transformed_data = [
        {
          "simUnixTime": item["unixTime"],
          "value": item["value"]
        }
        for item in tag_data
      ]

      # Add forecast data if requested
      if include_forecast and len(transformed_data) > 0:
        transformed_data = add_forecast_data(
          transformed_data,
          forecast_percentage
        )

      # Transform back to original format
      result_data = [
        {
          "deviceTagId": tag_id,
          "unixTime": item["simUnixTime"],
          "value": item["value"],
          "isForecast": item.get("isForecast", False)
        }
        for item in transformed_data
      ]

      return {
        "device_id": device_id,
        "tag_id": tag_id,
        "include_forecast": include_forecast,
        "data": result_data,
        "forecast_info": {
          "forecast_percentage": forecast_percentage,
          "forecast_points_added": (
            len(result_data) - len(tag_data)
            if include_forecast else 0
          ),
          "description": RESPONSE_MESSAGES["forecast_description"].format(
            percentage=f"{forecast_percentage * 100}%"
          )
        }
      }

  except httpx.RequestError as e:
    raise HTTPException(
      status_code=500,
      detail=get_error_message("error_connecting_api", error=str(e))
    )
  except Exception as e:
    raise HTTPException(
      status_code=500,
      detail=f"Error fetching device tag data: {str(e)}"
    )

3. Update Configuration​

Add to dwd-api-fastapi/config.py:

# Device-specific configuration
DEVICE_CONFIG = {
  "supported_tag_types": [
    "monthly_consumption",
    "quarterly_consumption",
    "yearly_consumption",
    "daily_flow",
    "hourly_flow"
  ],
  "min_data_points": 3,  # Minimum points needed for prediction
  "default_forecast_percentage": 0.5  # 50% forecast by default
}

Backend Implementation (ASP.NET API)​

New Controller Endpoint​

Add to dwd-api-aspnet/DwdApiAspNet/Controllers/DeviceController.cs:

/// <summary>
/// Get time-series data for a specific device tag.
/// </summary>
/// <param name="deviceId">Device ID</param>
/// <param name="tagId">Device Tag ID</param>
/// <param name="startTime">Optional start Unix timestamp</param>
/// <param name="endTime">Optional end Unix timestamp</param>
/// <returns>List of device tag data points</returns>
[HttpGet("{deviceId}/tag/{tagId}/data")]
public async Task<ActionResult<IEnumerable<object>>> GetDeviceTagData(
  int deviceId,
  int tagId,
  [FromQuery] long? startTime,
  [FromQuery] long? endTime)
{
  try
  {
    // Verify device exists
    var deviceExists = await _context.Devices
      .AnyAsync(d => d.Id == deviceId);

    if (!deviceExists)
    {
      return NotFound($"Device with ID {deviceId} not found");
    }

    // Verify tag exists and belongs to device
    var tag = await _context.DeviceTags
      .Where(t => t.Id == tagId && t.DeviceId == deviceId)
      .FirstOrDefaultAsync();

    if (tag == null)
    {
      return NotFound($"Tag with ID {tagId} not found for device {deviceId}");
    }

    // Query tag data
    var query = _context.DeviceTagData
      .Where(d => d.DeviceTagId == tagId);

    // Apply time filters if provided
    if (startTime.HasValue)
    {
      query = query.Where(d => d.UnixTime >= startTime.Value);
    }

    if (endTime.HasValue)
    {
      query = query.Where(d => d.UnixTime <= endTime.Value);
    }

    var data = await query
      .OrderBy(d => d.UnixTime)
      .Select(d => new {
        deviceTagId = d.DeviceTagId,
        unixTime = d.UnixTime,
        value = d.Value
      })
      .ToListAsync();

    return Ok(data);
  }
  catch (Exception ex)
  {
    return StatusCode(500, "Internal server error");
  }
}

/// <summary>
/// Get device tag information by ID.
/// </summary>
[HttpGet("{deviceId}/tag/{tagId}")]
public async Task<ActionResult<object>> GetDeviceTagInfo(
  int deviceId,
  int tagId)
{
  try
  {
    var tag = await _context.DeviceTags
      .Where(t => t.Id == tagId && t.DeviceId == deviceId)
      .Select(t => new {
        id = t.Id,
        name = t.Name,
        tagType = t.TagType,
        unit = t.Unit,
        deviceId = t.DeviceId
      })
      .FirstOrDefaultAsync();

    if (tag == null)
    {
      return NotFound($"Tag with ID {tagId} not found for device {deviceId}");
    }

    return Ok(tag);
  }
  catch (Exception ex)
  {
    return StatusCode(500, "Internal server error");
  }
}

Frontend Implementation (React)​

1. Update DeviceChart Component​

Modify dwd-frontend-react/src/components/DeviceChart.jsx:

import React, { useEffect, useState } from 'react';
import { Box, CircularProgress, Alert } from '@mui/material';
import GqcLineChart from './GqcLineChart';
import { FASTAPI_BASE_URL, API_BASE_URL } from '../constants';

const DeviceChart = ({
  deviceId,
  leftTag,
  rightTag,
  showPredictions = false  // NEW: Accept predictions prop
}) => {
  const [chartSeries, setChartSeries] = useState([]);
  const [yAxis, setYAxis] = useState([]);
  const [loading, setLoading] = useState(false);
  const [error, setError] = useState(null);

  useEffect(() => {
    const fetchDeviceData = async () => {
      if (!deviceId || (!leftTag?.id && !rightTag?.id)) {
        return;
      }

      setLoading(true);
      setError(null);

      try {
        const tasks = [];

        // Left tag data
        if (leftTag?.id) {
          const url = showPredictions
            ? `${FASTAPI_BASE_URL}device/${deviceId}/tag/${leftTag.id}/graph-data`
            : `${API_BASE_URL}Device/${deviceId}/tag/${leftTag.id}/data`;

          const params = new URLSearchParams();
          if (showPredictions) {
            params.append('base_url', API_BASE_URL);
            params.append('include_forecast', 'true');
            params.append('forecast_percentage', '1.0');
          }

          tasks.push(
            fetch(`${url}?${params}`)
              .then(res => res.json())
              .then(data => ({
                tag: leftTag,
                data: showPredictions ? data.data : data,
                position: 'left'
              }))
          );
        }

        // Right tag data
        if (rightTag?.id) {
          const url = showPredictions
            ? `${FASTAPI_BASE_URL}device/${deviceId}/tag/${rightTag.id}/graph-data`
            : `${API_BASE_URL}Device/${deviceId}/tag/${rightTag.id}/data`;

          const params = new URLSearchParams();
          if (showPredictions) {
            params.append('base_url', API_BASE_URL);
            params.append('include_forecast', 'true');
            params.append('forecast_percentage', '1.0');
          }

          tasks.push(
            fetch(`${url}?${params}`)
              .then(res => res.json())
              .then(data => ({
                tag: rightTag,
                data: showPredictions ? data.data : data,
                position: 'right'
              }))
          );
        }

        const results = await Promise.all(tasks);

        // Process results into chart format
        const series = [];
        const axes = [];

        results.forEach((result, index) => {
          const chartData = result.data.map(item => ({
            x: item.unixTime * 1000,  // Convert to milliseconds
            y: item.value
          }));

          // Separate forecast data for styling
          const historicalData = chartData.filter((_, i) =>
            !result.data[i].isForecast
          );
          const forecastData = chartData.filter((_, i) =>
            result.data[i].isForecast
          );

          if (historicalData.length > 0) {
            series.push({
              name: result.tag.name,
              data: historicalData,
              type: 'line'
            });
          }

          if (forecastData.length > 0) {
            series.push({
              name: `${result.tag.name} (Predicted)`,
              data: forecastData,
              type: 'line',
              // Style predictions differently
              dashArray: 5,
              strokeWidth: 2,
              opacity: 0.7
            });
          }

          axes.push({
            seriesName: result.tag.name,
            opposite: result.position === 'right',
            title: {
              text: `${result.tag.name} (${result.tag.unit || ''})`,
              style: { fontSize: '14px' }
            },
            axisTicks: { show: true }
          });
        });

        setChartSeries(series);
        setYAxis(axes);

      } catch (err) {
        console.error('Error fetching device data:', err);
        setError(`Failed to load device data: ${err.message}`);
      } finally {
        setLoading(false);
      }
    };

    fetchDeviceData();
  }, [deviceId, leftTag, rightTag, showPredictions]);  // Re-fetch when predictions toggle

  if (loading) {
    return (
      <Box sx={{ display: 'flex', justifyContent: 'center', p: 4 }}>
        <CircularProgress />
      </Box>
    );
  }

  if (error) {
    return (
      <Alert severity="error" sx={{ m: 2 }}>
        {error}
      </Alert>
    );
  }

  return (
    <Box sx={{ minHeight: '300px', height: '100%' }}>
      <GqcLineChart
        yAxis={yAxis}
        series={chartSeries}
        width="100%"
      />
    </Box>
  );
};

export default DeviceChart;

2. Update EsriCustomPopup​

Modify the Device chart section in dwd-frontend-react/src/components/EsriCustomPopup.jsx:

// Around line 250-280, update the Device chart rendering
{selectedDisplay === 1 && entityType === DEVICE && (
  <Box>
    {/* Device Tag Dropdowns with Prediction Button */}
    <Box sx={{ display: 'flex', gap: 2, alignItems: 'center', mb: 2 }}>
      <DeviceTagDropdown
        deviceTags={getLeftDeviceTagOptions()}
        selectedTag={selectedLeftDeviceTag}
        onChange={handleLeftDeviceTagChange}
        label="Left Axis Tag"
      />
      <DeviceTagDropdown
        deviceTags={getRightDeviceTagOptions()}
        selectedTag={selectedRightDeviceTag}
        onChange={handleRightDeviceTagChange}
        label="Right Axis Tag"
      />

      {/* NEW: Add Prediction Button for Device data */}
      {(selectedLeftDeviceTag?.id || selectedRightDeviceTag?.id) && (
        <PredictionButton
          onClick={handlePredictionToggle}
          showPredictions={showPredictions}
          loading={predictionLoading}
        />
      )}
    </Box>

    {/* Device Chart - now with predictions support */}
    <DeviceChart
      deviceId={entityId}
      leftTag={selectedLeftDeviceTag}
      rightTag={selectedRightDeviceTag}
      showPredictions={showPredictions}  // NEW: Pass predictions state
    />
  </Box>
)}

3. Add FastAPI Endpoints to Constants​

Update dwd-frontend-react/src/constants.jsx:

export const FASTAPI_ENDPOINTS = {
  GetTankGraphData: '/tanks/{tank_id}/graph-data',
  GetNodeGraphData: '/{node_type}/{node_id}/graph-data',
  // NEW: Device predictions endpoint
  GetDeviceTagGraphData: '/device/{device_id}/tag/{tag_id}/graph-data'
};

API Contracts​

ASP.NET API - Get Device Tag Data​

Request:

GET /api/Device/{deviceId}/tag/{tagId}/data?startTime={unix}&endTime={unix}

Response:

[
  {
    "deviceTagId": 123,
    "unixTime": 1698278400,
    "value": 1234.56
  },
  {
    "deviceTagId": 123,
    "unixTime": 1700956800,
    "value": 1150.23
  }
]

FastAPI - Get Device Tag Graph Data with Predictions​

Request:

GET /device/{device_id}/tag/{tag_id}/graph-data?base_url={url}&include_forecast=true&forecast_percentage=1.0

Response:

{
  "device_id": 456,
  "tag_id": 123,
  "include_forecast": true,
  "data": [
    {
      "deviceTagId": 123,
      "unixTime": 1698278400,
      "value": 1234.56,
      "isForecast": false
    },
    {
      "deviceTagId": 123,
      "unixTime": 1700956800,
      "value": 1150.23,
      "isForecast": false
    },
    {
      "deviceTagId": 123,
      "unixTime": 1703635200,
      "value": 1180.45,
      "isForecast": true
    }
  ],
  "forecast_info": {
    "forecast_percentage": 1.0,
    "forecast_points_added": 1,
    "description": "Forecast adds 100% additional data points using TSAI predictions"
  }
}

Implementation Plan​

Phase 1: Backend (ASP.NET API)​

  1. Add new endpoint /api/Device/{deviceId}/tag/{tagId}/data
  2. Add new endpoint /api/Device/{deviceId}/tag/{tagId} for tag metadata
  3. Add unit tests for new endpoints
  4. Test with existing device data

Estimated Effort: 2-3 hours

Phase 2: Backend (FastAPI)​

  1. Add DeviceTagData and DeviceTagInfo Pydantic models
  2. Add /device/{device_id}/tag/{tag_id}/graph-data endpoint
  3. Integrate with existing add_forecast_data() function
  4. Update configuration with device-specific settings
  5. Test predictions with sample device data

Estimated Effort: 3-4 hours

Phase 3: Frontend (React)​

  1. Update DeviceChart.jsx to accept showPredictions prop
  2. Modify chart logic to call FastAPI when predictions enabled
  3. Add visual styling for forecast vs. historical data
  4. Update EsriCustomPopup.jsx to include PredictionButton for device charts
  5. Add new constant for device prediction endpoint
  6. Test UI/UX flow

Estimated Effort: 3-4 hours

Phase 4: Testing & Refinement​

  1. End-to-end testing with real device data
  2. Performance testing with large datasets
  3. UI/UX refinement
  4. Documentation updates

Estimated Effort: 2-3 hours

Total Estimated Effort: 10-14 hours

Testing Strategy​

Unit Tests​

ASP.NET API:

  • Test endpoint returns correct data for valid device/tag
  • Test 404 responses for invalid IDs
  • Test time filtering (startTime, endTime)
  • Test empty data scenarios

FastAPI:

  • Test prediction pipeline with device data
  • Test fallback mechanisms
  • Test error handling
  • Test forecast percentage validation

Integration Tests​

  • Test full flow: Frontend → FastAPI → ASP.NET API → Database
  • Test with different tag types (monthly, quarterly, yearly)
  • Test with different forecast percentages
  • Test with insufficient data scenarios

Performance Tests​

  • Test with large datasets (1000+ data points)
  • Test concurrent requests
  • Test memory usage during predictions

User Acceptance Tests​

  • Test UX flow for toggling predictions on/off
  • Test visual distinction between historical and forecast data
  • Test with different device types and tag configurations
  • Test on mobile devices

Key Considerations​

Data Characteristics​

Device data differs from hydraulic model data:

  • Temporal Granularity: Monthly/Quarterly vs. Hourly/Sub-hourly
  • Data Volume: Much smaller datasets (12-48 points per year)
  • Seasonality: Strong seasonal patterns in consumption
  • Sparse Data: Potential gaps in readings

Implications for Predictions:

  • May need to adjust min_data_points threshold (lower for devices)
  • Consider seasonal decomposition for better predictions
  • May benefit from simpler models (linear regression) for sparse data
  • Should handle missing data gracefully

Prediction Accuracy​

Device consumption data typically has:

  • Strong seasonal trends (summer/winter)
  • Billing cycle patterns (monthly spikes)
  • External factors (weather, holidays, occupancy)

Recommendations:

  • Use longer historical windows for training
  • Consider incorporating external features (temperature, day of week)
  • Provide confidence intervals with predictions
  • Allow users to adjust forecast horizon

Performance Optimization​

For large numbers of devices:

  • Consider caching predictions
  • Implement batch prediction API
  • Use database indexing on UnixTime
  • Limit date ranges for initial requests

Future Enhancements​

  1. Batch Predictions: Predict multiple device tags simultaneously
  2. Anomaly Detection: Identify unusual consumption patterns
  3. Clustering: Group similar consumption patterns
  4. External Features: Incorporate weather, occupancy data
  5. Model Selection: Auto-select best model based on data characteristics
  6. Confidence Intervals: Show prediction uncertainty
  7. Custom Forecast Horizons: Allow users to specify prediction length
  8. Comparison Mode: Compare predictions vs. actual for past periods

Conclusion​

This design extends the existing AI prediction infrastructure to support Device (water meter) time-series data with minimal changes to the existing codebase. The approach:

  • ✅ Reuses existing TSAI prediction pipeline
  • ✅ Maintains consistent API patterns
  • ✅ Follows established UX patterns
  • ✅ Provides clear separation of concerns
  • ✅ Scales to handle device data characteristics

The implementation can be completed in approximately 10-14 hours and will provide valuable forecasting capabilities for water consumption analysis.

Last updated on by Jake, Cursor