EPANET Console Core Monitoring Design Discussion

Current Design Analysis

The current system consists of two main components:

1. EpanetConsoleCore - A .NET console application that runs EPANET simulations and stores results in a database
2. dwd-api-aspnet - An ASP.NET web API application that interacts with the database and triggers EPANET simulations

The main limitation of the current design is that once a simulation is started, there's no way to monitor its progress or status in real-time. This is particularly problematic for long-running simulations that can take tens of minutes to complete.

Important Constraints:

• The system is already under significant database load from bulk inserts of simulation results (2-3 minutes per bulk insert)
• The frontend will only consume progress data via API GET endpoints
• Direct database inserts from EpanetConsoleCore are preferred over API POSTs for performance

Revised Design Options

Option 1: Lightweight Database Monitoring (Recommended)

This approach uses minimal database operations to track progress while avoiding additional load.

Implementation Details:

1. Enhance the existing Scenario table with progress fields:

public class Scenario
{
    // ... existing fields ...

    // Progress tracking
    public string CurrentOperation { get; set; }
    public double ProgressPercentage { get; set; }
    public long LastProgressUpdate { get; set; }
    public string CurrentMetrics { get; set; } // JSON string for flexible metrics
}

2. Modify the console application to update progress:

// In EpanetConsoleCore
private async Task UpdateProgress(string operation, double progress, Dictionary<string, object> metrics)
{
    // Update the scenario record directly
    var scenario = await enContext.Scenarios.FindAsync(scenarioId);
    scenario.CurrentOperation = operation;
    scenario.ProgressPercentage = progress;
    scenario.LastProgressUpdate = DateTimeOffset.UtcNow.ToUnixTimeSeconds();
    scenario.CurrentMetrics = JsonSerializer.Serialize(metrics);

    // Use a lightweight update that doesn't trigger full entity tracking
    await enContext.Database.ExecuteSqlRawAsync(
        @"UPDATE ""Scenarios""
          SET ""CurrentOperation"" = {0},
              ""ProgressPercentage"" = {1},
              ""LastProgressUpdate"" = {2},
              ""CurrentMetrics"" = {3}
          WHERE ""Id"" = {4}",
        operation, progress, scenario.LastProgressUpdate, scenario.CurrentMetrics, scenarioId);
}

3. Add API endpoint for monitoring:

[HttpGet("progress/{id}")]
public async Task<ActionResult<ScenarioProgress>> GetProgress(int id)
{
    var scenario = await _context.Scenarios
        .Where(s => s.Id == id)
        .Select(s => new ScenarioProgress
        {
            CurrentOperation = s.CurrentOperation,
            ProgressPercentage = s.ProgressPercentage,
            LastUpdate = s.LastProgressUpdate,
            Metrics = s.CurrentMetrics
        })
        .FirstOrDefaultAsync();

    if (scenario == null)
        return NotFound();

    return Ok(scenario);
}

Pros:

• Minimal database impact - single table updates
• No additional tables or complex queries
• Works with existing database transactions
• Simple to implement and maintain
• No additional infrastructure needed

Cons:

• Less detailed historical tracking
• Progress updates might be slightly delayed
• Limited metrics storage

Option 2: Memory-Mapped Progress File

This approach uses a memory-mapped file for progress updates, with the API reading from it.

Implementation Details:

1. Create a progress file structure:

public struct SimulationProgress
{
    public int ScenarioId;
    public long Timestamp;
    public double ProgressPercentage;
    public fixed char CurrentOperation[100];
    public fixed char Metrics[1000];
}

2. Console application updates:

// In EpanetConsoleCore
private void UpdateProgress(string operation, double progress, Dictionary<string, object> metrics)
{
    using var mmf = MemoryMappedFile.CreateFromFile(
        $"progress_{scenarioId}.dat",
        FileMode.Create,
        null,
        Marshal.SizeOf<SimulationProgress>());

    using var view = mmf.CreateViewAccessor();
    var progress = new SimulationProgress
    {
        ScenarioId = scenarioId,
        Timestamp = DateTimeOffset.UtcNow.ToUnixTimeSeconds(),
        ProgressPercentage = progress,
        CurrentOperation = operation,
        Metrics = JsonSerializer.Serialize(metrics)
    };

    view.Write(0, ref progress);
}

3. API endpoint:

[HttpGet("progress/{id}")]
public ActionResult<ScenarioProgress> GetProgress(int id)
{
    var filePath = $"progress_{id}.dat";
    if (!System.IO.File.Exists(filePath))
        return NotFound();

    using var mmf = MemoryMappedFile.CreateFromFile(filePath, FileMode.Open);
    using var view = mmf.CreateViewAccessor();
    var progress = new SimulationProgress();
    view.Read(0, out progress);

    return Ok(new ScenarioProgress
    {
        CurrentOperation = new string(progress.CurrentOperation).TrimEnd('\0'),
        ProgressPercentage = progress.ProgressPercentage,
        LastUpdate = progress.Timestamp,
        Metrics = new string(progress.Metrics).TrimEnd('\0')
    });
}

Pros:

• Zero database impact
• Very fast updates
• Real-time progress tracking
• No additional infrastructure

Cons:

• More complex implementation
• Need to handle file cleanup
• No persistence between runs
• Platform-specific considerations

Recommendation

Given the performance constraints and requirements, I recommend implementing Option 1: Lightweight Database Monitoring with the following considerations:

1. Use direct database updates instead of API POSTs
2. Keep progress updates minimal and infrequent (e.g., every 5-10 seconds)
3. Use lightweight SQL updates that don't trigger full entity tracking
4. Store metrics as JSON strings to maintain flexibility
5. Add appropriate indexes to the Scenario table for progress queries

This approach provides a good balance between:

• Minimal database impact
• Simple implementation
• Adequate progress tracking
• Easy frontend integration
• No additional infrastructure

Next Steps

1. Add progress fields to the Scenario table
2. Implement lightweight progress updates in EpanetConsoleCore
3. Add progress monitoring endpoint to the API
4. Add appropriate indexes to the Scenario table
5. Implement progress cleanup on simulation completion
6. Add documentation and testing