Skip to main content

Azure Blob Storage - Design & Implementation

Version: 1.0 Date: October 20-21, 2025 Status: ✅ Implemented and Complete Feature: EPANET Input File Cloud Storage

Table of Contents​

  1. Overview
  2. Architecture
  3. Implementation Summary
  4. Storage Services
  5. Frontend Integration
  6. Security
  7. Future Enhancements

1. Overview​

1.1 Purpose​

Migrate EPANET input file storage from local file system to Azure Blob Storage for:

  • Improved scalability
  • Centralized management
  • Cloud-native architecture
  • Multi-environment file sharing

1.2 Migration Complete ✅​

Before:

  • Scenarios stored input file paths as strings (InputFile column)
  • Files stored on local file system only
  • Each EpanetConsoleCore instance needed local file access
  • No centralized file management

After:

  • Input files stored in Azure Blob Storage (with local fallback)
  • InputFile as a proper database entity with FK relationships
  • EpanetConsoleCore downloads files to temp directories on demand
  • Centralized file management through API
  • Full CRUD operations via frontend

1.3 Key Design Decisions​

✅ No File Validation on Upload

  • Files stored as-is without parsing or validation
  • EPANET validates during simulation runs
  • Simplifies upload process

✅ Container-per-Network Architecture

  • Each network/client gets its own blob container
  • Scoped access using SAS tokens (not account keys)
  • Better security isolation and access control

✅ Backward Compatibility

  • Supports legacy InputFile string paths during migration
  • Graceful fallback to local file system

2. Architecture​

2.1 System Overview​

┌─────────────────────────────────────────────────────────────┐
│                     Frontend (React)                         │
│  - Upload INP files                                         │
│  - Manage input files                                       │
│  - Assign files to scenarios                                │
└──────────────────────┬──────────────────────────────────────┘
                       │ HTTPS
┌──────────────────────▼──────────────────────────────────────┐
│                   API (ASP.NET Core)                         │
│  - InputFileController (upload, list, download)             │
│  - ScenarioController (modified to use InputFileId)         │
│  - IInputFileStorageService (abstraction)                   │
└──────────────────────┬──────────────────────────────────────┘
                       │
       ┌───────────────┴───────────────┐
       │                               │
┌──────▼───────┐               ┌───────▼────────┐
│   Azure      │               │  Local File    │
│   Blob       │               │  System        │
│   Storage    │               │  (Dev/Test)    │
└──────┬───────┘               └───────┬────────┘
       │                               │
       └───────────────┬───────────────┘
                       │ SAS Token (scoped)
┌──────────────────────▼──────────────────────────────────────┐
│            EpanetConsoleCore (Simulation)                    │
│  - Downloads files to temp directories                      │
│  - Automatic cleanup after simulation                       │
│  - Uses container-scoped SAS tokens (read-only)            │
└─────────────────────────────────────────────────────────────┘

2.2 Database Schema​

InputFile Entity:

public class InputFile
{
    public int Id { get; set; }
    public string Name { get; set; }           // User-friendly name
    public string Description { get; set; }
    public string ContainerName { get; set; }  // Azure container
    public string BlobName { get; set; }       // File identifier
    public string FilePath { get; set; }       // Local fallback
    public long FileSize { get; set; }
    public StorageProvider StorageProvider { get; set; }  // AzureBlob | LocalFileSystem
    public bool IsActive { get; set; }
    public DateTime UploadedDate { get; set; }

    // Navigation
    public virtual ICollection<Scenario> Scenarios { get; set; }
}

public enum StorageProvider
{
    AzureBlob = 0,
    LocalFileSystem = 1
}

Scenario Entity (Modified):

public class Scenario
{
    // ...existing properties...

    // New FK relationship
    public int? BaseInputFileId { get; set; }
    public virtual InputFile BaseInputFile { get; set; }

    // Legacy support (kept for backward compatibility)
    public string InputFile { get; set; }
}

2.3 Storage Provider Interface​

public interface IInputFileStorageService
{
    Task<string> UploadFileAsync(string containerName, string blobName,
        Stream fileStream, CancellationToken cancellationToken = default);

    Task<Stream> DownloadFileAsync(string containerName, string blobName,
        CancellationToken cancellationToken = default);

    Task<string> DownloadToTempFileAsync(string containerName, string blobName,
        CancellationToken cancellationToken = default);

    Task<bool> DeleteFileAsync(string containerName, string blobName,
        CancellationToken cancellationToken = default);

    Task<bool> ExistsAsync(string containerName, string blobName,
        CancellationToken cancellationToken = default);

    Task<long> GetFileSizeAsync(string containerName, string blobName,
        CancellationToken cancellationToken = default);
}

3. Implementation Summary​

Phase 1: Database Schema & Models ✅​

Files Created:

  1. efcorelibraries/HTModels/Models/EN2Models/InputFile.cs
  2. Modified Scenario.cs with new FK relationship
  3. Updated EN2PostgresContext.cs with DbSet and relationships

Phase 2: Storage Services ✅​

Shared Services Location: efcorelibraries/HTModels/Services/

Files:

  1. IInputFileStorageService.cs - Interface
  2. AzureBlobStorageService.cs - Azure implementation
    • Supports both account key (API) and SAS token (Console) modes
    • Auto-detects mode based on configuration
  3. LocalFileStorageService.cs - Local file system implementation
    • Supports legacy paths for migration

Service Registration:

  • API: Registered in Program.cs based on configuration
  • Console: Configured via appsettings.json

Phase 3: API Layer ✅​

New Controller: InputFileController.cs

Endpoints:

  • POST /api/inputfile/upload - Upload file with metadata
  • GET /api/inputfile - List all input files
  • GET /api/inputfile/{id} - Get single file metadata
  • GET /api/inputfile/{id}/download - Download file
  • PUT /api/inputfile/{id} - Update metadata
  • DELETE /api/inputfile/{id} - Soft delete file

Modified Controller: ScenarioController.cs

  • Updated to support BaseInputFileId FK
  • Maintains backward compatibility with legacy InputFile paths

Phase 4: EpanetConsoleCore Integration ✅​

Modified Files:

  1. Program.cs - Integrated storage service
  2. Simulation workflow updated to:
    • Download files to temp directory when using blob storage
    • Use local paths for legacy scenarios
    • Automatic temp file cleanup after simulation

Key Features:

  • Uses container-scoped SAS tokens (read-only)
  • No account keys stored in console configuration
  • Automatic temp directory management
  • Fallback to local file system if configured

4. Storage Services Consolidation​

Refactoring Completed ✅​

Problem: Originally duplicated storage service code between API and Console projects (~600 lines duplicated).

Solution: Consolidated into shared location (efcorelibraries/HTModels/Services/)

Benefits:

  • ✅ Single source of truth
  • ✅ No code duplication
  • ✅ Fix once, benefits all projects
  • ✅ Consistent behavior
  • ✅ Easier testing and maintenance

Service Implementation Details​

AzureBlobStorageService:

  • Supports two authentication modes:
    1. Account Key Mode (API): Full access for upload/management
    2. SAS Token Mode (Console): Scoped read-only access
  • Auto-detects mode based on configuration presence
  • Container-level isolation for security

LocalFileStorageService:

  • Development and testing support
  • Supports both legacy paths and new structure
  • Maintains folder hierarchy
  • Automatic directory creation

5. Frontend Integration​

Input Files Management Page ✅​

Location: /input-files

Features Implemented:

  1. Upload Dialog - File upload with drag-and-drop

    • Name, description, container selection
    • Upload progress indicator
    • File size validation
    • .inp file type filtering

  2. File Table View - Complete file listing

    • Columns: Name, Description, Container, Storage Provider, Size, Date, Status
    • Actions: Download, Edit, Delete
    • Storage provider badges (Azure Blob / Local)
    • Active/Inactive status chips

  3. Edit Dialog - Update file metadata

    • Edit name, description, active status
    • Read-only: Container, blob name, storage provider

  4. Delete Confirmation - Safe deletion with warnings

Scenario Integration ✅​

Scenario Create/Edit Dialog:

  • Dropdown to select Base Input File
  • Displays: File name, container, size
  • Link to manage files
  • Backward compatibility with legacy paths
  • Migration helper for old scenarios

Navigation ✅​

  • Added "Input Files" to sidebar menu
  • Route configured and protected
  • Folder icon for menu item

API Service Layer ✅​

File: src/services/inputFileService.js

Functions:

  • uploadInputFile() - With progress tracking
  • getInputFiles() - List all files
  • getInputFile(id) - Get single file
  • downloadInputFile(id) - Download file
  • updateInputFile(id) - Update metadata
  • deleteInputFile(id) - Delete file

6. Security​

Security Model​

API Tier:

  • Uses account-level keys for upload/management operations
  • Keys stored in appsettings (use Azure Key Vault in production)
  • Full CRUD access to blob storage

EpanetConsoleCore:

  • Uses container-scoped SAS tokens (read-only)
  • Tokens generated by API with expiration
  • No account keys in console configuration
  • Principle of least privilege

Container Isolation​

  • Each network/client gets dedicated container
  • Scoped access prevents cross-container access
  • SAS tokens limit permissions and duration

Best Practices​

✅ Implemented:

  • Container-scoped access
  • Read-only tokens for simulation engine
  • Automatic token expiration
  • Soft delete for audit trail

💡 Recommended for Production:

  • Use Azure Key Vault for account keys
  • Enable Managed Identity authentication
  • Implement SAS token rotation
  • Enable blob storage logging and monitoring

7. Future Enhancements​

NetworkManager Integration 💡​

Status: Idea - Not Implemented Priority: LOW

Concept:

  • Upload source INP file when importing networks
  • Create audit trail of network imports
  • Enable network versioning
  • Re-import from stored files

Use Cases:

  1. Network import audit trail
  2. Re-import from previously stored files
  3. Network versioning and rollback capability

Architecture:

Local INP file
  ↓
NetworkManager parses
  ↓
Database updated (Nodes, Links, etc.)
  ↓
Upload INP file to blob storage
  ↓
Create InputFile + NetworkImport records
  ↓
Preserve original file for audit/rollback

Other Potential Enhancements​

  1. File Versioning - Multiple versions of same file
  2. Bulk Upload - Upload multiple files at once
  3. File Templates - Library of template INP files
  4. File Comparison - Compare two INP files
  5. File Validation - Optional pre-validation before storage
  6. CDN Integration - Faster downloads via CDN
  7. File Compression - Compress large files for storage

Summary​

✅ Complete Implementation:

  • Database schema with InputFile entity
  • Dual storage providers (Azure Blob + Local)
  • API layer with full CRUD operations
  • Frontend file management interface
  • EpanetConsoleCore integration
  • Security with scoped SAS tokens
  • Service consolidation eliminating code duplication

✅ Production Ready:

  • Backward compatible with legacy paths
  • Automatic temp file cleanup
  • Error handling and logging
  • Configurable via appsettings

✅ Well Documented:

  • Configuration guides available
  • API documentation in Swagger
  • Frontend component documentation
  • Security best practices documented

Next Steps:

  • Deploy to test environment
  • Migrate existing scenarios to use InputFile entities
  • Monitor performance and adjust as needed
  • Consider future enhancements based on usage patterns

Related Documentation:

  • Configuration Guide: configuration.md
  • API Reference: Available in Swagger UI
  • Database Migration: See efcorelibraries documentation
Last updated on by Jake, Cursor