Current_Segmentation_Work

OperSight MRI Tumor Segmentation

1. Executive Summary

The segmentation work focuses on binary tumor segmentation for a nasopharyngeal carcinoma MRI dataset. The current workflow uses a clear patient-based NIfTI dataset representation and three independent single-modality segmentation models for CE-T1WI, T1WI, and T2WI. The work covers dataset preparation, patient-level splitting, 2D U-Net training, thresholded evaluation, prediction visualization, cross-modality evaluation, and a simplified prediction-only workflow.

• Current primary modalities: CE_T1WI, T1WI, and T2WI.
• Current model design: one independent 2D U-Net per modality; one MRI channel in and one binary tumor mask out.
• Current prepared dataset: patient-folder based NIfTI structure with each MRI and its corresponding ROI kept separately.
• Current training target size: 512 × 512 for the recent modality-specific training runs.
• Patient splits are patient-level, not slice-level, to prevent slices from the same patient leaking across train/validation/test.
• Evaluation was changed from soft Dice alone to thresholded binary Dice/IoU calculated from actual predicted tumor masks at threshold 0.5.
• A cross-modality 'Any Model, Any Dataset V2' evaluation workflow was designed to compare the latest CE-T1WI, T1WI, and T2WI models against all three datasets.
• A simpler prediction-only workflow was then created so one selected latest model can be run on one selected dataset without training or unnecessary dataset discovery.

2. Dataset and Problem Definition

The project uses a nasopharyngeal carcinoma MRI dataset with multiple MRI modalities and modality-specific ROI masks. The working dataset contains 277 patient IDs. The segmentation task is binary: background versus tumor.

2.1 Current prepared folder structure

Nasopharyngeal_Prepared_NIfTI/
001/
CE_T1WI.nii.gz
T1WI.nii.gz
T2WI.nii.gz
ROI_CE_T1.nii.gz
ROI_T1.nii.gz
ROI_T2.nii.gz
002/
...
...
277/
...

Each MRI modality has its own matching ROI. Do not assume that ROI_CE_T1, ROI_T1, and ROI_T2 are identical voxel arrays merely because they belong to the same patient.

Modality	Image	Ground-truth mask
CE-T1WI	CE_T1WI.nii.gz	ROI_CE_T1.nii.gz
T1WI	T1WI.nii.gz	ROI_T1.nii.gz
T2WI	T2WI.nii.gz	ROI_T2.nii.gz

3. Evolution of the Data Conversion Approach

3.1 Initial conversion and preparation work

Early conversion and preparation work focused on making the MRI image volumes and ROI masks usable for segmentation while verifying orientation and visual alignment. Manual rotation was tested during early debugging when image and ground-truth orientation appeared inconsistent. This was an experimental workaround and is not part of the current conversion standard.

3.2 Current conversion approach

The current three-modality workflow does not use manual rotation. SimpleITK is used to read each DICOM series using DICOM metadata and slice ordering and to write compressed NIfTI. Existing ROI NIfTI files are copied/compressed without combining the masks. This preserves the patient-folder organization and keeps each modality/ROI pairing explicit.

• No manual 90-degree rotation or transpose in the current conversion path.
• SimpleITK carries DICOM size, spacing, origin, direction, and series ordering.
• Existing ROI NIfTI voxel values and affine are preserved when saved as .nii.gz.
• Conversion validation checks image/mask shape and affine within each modality.
• Conversion summaries/failures can be written to the prepared dataset folder.

4. Geometry, Slice Spacing, and Registration Investigation

Geometry was investigated because the same patient ID and slice number did not always show the same tumor ground truth across modalities. For patient 031, CE-T1 and T1 were checked with nibabel.

CE-T1 shape: (512, 512, 34)
T1 shape: (512, 512, 34)

CE-T1 voxel spacing: approximately (0.4688, 0.4688, 6.0) T1 voxel spacing: approximately (0.4688, 0.4688, 6.0)
CE-T1 affine and T1 affine: matching in the inspected example

This showed that, for the inspected patient, the two image volumes share the same matrix size, voxel spacing, and affine/world geometry. However, matching image geometry does not guarantee that separately supplied modality-specific ROI masks occupy the same slice indices. The observed ROI slice shift must therefore be considered when visually comparing modalities.

Current conclusion: do not declare the complete dataset globally 'registered' or 'not registered' from one patient. The inspected CE-T1/T1 image geometry matched for patient 031, while ROI tumor slice occupancy differed. A dataset-wide geometry/annotation audit is still needed before building a truly multimodal model that assumes voxel-for-voxel correspondence.

5. Current Model Architecture

The current modality-specific models are standard 2D U-Nets. Each model receives one grayscale MRI slice and predicts a one-channel sigmoid tumor-probability mask.

• Input: (IMG_SIZE, IMG_SIZE, 1).
• Encoder: repeated Conv2D + BatchNormalization + ReLU blocks with MaxPooling.
• Bottleneck: higher-channel convolution block.
• Decoder: Conv2DTranspose upsampling plus skip connections from the encoder.
• Output: one-channel sigmoid layer named tumor_probability.
• Optimizer: Adam.
• Learning rate used in the recent training configuration: 1e-4.
• Prediction threshold: 0.5.

6. Training Data Preprocessing

6.1 Intensity normalization

MRI volumes are normalized using the 1st and 99th percentile of the non-zero voxel intensities. Values outside that range are clipped, then scaled to 0-1. This is intended to reduce the effect of extreme intensity outliers while retaining useful MRI contrast.

6.2 Slice selection

Training is 2D slice-based. For each patient, tumor-positive slices are identified from the mask. Negative slices are sampled relative to the number of positive slices using NEGATIVE_TO_POSITIVE_RATIO. The random seed includes the patient ID so the selected negative slices are deterministic for that patient.

for z in select_slices(mask, seed=SEED + abs(hash(pid)) % 100000):

This line does not resize or choose a patient. It returns the slice indices that will be used for that already-selected patient.

6.3 Resizing

The generator was revised so it does not resize when the source slice already has the model input size. If resizing is required, MRI images use linear interpolation and binary masks use nearest-neighbor interpolation so mask class labels are not blurred.

im = image[:, :, z]
ma = mask[:, :, z]

if im.shape != (self.img_size, self.img_size): im = cv2.resize(im, (self.img_size, self.img_size), interpolation=cv2.INTER_LINEAR) ma = cv2.resize(ma, (self.img_size, self.img_size), interpolation=cv2.INTER_NEAREST)

The recent modality-specific configuration used IMG_SIZE = 512. The important distinction is that IMG_SIZE is the network input size; it does not prove every original NIfTI modality has a native 512 × 512 in-plane size. If a native dataset differs, the inference workflow may resize it to the model input.

7. Patient Splitting and Reproducibility

The training workflow uses a patient-level split rather than a slice-level split. This is essential: all slices from a patient stay in one partition.

• Random seed: 42.
• Train: 70%.
• Validation: 15%.
• Test: 15%.
• Shared split file: /content/drive/MyDrive/OperSight/Nasopharyngeal_Three_Models/patient_split.json.
• Later training-only notebooks load the existing patient_split.json instead of creating a different split for each modality.

A later naming decision was to save a split JSON alongside model artifacts using the same timestamped model naming convention when needed for model-specific reproducibility. The key requirement is that a prediction/evaluation notebook must know exactly which patients belonged to the original hold-out test set if it is reporting test-set performance.

8. Model and Artifact Naming

Timestamped model names were introduced because multiple training runs existed and simply using 'final.keras' made it difficult to know which run was being evaluated.

timestamp = datetime.datetime.now().strftime("%Y%m%d%H%M")
model_filename = f"{MODALITY}_final{timestamp}.keras"
logfilename = f"{MODALITY}_training_log{timestamp}.csv"
historyfilename = f"{MODALITY}_history{timestamp}.json"

The Any Model, Any Dataset V2 workflow was later designed to select the latest model for each modality by extracting and sorting this timestamp from the filename.

9. Dice and Evaluation Changes

9.1 Why the Dice calculation was changed

The original tumor_dice metric operates directly on probability outputs. That is useful as a training metric, but it is not the clearest final segmentation score because the final output used by the project is a binary mask after applying THRESHOLD = 0.5.

Final evaluation was therefore revised to threshold predictions first, accumulate true-positive, false-positive, false-negative, and true-negative pixel counts across the test set, and calculate Dice from the resulting binary segmentation.

Dice = (2 × TP) / (2 × TP + FP + FN)

IoU = TP / (TP + FP + FN)
Precision = TP / (TP + FP)
Recall = TP / (TP + FN)

An observed CE-T1 evaluation example produced thresholded tumor Dice ≈ 0.600659 and thresholded tumor IoU ≈ 0.429244 at threshold 0.5. These values were used as a sanity check that the new evaluation was measuring tumor overlap after thresholding.

9.2 Float datatype issue encountered

A TensorFlow error occurred after changing TensorFlow computation to float64: Binary Cross-Entropy was float32 while the custom Dice branch returned float64, so AddV2 could not add the two tensors. The fix was to ensure both loss components use compatible dtypes. This is a dtype consistency issue, not a model-architecture issue.

10. Prediction Visualization

The prediction display was simplified to focus on interpretable segmentation outputs. The probability panel was removed from the standard five-panel review and replaced with both ground-truth and prediction overlays.

Image | Ground truth | Prediction | GT Overlay | Pred Overlay

Both overlays use the same autumn colormap and alpha blending so the ground truth and model output can be compared visually on the same grayscale MRI.

Earlier show_prediction(test_ids[0]) always displayed the first test patient, which made repeated testing look like the notebook was selecting the same image. It was later changed to iterate over all required patients or to select best/worst cases based on Dice.

11. Cross-Modality Evaluation: Any Model, Any Dataset V2

A broader evaluation workflow was designed to answer a different research question: how does a model trained on one MRI modality perform when it is applied to another modality?

• Use the latest timestamped CE_T1WI model.
• Use the latest timestamped T1WI model.
• Use the latest timestamped T2WI model.
• Evaluate each model on CE_T1WI, T1WI, and T2WI datasets.
• This produces up to 3 × 3 = 9 model-dataset runs.
• The cross-modality workflow removes the original test-patient restriction when the goal is dataset-wide exploratory comparison.
• Each run stores dataset-level metrics, patient-level metrics, predictions, and representative best/worst examples.

11.1 Compatibility table

The workflow includes model-input/native-dataset size checks. A row can be 'compatible = True' while 'native_size_matches_model = False'. That means inference is possible, but the dataset slice must be resized to the network's required input shape. This does not mean the model itself was trained at the native dataset size.

11.2 Main run controller

The main loop iterates over compatible model-dataset combinations and calls evaluate_model_on_dataset(run_row). That function performs the actual inference/evaluation and returns a run summary, a per-patient metrics table, and the run output folder. Failures are caught so one failed combination does not stop the remaining runs.

At the end, all run summaries are combined into all_run_metrics.csv and sorted by Dice for comparison.

11.3 Predictions and montages

Prediction NIfTI files are saved inside each run's predictions folder. Representative best and worst cases are stored as montage images and can be displayed later without rerunning inference. The intended montage format is Image, Ground truth, Prediction, GT Overlay, Pred Overlay.

12. Prediction-Only Notebook

After the larger V2 workflow, a cleaner operational notebook was created for the common use case: run one selected trained model on one selected MRI dataset. This notebook is intentionally simple and sequential.

1. Select SELECTED_MODEL_MODALITY and SELECTED_DATASET_MODALITY.
2. Use fixed DATASET_CONFIG mappings for image and mask filenames; do not rediscover modality from files.
3. Find the latest timestamped .keras model only within the selected model modality folder.
4. Copy only the selected image and mask files to Colab local storage.
5. Load the selected model with compile=False because this is inference only.
6. Normalize the selected MRI volume using the established percentile normalization.
7. Run slice-wise prediction across the complete selected volume.
8. Threshold probabilities at 0.5.
9. Save a prediction NIfTI for each patient using the original image affine/header geometry.
10. Save patient Dice values to CSV.
11. Show the best and worst prediction examples.

Current notebook artifact created in this work: Prediction_Only_One_Model_One_Dataset.ipynb.

13. Google Drive / Colab Working Paths

Purpose	Path
Prepared dataset on Drive	/content/drive/MyDrive/OperSight/Nasopharyngeal_Prepared_NIfTI
Local Colab copy	/content/Nasopharyngeal_Prepared_NIfTI
Model root	/content/drive/MyDrive/OperSight/Nasopharyngeal_Three_Models
Shared split	/content/drive/MyDrive/OperSight/Nasopharyngeal_Three_Models/patient_split.json
Prediction-only results	/content/drive/MyDrive/OperSight/Prediction_Only_Results

15. Grayscale / Precision Discussion

A separate investigation discussed whether the MRI/NIfTI processing path was effectively using 8-bit grayscale and whether higher precision would preserve more subtle intensity differences. The important distinction is:

• Bit depth/intensity precision controls how finely voxel intensity values can be represented.
• Spatial resolution controls how many pixels/voxels describe the tumor.
• Changing from 8-bit to float32 does not create new spatial detail or remove pixelation caused by enlarging a small region.
• For ML preprocessing, preserving source intensity information in float32 and normalizing it is preferable to quantizing to uint8 unless there is a deliberate reason to do so.
• For visualization, window/contrast mapping is separate from the datatype used internally.

This topic should be verified against the exact conversion/preprocessing notebook before claiming that the stored NIfTI files themselves are 8-bit. Loading a NIfTI with get_fdata(dtype=np.float32) changes the in-memory representation; it does not prove the source NIfTI was originally 32-bit.

16. Segmentation Integration Notes

Detailed model-development and evaluation notes already exist in the project documentation and should remain the primary source of truth for that work.

17. 3D Segmentation Visualization

A near-term next step is displaying segmentation results directly in the OperSight 3D viewer.

17.1 Main Visualization Options

Semi-Transparent Overlay

Display the segmentation as a translucent colored volume over the MRI.

This is the recommended first test.

Surface Mesh

Convert the segmentation mask into a smooth 3D surface over the MRI.

This is a common segmentation-visualization approach.

Mask-Guided Volume Rendering

Use the mask to tint or emphasize the corresponding MRI region while preserving the underlying MRI texture.

17.2 Additional Viewing Controls

Boundary Highlight

Emphasize the edge of the segmented area.

Isolation Mode

Retain the segmentation while fading the rest of the MRI.

Clipping View

Use a movable plane to expose the segmentation where it intersects the volume.

Confidence Map

Use visualization intensity/color strength to represent model confidence when probability output is available.

17.3 Recommended Implementation Sequence

Start by testing:

1. Semi-transparent overlay
2. Surface-mesh representation

Compare the two approaches before deciding on the final visualization strategy.

18. AI / MCP Segmentation Workflow

AI chat and MCP integration are already implemented.

A useful proof of concept for segmentation is:

1. The user provides a medical image and asks a question about a structure or abnormality.
2. The image/question are sent through the existing AI workflow.
3. The LLM invokes an MCP segmentation tool.
4. The tool runs a Python/PyScript segmentation step.
5. A segmentation result is returned to the LLM.
6. The LLM interprets the result and responds in the OperSight UI.

18.1 Initial Proof of Concept

The first version can use:

• A known image
• A dummy or pre-generated segmentation result

The goal is to validate the end-to-end tool pipeline before integrating a real model.

18.2 Potential Output Formats

• Annotated/overlay image
• Binary mask
• Coordinates / JSON
• NIfTI

An annotated image may be the simplest format for the first LLM-based proof of concept.

A raw NIfTI output would require additional visualization and processing.