Skip to main content

Questions and Answers

11/14/2023​

  1. Here are few more questions on WRc videos.

    1. Did you use any of WRc videos on the random tree-based defect pipe classification project?
      Vannary: I used only the condition data and header provided by Sean. Some of them do not have corresponding videos. Yes, the condition data has both PACP and WRc codes. I can separate the PACP and WRc condition reports by looking at the code that they use to enter the water level. 'MWL' is used in PACP and 'WL' is used in WRc.

    2. Has anything been done with 1650 WRc Videos dataset (in CCTV training, prediction or in the GIS)?
      Vannary: I only used those videos to check on WRc codes that were not found in the list of WRc codes provided by Sean. I have not used them in CCTV training, prediction, or in the GIS.

    3. For DNV WRc 1066​ dataset, I found the following blurred frames under CCTV/DNV/Data/Blurred_Frames/​ in VS_Research​ folder. Are these the latest and correct blurred frames for WRc 1066 ? wrc videos
      Vannary: Yes. I have created the list of videos for WRc 1066 videos, extracted every 30th frame, and created blurred frames. I have not done anything else
      Vannary: Clarification:
      1 and 2. I have used the condition data and headers provided by Sean for CCTV defect prediction. However, only some of the condition reports provided in the condition data and header have their corresponding video files.

  2. For PACP codes, does the following group-2 labels capture all Root defects group (similar to I&I) ? If you have a document that explains the grouping method, please send that too.

    • 'RF': 'R', # Root
    • 'RM': 'R',
    • 'RT': 'R',
    • 'RB': 'R'

    Vannary: Please see PACP_code_compressed_vs_detailed for information on how I group the defects. Refer to the 'PACP 1st tier Group Code/ Use In Paper 2)' column for the grouped labels I used for Paper 2 and the 'compressed label (use in paper 1)' column for the grouped labels I used for Paper 1. Yes, RF, RM, RB, and RT are grouped into the R group

  3. How 'I&I'​ and 'Roots' groups can be defined for WRc labels? I remember you mentioning that group-2 or compressed-labels were not defined for WRc.

    See list_unique_wrc_code_v2

  4. When you said "I only used those videos to check on WRc codes that were not found in the list of WRc codes provided by Sean. I have not used them in CCTV training, prediction, or in the GIS" as an answer to my initial question #2, what did you mean by "WRc codes provided by Sean"? and what is the "check" you did with the WRc 1650 dataset?
    Vannary:

    • Sean has uploaded file name “WRC codes.pdf” which contains the list of WRC codes. (Please see email that you send to me on May 10 with email subject: ‘New DNV databases’.
    • Location of WRC codes.pdf
    • I generated the list of unique WRc (see the csv file that I sent to you in the last email) does found in the condition data and used the WRC codes.pdf to get the description of the code. For codes that were not found the pdf file due to error or the different version, I used their videos to get the description of the code.

  5. Can you clarify further on what you meant by "1 and 2. I have used the condition data and headers provided by Sean for CCTV defect prediction. However, only some of the condition reports provided in the condition data and header have their corresponding video files."
    Vannary: See point 1.

  6. Did you do anything with the dataset "1811 Videos not found in CCTV Details" (which is listed as a dataset under DNV in the project summary excel sheet)?
    Vannary: No I did not do anything with it.

10/05/2023​

  1. Pavan: In a previous reply, you mentioned that "Also, you might want to consider removing some of the ND images, as only 7% of total images are defective/feature images. " to improve the results.
    How many frames or what percentage of defective are ideal for the training process? Did you do have to drop ND frames for this same reason in the previous datasets as well? If yes, how the frames to drop were selected, was it random selection?

Vannary: Ideally, you would want 50% or more defect/feature frames. I would recommend no smaller than 30%. No, I did not have this problem with any of the other three utilities. For DNV, ND is the class with the most images. However, when you combined all other defect/feature images together, the defect/feature group made up more than 50%. One time, I used random selection to drop frames with ND to match the number of ND images to the class with the second most images.

10/04/2023​

  1. Pavan: From my understanding those metrics we get out of this do not contain something we can directly compare with the performance of single-stage model (ND + defect label classifier). I might have missed something though. How do we compare the results of this two-stage architecture with single-stage architecture?

Vannary:

  1. F1 normal (ND) from the stage 1 model will be used to compare to F1 normal (ND) from the single-stage model.
  2. Average F1/F2 scores of defective/feature labels from the stage 2 model will be used to compare to the Average F1/F2 scores of defective/feature labels from the single-stage model. You can also compare the individual f1/f2 score of each label.
  3. I concatenate the prediction results from the stage 1 model (frames predicted to be ND) with the stage 2 model (frames predicted to be defective with their type of defect) before calculating f1 and f2 scores against the ground truth labels.

10/03/2023​

  1. Pavan: I get this repeated warning in the second-stage model training for SD1-D dataset: /home/gqc/mambaforge/envs/compute-msi/lib/python3.10/site-packages/sklearn/metrics/_classification.py:1757: UndefinedMetricWarning: F-score is ill-defined and being set to 0.0 in labels with no true nor predicted samples. Use zero_division parameter to control this behavior.

    This can be cleared with the parameter mentioned in the 'warning' itself. Just wanted to clarify whether it is okay to clear it, or can this be an indication of something I'm doing something wrong in the training/validation data generation part itself.

    Vannary: Yes, it is okay to clear the warning regarding F-scores.

  2. Pavan: this is the result of running cctv-multi-label-two-stage-approach_GQC_metric over the first and second stage models I trained for 5 epochs. Can you take a quick look and maybe tell us what are the important parameters to look for and where we get the overall accuracy metrics?

    Vannary: You should look at the Stage 1 F1/F2 test scores to get F-scores of the binary model (stage 1 model), where 0 stands for no defect and 1 stands for defect/feature. You should look at your stage 1 model as the f1/f2 scores of label = 1 are significantly lower than the other 3 utilities, even when you compare them with the VB models. Based on the true positive and false positive values, 3207 images, of which 2783 images were ND images, were passed to the stage 2 model and 6141 defective/feature images were labelled as ND by the stage 1 model.

    vannary image 1

    The overall metric of the model can be found:

    1. F1/F2 test average no ND (common_label only): the average f1/f2 scores of the common labels between the model and test labels, excluding the ND label. (note this score is calculated after running the stage 2 model.)
    2. F1/F2 test average no ND (intersection labels, GQC): the average f1/f2 scores of common labels between the model output predicted labels and the test labels, excluding ND. (note this score is calculated after running the stage 2 model.)
    3. Stage 1 F1 test score for 0 (see the previous image) for F1 score of ND label. (note this score is calculated after running the stage 1 model.)

    vannary image 2

    and you might want to consider removing some of the ND images, as only 7% of total images are defective/feature images.

09/27/2023​

Weighted loss​

This is a mechanism in model training that is used to balance out the impact of unbalanced datasets by assigning weights to the labels with larger weights for less frequent labels and smaller weights for frequent labels.

  1. Pavan: For which models, this technique is applied to?
    Vannary: It is used in the one-stage model and the second model in two-stage model. To run the weighted loss function, you will have to set settings.WEIGTHED_LOSS to true. This only works when you run on 1 GPU. I only ran this model configuration on DNV data.

  2. Pavan: How are the weights assigned for the labels? Vannary: The weights are calcualted based on the whole dataset. Find the notebook here. Please ignore the other sections. They are outdated. code snippet

Two stage modeling​

  1. Pavan: Can you point me to a WandB log of your best binary models on CCTV training on SD1 (or another utility if SD1 is not available)
    Vannary: Link to wandb: https://wandb.ai/lence_ubc/cctv-dnv-singlelabel/runs/ra2kc452/overview?workspace=user-sengv1

  2. Pavan: What are the key indicators you use to compare results?
    Vannary: I would use both F1 and F2 scores of '1' or 'positive' (defect) label and F1 and F2 scores of '0' or 'negative` (ND) label. However, when I compare the model performance, I viewed the binary models + defective classification model as one combination of a two-stage model. Because I want to compare the performance between three backbones (MobileNet, ResNet50, and ResNet101), I only use first-stage and second-stage models from the same backbones. I only evaluate the model performance after running the two-stage model approach prediction Python script. The key indicators I used to compare the results are average F1, F2 scores (exclude Non-defect label) and F1 of Non-defect label.

  3. Pavan: What are the hyper-parameters you considered when optimizing the model architecture?
    Vannary: Aside from looking at different backbone types, I focus on the different numbers of epochs/freezing epochs, class weight, and data preprocessing, such as various image augmentation methods (flip, rotate, etc...)

09/12/2023​

Pavan: Workflow of the entire CCTV defect classification process
Vannary:defect classification

Pavan: Questions regarding two-stage modeling approach.
Vannary:

  1. In compute-canada, top of each .py file contains the description what each script does.
  2. To train two stage approach models, you need to use two scripts
    1. ND vs Defect model (binary model) nd vs defect
    2. Defect multi-labels model (without ND). Use the same script that you use for one-stage model but change NO_ND = True in the settings.py settings py for nd
  3. The same csv file used for single-stage model can be used.
  4. In Weights and Bias project following convention is used to host the model logs.
    1. ND vs D's project name: cctv-<utility>-singlelabel
    2. No ND defective model's project name: cctv-<utility>-defect-multilabel
  5. Run cctv-multi-label-two-stage-approach_GQC_metric.py to run the prediction script. You will need to change STAGE1_MODEL to the ND vs D that you want to use in the settings.py. The second model is entered when submitting the .sh file using -m tag.
  6. The difference between cctv-multi-label-two-stage-approach_GQC_metric.py and cctv-multi-label-two-stage-approach.py is:
    1. the first file will output the averaged F1 and F2 scores of the common labels found in the model output predicted labels and the test data (known as GQC metric).
    2. GQC metric F1 and F2 scores found in output txt files:
      1. f1 and f2 metric
      2. overlap of labels
      3. Both of them will output the averaged F1 and F2 scores of the common labels found in the train and test data. overlap
      4. Averaged F1 and F2 scores of the common labels can be found at the end of the output .txt file. averaged f1 and f2

09/11/2023​

Pavan: Can you confirm if this workflow is correct? And please add any missing parts.

  1. Obtain the output from 'image_label_creation' notebook which contains label information for all frames.

  2. Pass the output from #1 to this this notebook,

    1. Use split_data_into_train_and_test​ or split_data_into_train_and_test_VB​ to separate training+validation set and test set.
    2. Then use k_fold_split​ function to separate the train and validation set. (Provides me with 4 different combinations)
    3. Use one of the outputs from 2(b) to train a model.
    4. Use the output for test​ set from 2(a) for testing the model.

    Vannary: Yes, that is correct. But for VB, you will need to create a list of videos you would like to exclude from training and use for testing before doing step 2.1.

Pavan:

  1. Is there a criterion you use in picking which videos to be used for testing in VB method? or is it random?
    Vannary: You can pick the videos according to pipe materials or do it randomly. The important part is to ensure that the sum of the test number is around 20% of the total number of images. Here is an example that I did for COV: count the frames
  2. Out of the 4 training csv files we get from 2(b), do you use all of them to launch separate training runs? or if you pick just one, how do you do that?
    Vannary: Initially, I trained all four and then averaged their F1 and F2 scores for cross-validation. However, Sudhir and Deven suggested choosing one because it was computation-intensive, and the results of the four runs were similar. You should do a k=4 cross-validation run on the new dataset to see if there is a significant discrepancy in the F1 and F2 scores.

09/01/2023​

Email conversations with Vannary​

Pavan: Can you help me in identifying the corresponding test CSV for this selected Train+Validation CSV:https://drive.google.com/file/d/10kh2SZ9225_rVkEuUEuJXFAGHxvVWNi-/view?usp=drive_link in SD1_114 dataset? Is there a convention you used in file naming to identify the test CSV's used with each Train+Validation CSVs?

Vannary:

SD1 114 (71 sanitary video) label csv: https://drive.google.com/drive/folders/1-fkM7aWLXeTaH5hNlzLGsRQvaRLg3ZIG

  1. Train+Validation CSV file is the file with the name containing "train_val." The numbers 0 to 3 correspond to the set of the cross-validation splits. I only did k=4 split.
  2. These are the four files for train+validation CSV for 71 sanitary videos only. The file with 114 videos contains labels from the stormwater pipes as well. van email
  3. Test CSV file is the file with a name containing "test". test email
  4. The file with a name containing "simplified" or "test_S" or "train_val_S" is group labels csv. This file was created from the old grouping method. You will need to run prepare_cctv_training_csv notebook for the new grouping.
  5. The video-based dataset is located in the "set aside test videos" folder.
  6. In addition, these names are the old convention I used to identify files. Here are the new naming conventions that I used for the other three utilities:

train+validation csv: train_val + utility + number of video + SS_CB (for sanitary and combined pipes) + FB or VB (frame-based or video-based) + version + set of cross-validation split example: train_val_cov_74_SS_CB_FB_v2_0.csv test csv: test + utility + number of videos + SS_CB + FB or VB + version example: test_cov_74_SS_CB_FB_v2.csv

Pavan: We tested fastai_multi_label_v3.py script to run a dataset and it tried to read all the image files inside train​ folder for a prediction run. Seemed to be functionally wrong unless we miss something here. pavan

Also, we did a comparison between fastai_multi_label_v3.py and fastai_multi_label_v3_multi_gpus.py. There seems to be functional differences which are independent of multi gpu related changes. Do you think that is possible? pavan

Multi GPU version does not have 'prediction on test dataset' section and there seemed to be differences in the functionality.

Let me know what are the final set of scripts you have used to make the runs. Even if there are multi-gpu scripts, I can try to modify them to work for our single gpu scenario.

Vannary:

Initially, we have prediction script inside the training script. However, we have to separate them into two scripts when using multi-GPU because of GPU out of memory error.

The old prediction script will predict the all the images inside the given directory and then it will output only frame with matching frame name in the test.csv.

I modified the new prediction script to either load all images from directory or only selected images listed in the csv file:

vannary

RE pretrained variable:

We have to run multi-GPUs on the Narval cluster which does not have access to the internet. Therefore, in order to load the backbone models such as resnet50, we will have to download to the local directory and use the custom_loaded_model() function created by Deven to load the model.

If pretrained=True inside vision_learner, it will try to load the backbone model from the internet. Therefore, we need to set pretrained = False when running on the Narval Cluster.

We set pretrained=True in the Cedar cluster because it has an internet connection, and the backbone model can be downloaded from the internet without having to download and store it in the local directory.

vannary

Link to issue regarding pretrained=False for the naval cluster: link

Pavan:

Few questions about 'prepare_cctv_training_csv.ipynb - Colaboratory (google.com)' Is the only function you use to create the k fold split, k_fold_split? How do you verify the output csv from this notebook has a valid combination of training and validation data?

Vannary:

There is a function called k_fold_split. Note that the test refers to validation data. Inside the function, there are lines of code to check if all labels in the validation data are found in the train data. If it is not found in the train data, the function will create a duplicate row of an image containing the label and assign it to the train dataset.

vannary answer

08/31/2023​

The notes below comes form a meeting between Pavan, Vannary, Sudhir and Srujana on training utility data like SD1-D after labelling (which is the last step of data pre-processing). This has been communicated through email too.

  1. All scripts have docstring on the top, which we should be scraping out for documentation site. Should be an action item on us.
  2. We should be using the latest versions of the training scripts (v3 if we have v1, v2 and v3). And for MSI we should be using the scripts without multi_gpus in the name. For example, we should be using the last script from this group.

    example-list-of-python-scripts

  3. V3​ scripts require the input csv to be of this format.
    1. It is created through Vannary's notebook: prepare_cctv_training_csv.ipynb
    2. An example output csv: https://drive.google.com/file/d/105omTROiYkMLHCiY-uaAgQgLvgRWS52o/view?usp=sharing

      training-input-data-snippet

  4. Scripts used for prediction will contain the name prediction in the name.
    • For e.g. fatsai_multi_label_prediction_with_labels_GQC_metric.py
  5. Vannary will share us a snapshot of the pip modules she has in the compute-canada virtual env.(She sent in the email)
  6. The Columns of interest for training are the fname (fnames with extensions) and labels.
    • [VS-reply] You can change the label column to another column by changing the LABELS_COL variable in the settings.py
  7. In the localrunner.sh, getting rid of the -t and -m args from the python command below should give no issues while running.
      python ./script_runner.py -s $script -r $runid -b $batchsize -e $epochs -i $image_csv -t $test_csv -m $model_name > "${runid}_${script}_${batchsize}_${epochs}_${image_csv}_${model_name}.txt"
  8. In order to use the _v3.py scripts, we need the data to be pre-split.