CCTV data in VS_Research summary
DNV dataset summary
| Upload directory name | 81 PACP Videos | 406 PACP Videos (Uploaded on January 23) | 1066 WRc Videos (Uploaded on January 23) | 270 PACP Videos (Uploaded on March 23)* | 35 PACP Videos (Uploaded May 1)* | 1650 WRc Videos | 1811 Videos not found in CCTV Details |
|---|---|---|---|---|---|---|---|
| Uploaded location (gdrive email, HD, etc) | |||||||
| Dataset | A | B | C | ||||
| Zip Videos into groups of 10 and create video lists csv | NA | NA | done | done | done | ||
| Extract every 30th Frames from videos | done | done | done | done | done | ||
| Extract mid frames | NA | NA | done | done | done | ||
| Classify video type | done | done | done | done | done | ||
| Run Azure OCR | done | done | done | done | done | ||
| Blurred Images | done | done | done | done | done | ||
| Distance/Label Extraction based on JSON | done | done | done | done | done | ||
| Distance value post processing | NA | NA | done | done | |||
| Create metada csv | done | done | done | done | done | ||
| Create Labelled Images (Dataset) | done | done | redo after completing step 8 | done | done | ||
| Split Data to Train & Validation / Test | done | done | done | done | |||
| Train a baseline model | done | done | done | done | |||
| Run prediction on test data | done | done | done | done | |||
| * One of the videos in the 81 videos have abandoned the survey in the beginning. Hence, not included in the training. | * 270 PACP and 35 PACP datasets contain the same videos found in 81 PACP datasets but with different file names. | * 270 PACP and 35 PACP datasets contain the same videos found in 81 PACP datasets but with different file names. |
DNV / Data
Here is a summary of the files and folders in the upper level. For details on each, refer the next subsections.
├── Azure_JSON --> zipped Azure OCR JSON files for `PACP 35` + `wrc_1066`
├── Binary_classification
├── Binary_Training_Data
├── Blurred_Frames
├── Blurred Videos
├── CCTV_Original_Frames
├── CCTV_Training_Data
├── Color histogram
├── condition_data_csv
├── Extracted_CSV
├── Extracted_Frames
├── feature_engineer_data
├── GVision_JSON
├── Labelled_Images_CSV
├── Labels_CSV
├── list_of_videos_from_January_2023_upload
├── list_pickle_files
├── PACP_270 --> Contains data for `PACP 270` videos (More details below)
├── pixel_flip
├── Plots
├── read_json_from_Azure.ipynb
├── Sample_middle_frames
├── tranform_img
├── video
├── Video_Lists
├── Videos
├── WRc
└── zip_unblurred_images
/ Azure_JSON
- Contains Azure OCR result json files in
video-groupbased zips forDNV-PACP-35andDNV-WRc-1066datasets. - Naming of the zip files have
JSON_prefix comparing to our standard. - There are some folders and zip files for individual videos.
upload 2023contains subfolders for set of videos which I'm not sure of the dataset they belong to.
├── Azure_JSON
│ ├── JSON_dnv_pacp_35_01_10.zip
│ ├── JSON_dnv_pacp_35_11_20.zip
│ ├── JSON_dnv_pacp_35_21_30.zip
│ ├── JSON_dnv_pacp_35_31_35.zip
│ ├── JSON_wrc_1066_01_10.zip
...
│ ├── JSON_wrc_1066_1001_1010.zip
│ ├── JSON_wrc_1066_1011_1020.zip
│ ├── JSON_wrc_1066_101_110.zip
│ ├── JSON_wrc_1066_991_1000.zip
│ ├── old_wrc_1066_211_220_and_961_970
│ ├── SANMN00304_1
│ ├── SANMN00396_12
│ ├── SANMN08983_19
│ ├── SANMN08983_19.zip
│ └── upload_2023 --> contains sub-folders with video names containing azure json files
/ Binary_classification
├── Binary_classification
│ ├── binary_text_notext.pkl
│ ├── binary_text_simple_tfms.pkl
│ └── GVision_binary_JSON
/ Binary_Training_Data
├── Binary_Training_Data
│ └── SANMN00893_1
/ Blurred_Frames
- Blurred Frames for
DNV-PACP-35andDNV-WRc-1066datasets are present in zip format. - Naming has
├── Blurred_Frames
│ ├── Training_Frames_dnv_pacp_35_01_10.zip
│ ├── Training_Frames_dnv_pacp_35_11_20.zip
│ ├── Training_Frames_dnv_pacp_35_21_30.zip
│ ├── Training_Frames_dnv_pacp_35_31_35.zip
│ ├── Training_Frames_dnv_pacp_B_406_01_406.zip
│ ├── Training_Frames_wrc_1066_01_10.zip
│ ├── Training_Frames_wrc_1066_1001_1010.zip
...
│ └── Training_Frames_wrc_1066_991_1000.zip
/ Blurred Videos
├── Blurred Videos
│ └── SANMN00893_1_blurred.avi
/ CCTV_Original_Frames
├── CCTV_Original_Frames
│ ├── 001-3648
│ ├── 001-3648.csv
│ ├── 03-APR-002
│ ├── 03-APR-002.csv
.
.
.
│ ├── SANMN86600_4
│ ├── SANMN86600_4.csv
│ ├── test_azure_2.csv
│ ├── test_azure.csv
│ └── upload_2023
/ CCTV_Training_Data
├── CCTV_Training_Data
│ ├── Gvision
│ ├── SANMN00304_1
│ ├── SANMN00304_1.csv
│ ├── SANMN00396_12
│ ├── SANMN00396_12.csv
.
.
.
│ ├── SANMN86600_4
│ ├── SANMN86600_4.csv
│ ├── sharpen_sample
│ ├── test_90_10_split_v3.zip
│ ├── test_95_5_split_v3.zip
│ ├── test_azure_2.csv
│ ├── test_azure.csv
│ └── upload_2023
/ Color histogram
├── Color histogram
│ ├── color histogram.ipynb
│ ├── SANMN00304_1.pkl
│ ├── SANMN00396_12.pkl
.
.
.
│ └── SANMN86600_4.pkl
/ condition_data_csv
├── condition_data_csv
│ ├── CCTV_details_for_image_labelling.csv
│ ├── CCTV_Details_sorted_by_jobnumber_distance_code.csv
│ ├── CCTV_Details_sorted_by_jobnumber_distance_code_edit.csv
│ ├── CCTV_Headers.csv
│ ├── dnv_B_C_condition_data.csv
│ ├── dnv_B_C_condition_data_with_remarks_col.csv
│ ├── DNV_PACP_406_out_3572videos.csv
│ ├── DNV_PACP_676_out_4483videos.csv
│ ├── DNV_PACP_711_out_5238videos.csv
│ ├── DNV_PACP_81videos_edit.csv
│ ├── dnv_pacp_A_81_condition_data.csv
│ ├── DNV_WRc_1066_out_3572videos.csv
│ ├── DNV_WRc_1335_out_4483videos.csv
│ ├── DNV_WRc_2716_out_5238videos.csv
│ ├── inspection_info_406_PACP_videos.csv
│ ├── jobnumber_in_details_with_pipe_id.csv
│ ├── list_1472_out_3572_found_in_CCTV_Details.csv
│ ├── list_2011_out_4483_found_in_CCTV_Details.csv
│ ├── list_3427_out_5238_found_in_CCTV_Details.csv
│ ├── list_730_PACP_videos_in_CCTV_Details.csv
│ ├── list_unique_wrc_code.csv
│ ├── list_unique_wrc_code_v2.csv
│ ├── list_videos_on_gdrive_found_in_CCTV_Details.csv
│ ├── new_DNV_PACP_270_out_4483videos.csv
│ ├── new_DNV_PACP_270_out_4483videos.xlsx
│ ├── new_DNV_PACP_35_out_5238videos.csv
│ ├── new_DNV_WRc_1650_out_5238videos.csv
│ ├── pacp_792_out_5319_videos_info.csv
│ ├── PACP codes
│ ├── pipe properties
│ ├── pipe_properties_of_74_videos.csv
│ ├── pipe_properties_of_81_videos.csv
│ └── readme.docx
/ Extracted_CSV
├── Extracted_CSV
│ ├── csv_from_dnv_pacp_35_01_10.zip
│ ├── csv_from_dnv_pacp_35_11_20.zip
│ ├── csv_from_dnv_pacp_35_21_30.zip
│ ├── csv_from_dnv_pacp_35_31_35.zip
│ ├── csv_from_wrc_1066_01_10.zip
.
.
.
│ └── csv_from_wrc_1066_991_1000.zip
/ Extracted_Frames
├── Extracted_Frames
│ ├── unblurred_images_dnv_pacp_35_01_10.zip
│ ├── unblurred_images_dnv_pacp_35_11_20.zip
│ ├── unblurred_images_dnv_pacp_35_21_30.zip
│ ├── unblurred_images_dnv_pacp_35_31_35.zip
│ ├── unblurred_images_wrc_1066_01_10.zip
│ ├── unblurred_images_wrc_1066_1001_1010.zip
.
.
.
│ └── unblurred_images_wrc_1066_991_1000.zip
/ feature_engineer_data
├── feature_engineer_data
│ ├── condition_data_DNV_B_C_version_2.csv
│ ├── input_DNV_487pipes.csv
│ ├── input_DNV_A_B_C.csv
│ ├── input_DNV_A_B_C_version_2.csv
│ ├── input_DNV_A_B_C_version_3_(row_per_pipe_id).csv
│ ├── inspection_info_406pipes.csv
│ ├── inspection_info_487pipes.csv
│ ├── inspection_info_81pipes.csv
│ ├── inspection_info_DNV_A_version_2.csv
│ └── inspection_info_DNV_B_C_version_2.csv
/ GVision_JSON
├── GVision_JSON
│ ├── all_frames_SANMN00304_1
│ ├── all_frames_SANMN00396_12
.
.
.
│ ├── SANMN08983_19
│ └── SANMN86600_4
/ Labelled_Images_CSV
├── Labelled_Images_CSV
│ ├── labels_dnv_pacp_35_01_10.csv
│ ├── labels_dnv_pacp_35_11_20.csv
│ ├── labels_dnv_pacp_35_21_30.csv
│ ├── labels_dnv_pacp_35_31_35.csv
│ ├── labels_dnv_pacp_35_all_35.csv
│ ├── labels_dnv_pacp_791.csv
│ ├── labels_pacp_270_01_10.csv
│ ├── labels_pacp_270_101_110.csv
.
.
.
│ ├── labels_pacp_270_91_100.csv
│ ├── labels_pacp_270_all_270.csv
│ ├── labels_SANMN02116_160_new_v2.csv
│ ├── labels_SANMN03021_164_new_v2.csv
.
.
.
│ ├── labels_SANMN07882_154_new_v2.csv
│ ├── labels_SANMN07924_634_new_v2.csv
│ ├── labels_SANMN07935_596_new_v2_ccoeff.csv
│ ├── labels_SANMN07935_596_new_v2.csv
│ ├── labels_SANMN08095_570_new_v2.csv
│ ├── labels_STM01764_443_new_v2.csv
│ ├── New_June_21_version_2
│ ├── New_June_29_version_3
│ └── train_valid_test_split
/ Labels_CSV
├── Labels_CSV
│ ├── Extracted_data_dnv_pacp_35_01_10.zip
│ ├── Extracted_data_dnv_pacp_35_11_20.zip
│ ├── Extracted_data_dnv_pacp_35_21_30.zip
│ ├── Extracted_data_dnv_pacp_35_31_35.zip
│ ├── Extracted_data_dnv_pacp_A_81_01_81.zip
│ ├── Extracted_data_dnv_pacp_B_406_01_406.zip
│ ├── Extracted_data_pacp_270_01_10.zip
.
.
.
│ ├── Extracted_data_pacp_270_91_100.zip
│ ├── Extracted_labels_wrc_1066_01_10.zip
.
.
.
│ ├── Extracted_labels_wrc_1066_1061_1066.zip
│ ├── Filtered_Extracted_data_dnv_pacp_35_01_10.zip
│ ├── Filtered_Extracted_data_dnv_pacp_35_11_20.zip
│ ├── Filtered_Extracted_data_dnv_pacp_35_21_30.zip
│ ├── Filtered_Extracted_data_dnv_pacp_35_31_35.zip
│ ├── Filtered_Extracted_data_dnv_pacp_A_81_01_81.zip
│ ├── Filtered_Extracted_data_dnv_pacp_B_406_01_406.zip
│ ├── Filtered_Extracted_data_pacp_270_01_10.zip
│ ├── Filtered_Extracted_data_pacp_270_101_110.zip
.
.
.
│ ├── Filtered_Extracted_data_pacp_270_91_100.zip
│ ├── Filtered_Extracted_data_wrc_1066_01_10.zip
.
.
.
│ ├── Filtered_Extracted_data_wrc_1066_1061_1066.zip
│ ├── labels_wrc_1066_01_10.csv
.
.
.
│ ├── labels_wrc_1066_1061_1066.csv
│ ├── list_video_group_for_rerun.data
│ ├── list_video_group_for_rerun_end_1061_1066.data
│ ├── list_video_name_with_bad_cont_defect.data
│ └── list_video_name_with_bad_cont_defect_end_1061_1066.data
/ list_of_videos_from_January_2023_upload
├── list_of_videos_from_January_2023_upload
│ ├── list_of_csv_of_11_videos_from_the_first_120_videos_need_to_be_rerun.csv
│ ├── list_of_videos_on_gdrive_found_in_CCTV_details_with_PACP_121_to_150_videos.csv
│ ├── list_of_videos_on_gdrive_found_in_CCTV_details_with_PACP_151_to_180_videos.csv
│ ├── list_of_videos_on_gdrive_found_in_CCTV_details_with_PACP_181_to_210_videos.csv
│ ├── list_of_videos_on_gdrive_found_in_CCTV_details_with_PACP_211_to_240_videos.csv
│ ├── list_of_videos_on_gdrive_found_in_CCTV_details_with_PACP_241_to_270_videos.csv
│ ├── list_of_videos_on_gdrive_found_in_CCTV_details_with_PACP_271_to_300_videos.csv
│ ├── list_of_videos_on_gdrive_found_in_CCTV_details_with_PACP_301_to_330_videos.csv
│ ├── list_of_videos_on_gdrive_found_in_CCTV_details_with_PACP_331_to_360_videos.csv
│ ├── list_of_videos_on_gdrive_found_in_CCTV_details_with_PACP_361_to_390_videos.csv
│ ├── list_of_videos_on_gdrive_found_in_CCTV_details_with_PACP_391_to_406_videos.csv
│ ├── list_of_videos_on_gdrive_found_in_CCTV_details_with_PACP.csv
│ ├── list_of_videos_on_gdrive_found_in_CCTV_details_with_PACP_first_120.csv
│ ├── list_unique_wrc_code.csv
│ ├── list_videos_PACP.csv
│ └── video_path_to_SANMN01658_143.csv
/ list_pickle_files
├── list_pickle_files
│ └── list_of_DNV_A_and_B_unblurred_images.pkl
/ PACP_270
├── PACP_270
│ └── Data
/ Data
Following folders follow the general organization structure we use to some extent. Notes are added to the exceptions.
├── Azure_JSON
├── Blurred_Frames
├── Extracted_CSV
├── Extracted_Frames
├── Labelled_Images_CSV --> image labels csv for group 1 and few selected videos
├── Labels_CSV
├── pacp270_zips --> Corresponds to `Videos` folder in org structure.
└── Video_Lists
/ pixel_flip
├── pixel_flip
│ ├── 03-APR-002
│ └── 03-APR-004
/ Plots
├── Plots
│ ├── DNV 792 PACP videos: Contractors.png
│ ├── DNV 792 PACP videos: count of codes.png
│ ├── DNV 792 PACP videos: inspection_date.png
│ ├── DNV 792 PACP videos: Inspection Purpose.png
│ ├── DNV 792 PACP videos: inspection_year.png
│ ├── DNV 792 PACP videos: length_survey_in_header.png
│ ├── DNV 792 PACP videos: number of video with MSA.png
│ ├── DNV 792 PACP videos: Pipe Diameters.png
│ ├── DNV 792 PACP videos: Pipe Materials.png
│ ├── DNV 792 PACP videos: Pipe Shapes.png
│ └── DNV 792 PACP videos: Reasons why MSA.png
/ read_json_from_Azure.ipynb
├── read_json_from_Azure.ipynb
/ Sample_middle_frames
├── Sample_middle_frames
│ ├── DNV_list_of_sample_middle_frames.csv
│ ├── dnv_pacp_270_distance_bounding_box.csv
│ ├── dnv-pacp-35-distance-bouding-box.csv
│ ├── DNV-WRc-video_type.txt
│ ├── example images of different video types
│ ├── Frames
│ ├── JSON
│ ├── mid_frames_pacp_35_all35_video_types.csv
│ ├── pacp_270_all270_video_types.csv
│ ├── Sample_middle_frames_wrc_1066videos.zip
│ ├── wrc-1066-distance-bounding-box.csv
│ ├── wRC_1066_video_dimensions.csv
│ └── wrc-1066-video_type.csv
/ tranform_img
├── tranform_img
│ ├── FlipItem
│ └── sharp
/ video
├── video
│ ├── #1 DHL Lynn Valley Oct 6 2021
│ ├── 27_videos_type_of_annoation.csv
│ ├── DHL Clements Ave Oct 2020
│ └── DHL Lynn Valley March 3 2022
/ Video_Lists
├── Video_Lists
│ ├── dnv_pacp_35_01_10.csv
│ ├── dnv_pacp_35_11_20.csv
│ ├── dnv_pacp_35_21_30.csv
│ ├── dnv_pacp_35_31_35.csv
│ ├── dnv_pacp_A_81_01_81.csv
│ ├── dnv_pacp_A_81_01_81.xlsx
│ ├── dnv_pacp_B_406_01_406.csv
│ ├── dnv_wrc_B_1650_01_10.csv
.
.
.
│ ├── dnv_wrc_B_1650_1641_1650.csv
│ ├── wrc_1066_01_10.csv
.
.
.
│ ├── wrc_1066_1061_1066.csv
/ Videos
├── Videos
│ ├── dnv_pacp_35_01_10.zip
│ ├── dnv_pacp_35_11_20.zip
│ ├── dnv_pacp_35_21_30.zip
│ ├── dnv_pacp_35_31_35.zip
│ ├── dnv_wrc_B_1650_01_10.zip
.
.
.
│ ├── dnv_wrc_B_1650_1641_1650.zip
│ ├── wrc_1066_01_10
.
.
.
│ ├── wrc_1066_1061_1066.zip
/ WRc
├── WRc
│ ├── Extracted_CSV
│ ├── Labelled_Images_CSV
│ ├── Labels_CSV
│ └── temp
/ zip_unblurred_images
└── zip_unblurred_images
├── DNV_A_B_and_plus_unblurred_01_10000.zip
├── DNV_A_B_and_plus_unblurred_100001_110000.zip
├── DNV_A_B_and_plus_unblurred_10001_20000.zip
├── DNV_A_B_and_plus_unblurred_110001_120000.zip
├── DNV_A_B_and_plus_unblurred_120001_130000.zip
├── DNV_A_B_and_plus_unblurred_130001_140000.zip
├── DNV_A_B_and_plus_unblurred_140001_150000.zip
├── DNV_A_B_and_plus_unblurred_150001_160000.zip
├── DNV_A_B_and_plus_unblurred_160001_170000.zip
├── DNV_A_B_and_plus_unblurred_170001_180000.zip
├── DNV_A_B_and_plus_unblurred_180001_190000.zip
├── DNV_A_B_and_plus_unblurred_190001_200000.zip
├── DNV_A_B_and_plus_unblurred_200001_210000.zip
├── DNV_A_B_and_plus_unblurred_20001_30000.zip
├── DNV_A_B_and_plus_unblurred_210001_220000.zip
├── DNV_A_B_and_plus_unblurred_220001_230000.zip
├── DNV_A_B_and_plus_unblurred_230001_240000.zip
├── DNV_A_B_and_plus_unblurred_240001_250000.zip
├── DNV_A_B_and_plus_unblurred_250001_260000.zip
├── DNV_A_B_and_plus_unblurred_260001_270000.zip
├── DNV_A_B_and_plus_unblurred_270001_280000.zip
├── DNV_A_B_and_plus_unblurred_280001_290000.zip
├── DNV_A_B_and_plus_unblurred_290001_300000.zip
├── DNV_A_B_and_plus_unblurred_300001_310000.zip
├── DNV_A_B_and_plus_unblurred_30001_40000.zip
├── DNV_A_B_and_plus_unblurred_310001_320000.zip
├── DNV_A_B_and_plus_unblurred_320001_330000.zip
├── DNV_A_B_and_plus_unblurred_330001_340000.zip
├── DNV_A_B_and_plus_unblurred_340001_350000.zip
├── DNV_A_B_and_plus_unblurred_350001_360000.zip
├── DNV_A_B_and_plus_unblurred_360001_370000.zip
├── DNV_A_B_and_plus_unblurred_370001_380000.zip
├── DNV_A_B_and_plus_unblurred_380001_390000.zip
├── DNV_A_B_and_plus_unblurred_390001_400000.zip
├── DNV_A_B_and_plus_unblurred_400001_410000.zip
├── DNV_A_B_and_plus_unblurred_40001_50000.zip
├── DNV_A_B_and_plus_unblurred_410001_420000.zip
├── DNV_A_B_and_plus_unblurred_420001_430000.zip
├── DNV_A_B_and_plus_unblurred_430001_440000.zip
├── DNV_A_B_and_plus_unblurred_440001_450000.zip
├── DNV_A_B_and_plus_unblurred_450001_460000.zip
├── DNV_A_B_and_plus_unblurred_460001_470000.zip
├── DNV_A_B_and_plus_unblurred_470001_480000.zip
├── DNV_A_B_and_plus_unblurred_480001_490000.zip
├── DNV_A_B_and_plus_unblurred_490001_500000.zip
├── DNV_A_B_and_plus_unblurred_500001_510000.zip
├── DNV_A_B_and_plus_unblurred_50001_60000.zip
├── DNV_A_B_and_plus_unblurred_510001_520000.zip
├── DNV_A_B_and_plus_unblurred_520001_530000.zip
├── DNV_A_B_and_plus_unblurred_530001_540000.zip
├── DNV_A_B_and_plus_unblurred_540001_550000.zip
├── DNV_A_B_and_plus_unblurred_550001_560000.zip
├── DNV_A_B_and_plus_unblurred_560001_570000.zip
├── DNV_A_B_and_plus_unblurred_570001_580000.zip
├── DNV_A_B_and_plus_unblurred_580001_590000.zip
├── DNV_A_B_and_plus_unblurred_590001_600000.zip
├── DNV_A_B_and_plus_unblurred_600001_610000.zip
├── DNV_A_B_and_plus_unblurred_60001_70000.zip
├── DNV_A_B_and_plus_unblurred_610001_620000.zip
├── DNV_A_B_and_plus_unblurred_620001_630000.zip
├── DNV_A_B_and_plus_unblurred_630001_640000.zip
├── DNV_A_B_and_plus_unblurred_640001_650000.zip
├── DNV_A_B_and_plus_unblurred_650001_660000.zip
├── DNV_A_B_and_plus_unblurred_660001_670000.zip
├── DNV_A_B_and_plus_unblurred_670001_680000.zip
├── DNV_A_B_and_plus_unblurred_680001_690000.zip
├── DNV_A_B_and_plus_unblurred_690001_700000.zip
├── DNV_A_B_and_plus_unblurred_700001_710000.zip
├── DNV_A_B_and_plus_unblurred_70001_80000.zip
├── DNV_A_B_and_plus_unblurred_710001_717103.zip
├── DNV_A_B_and_plus_unblurred_80001_90000.zip
└── DNV_A_B_and_plus_unblurred_90001_100000.zip
480 directories, 1886 files