Notes for Running DNV
This is the start of the CCTV Pipeline running the notebooks populate the database and set you up for training.
YAML
Setting up the YAML file: Most of values were self explanatory the ones that were not are listed below.
- DATASET - is list of videos received from a utility from the same day.
- EXTRACT_N_TH_FRAME - We do not use every single frame given due to the number of frames captured currently we choose every 30th frame this setting contains the integer for the n number in this case 30.
- VIDEO_GROUP_LIST - this list contains the Video Groups used in this model.
- DEFECT_CODE_STANDARD - There are 2 code types one for US and one for Canada the one we are currently using is 'PACP'
- IMAGE_EXT - what is the image extention we want to save the frames as.
- DROP_ND - Drop ND images till it matches the specified portion from the full dataset. Set this to 1 to not drop any ND labels
- SPLIT_BASIS - How are we splitting the datasets 'FB' # 'FB' for 'Frame Based' and 'VB' for 'Video Based'
Odd things with this yaml.
- DISTANCE_BOUNDING_BOX_CSV: None - This happens because for the DNV network the type handles what the Distance bounding Box will handle This is part of the network specific code in the notebooks.
- CONDITION_STANDARD_CSV: None
Order of Notebooks
| Run Order | Notebook Name | Status | Time running Notebook | Notes |
|---|---|---|---|---|
| 1 | cctv-video_list-creator.ipynb | Finished | 1 min | |
| 2 | metadata_DB_extract_inspection_condition_data.ipynb | Skipped | N/A | This is being handled by video type |
| 3 | cctv-import_supplementary_data.ipynb | Finished | 1 min | distance_bounding_box and condition standard is skipped as the CSV is set to None |
| 4 | cctv-frame-extraction.ipynb | Finished | 1 min | |
| 5 | cctv-full-frame-stitch-and-Azure-OCR.ipynb | Finished | 24 min | |
| 6 | cctv-save-blurred-frames.ipynb | Finished | 1 min | |
| 7 | cctv-extract-distance-values.ipynb | Finished | 1 min | |
| 8 | cctv-distance-value-post-processing.ipynb | Finished | 1 min | |
| 9 | cctv-extract-condition_codes.ipynb | Finished | 1 min | |
| 10 | cctv_SD1_label_images.ipynb | 1 min | ||
| 11 | cctv-prepare-cctv-training-csv.ipynb | 1 min |
Notebook issues
- When running the notebook you try to launch the virtual env. aka Kernel and nothing is showing up but you can see the virtual env in the terminal update the Visual Studio Code. It is likely out of date.
- The YAML will need to be changed in every notebook This is a hardcoded value and it needs the path. So copy it once and save it to a notepad.
- For some reason and yes I talked to Jake about it my remote connection askes for the Password for the MSI. Jake gave it to me so no big deal I just added it to the notepad from the previous page.
- Also I am keeping the path to the database in the notepad as well.
- I am skipping the notebook metadata_DB_extract_inspection_condition_data.ipynb. The Video Type handles this for DNV. There is a hardcoded section in a future notebook that handles the distance. When I get to this notebook I will note it in the note.
- cctv-import_supplementary_data.ipynb There are 2 CSV's set to none Bounding Box and Conditon Standard are both set to none.
Distance bounding box CSV: None
Video Types CSV: /media/gqc/unionsine2/cctv/dnv/Data/dnv_pacp_35_01_10/video_lists/dnv_pacp_35_01_10_video_types.csv
Condition standard CSV: None
All Conditions CSV: /media/gqc/unionsine2/cctv/dnv/Data/dnv_pacp_35_01_10/condition_data/dnv_B_C_condition_data.csv
Video DB: /media/gqc/unionsine2/cctv/dnv/Data/dnv_pacp_35_01_10/video_db/Video_DB_DNV_35_01_10.db
Skipping distance_bounding_box processing as the CSV is set to None
Skipping condition standard processing as the CSV is set to None
`all_conditions` table created
Importing external data complete.
- This morning the MSI server had to be restarted due to battery bank issues. After that I was getting an error where I could not find the database. gqc-utility-notebooksgqc@msi-server:~/git/gqc/gqc-utility-notebooks/yaml$ cd /media/gqc/unionsine2/cctv/dnv/Data/dnv_pacp_35_01_10/video_db bash: cd: /media/gqc/unionsine2/cctv/dnv/Data/dnv_pacp_35_01_10/video_db: No such file or directory This was because the Drives were not mounted. I called Dhananjay and he showed me how to mount the drives on the MSI. The documentation is in General Mounting External Drives.
- Error connecting to /media/gqc/unionsine2/cctv/dnv/Data/dnv_pacp_35_01_10/video_db/Video_DB_DNV_35_01_10.db Execution failed on sql 'SELECT FROM all_conditions order by video_file, distance asc': no such column: video_file Error connecting to /media/gqc/unionsine2/cctv/dnv/Data/dnv_pacp_35_01_10/video_db/Video_DB_DNV_35_01_10.db Execution failed on sql 'SELECT FROM all_conditions order by video_file, distance asc': no such column: video_file Error connecting to /media/gqc/unionsine2/cctv/dnv/Data/dnv_pacp_35_01_10/video_db/Video_DB_DNV_35_01_10.db Execution failed on sql 'SELECT * FROM all_conditions order by video_file, distance asc': no such column: video_file Time taken for selecting frames: 0.62s
The SD1 database has the following columns in the table all_conditions. pragma table_info(all_conditions);
- 0|video_file|TEXT|0||0
- 1|video_location|TEXT|0||0
- 2|distance|REAL|0||0
- 3|counter|REAL|0||0
- 4|Code|TEXT|0||0
- 5|continuous_defect|TEXT|0||0
The general database has the following columns in the table all_conditions.
- 0|cross_reference_id|TEXT|0||0
- 1|Distance|REAL|0||0
- 2|Continuous|TEXT|0||0
- 3|Code|TEXT|0||0
- 4|Value_Percent|INTEGER|0||0
YAML cleanup
VIDEO_LIST_CSV_STRING
Commented out in the following:
- extract_condition_codes
- extract_distance_values
- cctv extract_condition_codes
Used in 01-usage-Create_cctv-settings-yaml.ipynb
EXTRACTED_FRAMES_ZIP_STRING
Used in several notebooks
BLURRED_FRAMES_ZIP_STRING
Used in sevareal incluing cctv-save-blurred-frames.ipynb and cctvSD1_lable_images
AZURE_JSON_ZIP_STRING
Used in 01-usage-Create_cctv-settings-yaml.ipynb commented out everywhere else.
LABELS_CSV_ZIP_STRING
Used in 01-usage-Create_cctv-settings-yaml.ipynb commented out everywhere else.
VIDEO_DB_PATH_STRING
Used in several Notebooks.
RECEIVED_VIDEO_DIR_STRING
Used in several Notebooks.
LOG_DIR_STRING
Used in several Notebooks.
TEMP_DIR_STRING
Used in several Notebooks.
TRAINING_DATA_CSV_STRING
Used in several Notebooks.
TEST_DATA_CSV_STRING
Used in 01-usage-create-cctv-settings-yaml, training-data-splitter and cctv-prepare-cctv-training-csv