CCTV CV Workflow
This is the CCTV CV project. Below you will find the high level workflow for CCTV CV. For utility specific workflows, check out the utility specific pages in this section.
High Level Overview of CCTV CV
SD1 Database backups
The backups for the SD1 database are stored in T7 in the folder /media/gqc/T7/cctv/SD1/SD1_BackupDBs. They have dates at the end of the file names to indicated when they were created as a backup.
db.py file for handling database connections:
There is a db.py file that is used in the cctv_cv notebooks for handling database connections. It is generated from the cctv-db-connection.ipynb notebook. The db.py file is a wrapper around sqlalchemy which was created by Pavan for CCTV CV. We are moving away from using this wrapper and using sqlalchemy directly.
Utility Specific Table locations
| Abstract Concept | DNV | COV | SD1 |
|---|---|---|---|
| Header in Inspection Dataset | DNV/Data/condition_data_csv/CCTV_Headers.csv | covcombined_92videofile_insepction_id cross_reference_for_image_labelling.csv | A table in the Access Database |
| Details in Inspection Dataset | DNV/Data/condition_data_csv/ | cov_combined_92videos_PACP_details.csv | A table in the Access Database |
| Extracted Frames | unionsine1/VS_Research/CCTV/DNV/Data/Extracted_Frames | unionsine1/VS_Research/CCTV/COV/Data/Extracted_Frames | /home/gqc/CCTV/SD1/Data/2023_05_26/Extracted_Frames |
| Azure JSON | unionsine1/VS_Research/CCTV/DNV/Data/Azure_JSON | unionsine1/VS_Research/CCTV/COV/Data/Azure_JSON | /home/gqc/CCTV/SD1/Data/2023_05_26/Azure_JSON |
file_organizer.sh
There is a file_reorganizer.sh script in the received_data folder in for SD1 in dataset 2023_05_26 (SD1_700) which:
- Finds all first-level subdirectories.
- Iterates over each subdirectory.
- Changes into each subdirectory and prints its path.
- Removes the Videos directory within each subdirectory.
There are lines that are commented out such as creating an MDB directory, moving .mdb files into it, and moving files from a Videos directory to the current directory.
CCTV CV Processing Pipeline Guide
The CCTV CV processing notebooks are driven by the .yml files stored in the gqc-utility-notebooks/yaml folder. There are various flags that need to be set to run the notebooks correctly. When creating a new yaml file, please check the cctv_settings_ds_SD1_B.yml file for a basic overview of what flags are present and how each flag gets filled. It also has helpful comments to show you what to fill into the flags.
Miscellaneous notes
Note:DNV uses the dnv_B_C_condition_data.csv, located in /media/gqc/unionsine1/VS_Research/CCTV/DNV/Data/condition_data_csv, as the 'all_conditions' csv for the CV pipeline. This was found out by looking at the notebook /media/gqc/unionsine1/VS_Research/CCTV/notebooks/CCTV videos pipelines (Paper 1)/7-2023-07-06_create_image_labels_v2_access_database_updated_version.ipynb.
Currently, only the utl_x/Data/Dataset_X/video_db/Video_DB_new.db has foreign key relationships defined between the different tables. The SD1, and COV utilities do not have foreign keys defined for their databases.