Skip to main content

Data sources and destinations

info

This documentation needs to be updated with latest data. The content is in reference to data available on 06/02/2023.

Useful commands​

  1. List the different file extensions present in files under a folder including subfolders.
    find . -type f | sed -E 's/.*\.([^.]+)$/\1/' | sort -u

Dataset locations​

SD1-D (or SD1 May 26 2023)​

There are 1101 videos present. SD1-D processing steps and details are present here

Location: MSI /home/gqc/CCTV/SD1/Data/2023_05_26

Data storage locations​

Most of the data required is available on Google drive.

  1. hydrotrekai@gmail.com: Original videos of SD1 are uploaded here. A snapshot of the same is below: Google drive snapshot - 06/02/2023

  2. hydrotrek@gmail.com: There are directories for each project. All the data and notebooks used for them are located in multiple folders. A snapshot of root directory is below: Google drive snapshot - 06/02/2023

Data Destinations​

MSI​

  • SD1_230526 - Downloaded on MSI Ubuntu - /home/gqc/CCTV/SD1/ZIP_folder
  • SD1_230526 - Extracted there - /home/gqc/CCTV/SD1/Data/2023_05_26/Received_Data
  • SD1C - Downloaded zips and extracted files
  • Metro 530 - ZIPs and extracted

Dhananjay:
The MSI Server contains two folders relating to the CCTV project: cctv and CCTV (Note the upper and lower cases).

CCTV folder:​

The tree structure of the first and second levels of the CCTV folder is shown below: alt text As we can see, in the CCTV folder there are sub-folders for different utilities and within each utility subfolder there is a Data folder.

  • SD1_old_structure: This folder contains a folder (Data) for data, a folder (Notebooks) that contains the ipynb files for preprocessing and Azure OCR, as well as CSV files that contain the locations for frames and defect annotations. The CSV files reference locations within the Data folder.

cctv folder:​

The cctv folder contains the data for the SD1 utility in a restructured format within .db files. The tree of the cctv folder is shown below: alt text Here, there are two DB files, namely cctv_sd1.db and cctv_sd1_old.db.

External HD / D-Drive​

  • SD1C - ZIPs, extracted and converted MPG & MP4 files.

Notebooks​

GDrive / Google Colab​

  • Google collab notebooks used across projects are under Colab Notebooks folder. Colab Notebooks - 06/02/2023

Github​

Last updated on by Srujana