Skip to main content

New Data Notes

This document contains Jake's notes of receiving and processing new data from DNV.

Notes​

  1. The Access DB file we received is "cameratruck database.accdb". This DB is comparable to the sample DNV databases (which are all empty) and is also comparable in structure to the many SD1 databases we've processed.
  2. I was unable to run the notebook "metadata_DB_extract_inspection_condition_data.ipynb" due to a SQL issue.
    Could not run SQL query
    C:\Users\jacob\AppData\Local\Temp\ipykernel_19548\3067042700.py:22: UserWarning: pandas only supports SQLAlchemy connectable (engine/connection) or database string URI or sqlite3 DBAPI2 connection. Other DBAPI2 objects are not tested. Please consider using SQLAlchemy.
    sql_output = pd.read_sql(ACCESS_SQL, con)
  3. I ran the SQL query in that notebook manually, but it doesn't provide all the fields we need for the rest of the flow. Upon discovery, as documented in our docs, these fields come from CSVs that are manually extracted from Access DB.
  4. As part of my development here, I plan to automate this process. This will most likely be done with a Python script, NOT a notebook.
  5. Also while learning the flow myself, I plan to update the variables and naming conventions used. For example, all_conditions.csv might be insightful for what it contains, but I'd like to be more explicit. I will make sure to note my suggested standards and naming conventions.
  6. I have the py script done for extracting CSVs from Access. It isn't immediately clear to me what the next steps should be... I'm going to break the notes section here and try to figure out the existing workflow as documented in our docs