Skip to main content

AI3 - Phase 3

  • Documentation available as web app, through mkdocs. It can be run from ai3-hackathon repository using mkdocs serve.

Updated flow charts​

The flowcharts can be found in the flowcharts section

File locations​

Notebooks​

  1. Notebooks created for data processing and mkodcs enabled: MSI > /home/gqc/git/gqc/AI3/ai3-hackathon/docs/nbs
  2. Notebooks used to train the LLMs and the saved LLMs: MSI > /home/gqc/git/gqc/AI3
  3. Notebooks for YOLO:
    • MSI > ~/testing-yolo - Also has original and output videos
    • WSL > /home/devengqc/git/projects/notebooks-from-colab/nbs

Data​

  1. PSIAP IoT messages: MSI > /home/gqc/git/gqc/AI3/ai3-hackathon/docs/data
  2. AI3 PSIAP GDrive data: WDBlack > /media/gqc/WD_BLACK1/ai3-new-data
  3. NIST IoTData_EmergencyScenarios: WDBlack > /media/gqc/WD_BLACK1/ai3-old-data
  4. The saved files that contain LLAMA and OPT models' along with finetuned files are here: MSI > /home/gqc/git/gqc/AI3

Notebooks - Functions​

  • We use BeautifulSoup for web scraping.
  • We have utils functions used in notebooks under ./iotgpt/
  1. generate-sentences.ipynb
    • For a given JSON file, it generates sentences and saves as ./nbs/all-json-sentencified.csv.
    • Functions used: iotgpt.mqtt.publisher.clean_obj, iotgpt.utils.generate_sentences_from_json
  2. json-to-db.ipynb
    • This notebook will write all the sentences from JSON strings into a SQLite database ./questions-answers.db/q&a.
    • The table columns are - sentences, question, answers(default=None)
    • Functions used: from fastcore.utils import Path, globtastic
    • Note:
      • There is no implementation to save to a DB.
      • Looks like this was a work in-progress, to save the sentences data.
  3. label.ipynb
    • For labelling the data with the help of Bard API, by reading sentences from q&a DB.
    • It reads the data from tables, pings Bard API with 'hello' intially, checks for error responses, and then updates the answers column with the responses for questions in the DB.
    • Functions used: ask_bard function -> has bardapi.core.Bard().get_answer(prompt)['content']
  4. lookup-db.ipynb
    • This notebook creates the SQLiteDB structure from all the JSON files and the csv files.
    • It gets a sorted list of keys, splits the keys on '_', gets the synonyms for each word in the key and generates combinations with the synonyms. It then saves data into table ai3-lookup.db as {keys, combined_keys}
    • It then connects to ai3-lookup.db and saves the tables, phrase_synonyms and all_phrases_synonyms, into it.
    • We now find closest key based on Levenshtein distance on all_phrases_synonyms table and then finds parent key on phrase_synonyms.
    • This should also deal NK keys, but example isn't available.
    • Functions used: iotgpt.utils, iotgpt.classification, generate_combinations, get_synonyms_thesaurus, Levenshtein.distance
  5. merge-nb.ipynb
    • This notebook merges (concatenates) the outputs of the database containing labelled Bard outputs and GPT-3.5 outputs.
    • Reads labelled-samples and 8000-samples csv files from /nbs with questions and answers, and merges the files after renaming columns.
    • Functions used: iotgpt.utils.generate_sentences_from_json
  6. missing-keys.ipynb
    • Finds the closest sentence based on a key using levenstein distance.
    • Has commented code for saving the keys as 'all-json-keys.pkl'
    • It also finds missing keys.
    • Input: JSON data from 'PSIAP_AI3/'
    • Output: List of Keys is saved as 'all-json-keys.pkl'
    • Functions used: iotgpt.utils.{extract_keys, get_abbreviations, get_synonyms_thesaurus}, replace_word_with_underscore, find_missing_key
  7. openai-exploration.ipynb
    • This notebook explores various prompts and models.
    • Functions used: OpenAI API
  8. openllama-13b-instruct.ipynb
    • Has the Llama and transformers usage code.
    • Looks like there are no successful runs yet, need to verify
    • Functions used: find_missing_key.{AutoModelForCausalLM, AutoTokenizer}, peft.{LoraConfig, TaskType, get_peft_model}
  9. sample-for-training.ipynb
    • In this notebook, we create a sample of 2000 JSON sentences for training.
    • Randomly selects json entries from /Data, standardizes and sentencifies it, asks OpenAI models the questions for respective sentences and saves the answers. It also iterates over failed reponses to label missed JSON entries.
    • Functions used: openai, fastcore.utils import Path, globtastic, coll_repr, iotgpt.utils.generate_sentences_from_json, standardize_keys
  10. test.ipynb
    • This notebook creates the custom data structure from all the JSON files and the csv files.
    • Saves abbreviations as a pickle file 'abbreviations-of-complex-keys-json-csv-3.pkl'
  11. LLAMA2.ipynb
    • This is used for inferencing base LLAMA model from hugging face. Might have been for testing LLAMA usage in local.
  12. finetune-LLAMA2.ipynb
    • This notebook is used for finetuning the base LLAMA model on custom data.

Notes​

  1. Data source : https://github.com/taugroup/PSIAP_AI3.git

    • In MSI/AI3/ai3-hackathon, JSON data is available at '/PSIAP_AI3/Phase2/' and NK data is under '/PSIAP_AI3/Phase3/'.
    • Additional data source (from notebooks) - AI3/docs/nbs
      • ai3-lookup.db

  2. Process:

    • Receive the payload and convert it into a python dictionary object.
    • Sentencify the dictionary object to get a sentencified JSON.
    • Join sentences with a whitespace for Bard and the in-house model.
    • Write the json payload to the sqlite database
    • Notebooks
      1. Find missing keys
        • This assumes missing keys will be '_' in JSON
        • Extract keys from dictionary and join on ','.
        • Use iotgpt.classification.find_missing_key to get missing keys, using Levenshtein distance on all-json-keys.pkl.
        • If no missing keys found, proceed after notifying.
        • Where is 'all-json-keys.pkl' - ai3-hackathon/search_dir
      2. Standardize and classsify
        • We use the DB table with standard keys and synonyms to replace the JSON keys with standard keys.
        • If no standard key found, we don't replace it and proceed.
      3. Sentencify
      4. Use Bard/GPT
      5. Use in house model
      6. Text to speech

  3. Things to note :

    1. While calculating the mean for the phrases and synonyms based on json values, we are ignoring strings because we can't find their mean. This means, we are also excluding the cases where strings have values like timestamps, dates etc.

    2. One way to solve this is by converting dates to UNIX and for strings containing numbers to float/integers.

      • SK - It won't work because there is no distribution that we can use to understand the key for that value. It won't work unless we use LLMs like CHATGPT.

      • For eg:

            if 2023-03-02T18:01:00.000Z is a value in a json, what would be it's key? Don't give explanations. Just give key value pair.

            {
               "timestamp": "2023-03-02T18:01:00.000Z"
            }