TODO actions

Issues in notebook

1. In finetune-llama.ipynb, the model inference is slow in google collab with v100 GPU for some reason. Where as, same code on MSI runs within few seconds.
   • We think it might be because of the environment, configuration and GPU capacity that might be causing this.

Known issues

1. List of things to be finished on AI3 front (From conversation with Deven)
   1. Classification of unknown keys needs to be implemented.
      • The idea is to look at the distribution of the values coming in and classify based on that.
   2. On the LLM training side:
      1. Label more data
      2. Create a validation dataset
      3. Train and test the model
      4. Look at the performance (output validation)
   3. MQTT side:
      1. Code clean-up
      2. Confirm with Sudhir - Where the switches (actions) are needed in the final version of the app or not.
2. Need to work on finetuning the trained model with PSIAP data, for better accuracy (since the recently trained model doesn’t work better).
3. Need to validate the accuracy of in-house Llama model.
4. Need to work on YOLO training and try to use the resources Sudhir sent on YOLO.
5. Need to handle NK keys with string values. We discussed on this to go about 2 ways :
   1. Currently, the keys with string values have null in mean column in the lookup.db. We discussed to place the most frequent word in the JSON data as a mean.
   2. We can use LLM like OpenAI API or LLAMA to interpret the key based on the context of the message, by looking at other keys. We have tested this using the UI and it gives positive results.

Current Process and Areas for Improvement

• We should keep a running average of the NK values instead of checking EACH MESSAGE that comes in, add the message val to the average and then check the updated average with DB lookup.
• When we insert each NK message, we should add our 'estimated_key' or something like that
• We need to insert ALL data into a database so that we can visualize it.
• We can keep a single table into which all data is inserted. Or we can keep two tables, one for NK streams and one for Known streams.
• We should consider taking dates into consideration when guessing NK. For example, if the dates between messages are sub-daily, then the NK won't be any daily values. Same with monthly. I.e. monthlyrainin.
• We should also compare NK streams to Known streams to see if there's any comparable data that's been labeled, i.e. temperature or magnitude at a nearby sensor could help identify NK streams.
• We need to ensure that the processing time doesn't block incoming messages. Where is the bottleneck? Maybe we can spawn an async thread after each message is received to do the processing. Or when the program starts we start a worker thread to handle the processing, then when receiving messages on a different thread, we insert into a queue which is used by the worker.
• If the known topics aren't labeled as keys but rather their contained keys are the only data type reference, then we will want to keep track of what values are in each stream?
• Visualization - We can explore displaying heat maps based on sensor locations, if we get co-ordinates in the data

Proposed DB Structure for Generic Messages

Worth discussing: do we want one table per topic.

    temperature, 1, 'temp', 30
    temperature, 1, 'date', '10/02/23'
    temperature, 2, 'temp', 30
    temperature, 2, 'date', '10/03/23'
    temperature, 3, 'temp', 30
    temperature, 3, 'date', '10/04/23'
    temperature, 4, 'temp', 30
    temperature, 4, 'date', '10/05/23'

    nk_1, 1, 'nk', 1.01, 'rain'
    nk_1, 1, 'date', '10/02/23', 'date'
    nk_1, 2, 'nk', 0.65, 'rain'
    nk_1, 2, 'date', '10/03/23', 'date'
    nk_1, 3, 'nk', 0.29, 'mag'
    nk_1, 3, 'date', '10/04/23', 'date'
    nk_1, 4, 'nk', 1.09, 'mag'
    nk_1, 4, 'date', '10/05/23', 'date'