LLaMA

Flowchart

The below flowchart describes the flow and usage of files and data between llama2 and finetune-llama2 notebooks that uses Llama-2-7b-chat-hf model. Abbreviations appear after flowcharts.

info

TODO

• Need to write clean script for creating and saving standard keys and synonyms into DB for phase 3 data (The notebooks under docs have some code to do this but it needs to be cleaned up and made more generalised for DB)

LLAMA2 notebook (llama2.ipynb)

LLAMA2 finetuning notebook (finetune-llama2.ipynb)

Abbreviations

  Abbreviation: Definition
  LLAMA: Llama-2-7b-chat-hf
  NIST: National Institute of Standards and Technology
  HF: Hugging Face

Results

Ignore the finetuned trained LLAMA2 section since we aren't using it anymore

Finetuned base LLAMA2

The training job wandb run is here: https://wandb.ai/deven-gqc/huggingface/runs/xl0endrm/overview?workspace=user-deven-gqc

1. Fire data:

   Input:

     {"16": "{'smoke_sensor_status': 'normal', 'battery_state': 'high', 'muffling': False}", "22": "{'smoke_sensor_status': 'normal', 'battery_state': 'high', 'muffling': False}", "21": "{'smoke_sensor_status': 'normal', 'battery_state': 'high', 'muffling': False}", "15": "{'smoke_sensor_status': 'normal', 'battery_state': 'high', 'muffling': False}", "14": "{'smoke_sensor_status': 'normal', 'battery_state': 'high', 'muffling': False}", "11": "{'smoke_sensor_status': 'normal', 'battery_state': 'high', 'muffling': False}", "13": "{'smoke_sensor_status': 'normal', 'battery_state': 'high', 'muffling': False}", "12": "{'smoke_sensor_status': 'normal', 'battery_state': 'high', 'muffling': False}", "17": "{'smoke_sensor_status': 'normal', 'battery_state': 'high', 'muffling': False}", "18": "{'smoke_sensor_status': 'normal', 'battery_state': 'high', 'muffling': False}", "20": "{'smoke_sensor_status': 'normal', 'battery_state': 'high', 'muffling': False}", "19": "{'smoke_sensor_status': 'normal', 'battery_state': 'high', 'muffling': False}", "NK": 1678406410187, "changedKey": "time_stamp", "message_id": "2023-09-25 13:51:13.438619"}

   Output:

     What is the most important insight in 100 words from this data? The 16 is  The Status of the smoke sensor is  normal. The Current state of the battery in the fire alarm device is  high. The State of muting or muffling in the fire alarm system is  False. The 22 is  The Status of the smoke sensor is  normal. The Current state of the battery in the fire alarm device is  high. The State of muting or muffling in the fire alarm system is  False. The 21 is  The Status of the smoke sensor is  normal. The Current state of the battery in the fire alarm device is  high. The State of muting or muffling in the fire alarm system is  False. The 15 is  The Status of the smoke sensor is  normal. The Current state of the battery in the fire alarm device is  high. The State of muting or muffling in the fire alarm system is  False. The 14 is  The Status of the smoke sensor is  normal. The Current state of the battery in the fire alarm device is  high. The State of muting or muffling in the fire alarm system is  False. The 11 is  The Status of the smoke sensor is  normal. The Current state of the battery in the fire alarm device is  high. The State of muting or muffling in the fire alarm system is  False. The 13 is  The Status of the smoke sensor is  normal. The Current state of the battery in the fire alarm device is  high. The State of muting or muffling in the fire alarm system is  False. The 12 is  The Status of the smoke sensor is  normal. The Current state of the battery in the fire alarm device is  high. The State of muting or muffling in the fire alarm system is  False. The 17 is  The Status of the smoke sensor is  normal. The Current state of the battery in the fire alarm device is  high. The State of muting or muffling in the fire alarm system is  False. The 18 is  The Status of the smoke sensor is  normal. The Current state of the battery in the fire alarm device is  high. The State of muting or muffling in the fire alarm system is  False. The 20 is  The Status of the smoke sensor is  normal. The Current state of the battery in the fire alarm device is  high. The State of muting or muffling in the fire alarm system is  False. The 19 is  The Status of the smoke sensor is  normal. The Current state of the battery in the fire alarm device is  high. The State of muting or muffling in the fire alarm system is  False. The Unix time at which the reading was collected is  1678406410187. The message id is  2023-09-25 13:51:13.438619. The latitude is  37.527547. The longitude is  -77.45437.
     The most important insight from this data is that all the fire and smoke sensors are in good condition and have a high battery level. The system is not muting or muffling any alerts for any of the sensors. The readings were collected at 2023-09-25 13:51:13.438619 UTC. The device is located at latitude 37.527547 and longitude -77.45437.

2. Weather data:

   Input:

     {"dateutc": 1678488300000, "tempf": null, "humidity": null, "windspeedmph": null, "windgustmph": null, "maxdailygust": null, "winddir": null, "uv": null, "solarradiation": null, "hourlyrainin": null, "eventrainin": 258.012, "dailyrainin": null, "weeklyrainin": null, "monthlyrainin": null, "totalrainin": null, "battout": 1, "humidityin": 48, "baromrelin": 30.224, "baromabsin": 29.698, "feelsLike": null, "dewPoint": null, "feelsLikein": 76.2, "dewPointin": 55.4, "lastRain": "2023-03-02T18:01:00.000Z", "NK": 76.6, "changedKey": "tempinf", "message_id": "2023-09-25 14:36:51.333359"}

   Output:

     What is the most important insight in 100 words from this data? The Unix time at which the reading was collected is  1678488300000. The Outdoor Temperature (Fahrenheit) is  None. The Outdoor humidity is  None. The Wind Speed (average wind speed in MPH) is  None. The Wind gust (peak wind speed in MPH) is  None. The Maximum daily value of wind gust is  None. The Direction of Wind is  None. The UV Index is  None. The Solar radiation (Watt/m^2) is  None. The Last 10 minutes of rainfall, multiplied by six (10 minutes x 6 = 1 hour) is  None. The Total rainfall during the current event (in) is  258.012. The Rainfall since midnight (00:00) is  None. The Calendar week total and resets on Sunday morning at midnight is  None. The Calendar month total and resets on the first day of the month is  None. The Running total since the station was last powered up is  None. The Battery Indicator (1 -> Battery available) is  1. The Indoor Humidity is  48. The Relative Barometric Pressure is  30.224. The Absolute Barometric Pressure is  29.698. The Outside feels like temperature (Fahrenheit) is  None. The Outside Dew Point is  None. The Inside feels like temperature (Fahrenheit) is  76.2. The Inside Dew Point is  55.4. The Time of last rain is  2023-03-02T18:01:00.000Z. The Indoor Temperature (Fahrenheit) is  76.6. The message id is  2023-09-25 14:36:51.333359. The location is  37.534663. The latitude is  37.534663. The longitude is  -77.4247964.
     The most important insight from this data is that the device is not collecting temperature data. The last temperature reading was taken on March 2, 2023, at 18:01:00.000 Z, and the device has been inoperative since then. Additionally, the device is not providing other environmental data such as humidity, wind speed, or wind direction. The device is, however, tracking rainfall and the calendar week and month total rainfall. The device's battery indicator indicates that the device is powered up, but it is not functioning properly.

3. EQ data: Note here is that the standardization of mag key was not correct.

   Input:

     {"time": "2023-03-10 00:54:48.811953408", "NK": 1.1026406882929414, "changedKey": "mag", "message_id": "2023-09-25 14:39:56.985513"}` from `ai3

   Output:

     What is the most important insight in 100 words from this data? The The time at which reading was collected is  2023-03-10 00:54:48.811953408. The Calendar month total and resets on the first day of the month is  1.1026406882929414. The message id is  2023-09-25 14:39:56.985513. The type is  SOS. The technology is  GPS. The coordinates is  The latitude is  37.5284945. The longitude is  -77.4493453.
     The most important insight from this data is that there was an SOS signal transmitted on September 25, 2023, at 14:39:56.985513 UTC. The signal was collected at 2023-09-25 14:43:58.811953408. The coordinates of the signal are latitude 37.5284945 and longitude -77.4493453. The signal type is SOS, indicating a distress signal. The technology used to transmit the signal is GPS.

Finetuned trained LLAMA2

The training job wandb run is here: https://wandb.ai/deven-gqc/huggingface/runs/5sx3a1ly/overview?workspace=user-deven-gqc

1. Fire data: Tested case with and without NK keys. Below example is with NK. The case without NK key gave an output where instead of summary, the answer repeated the question again until max token number.

   Input:

     {"16": "{'smoke_sensor_status': 'normal', 'battery_state': 'high', 'muffling': False}", "22": "{'smoke_sensor_status': 'normal', 'battery_state': 'high', 'muffling': False}", "21": "{'smoke_sensor_status': 'normal', 'battery_state': 'high', 'muffling': False}", "15": "{'smoke_sensor_status': 'normal', 'battery_state': 'high', 'muffling': False}", "14": "{'smoke_sensor_status': 'normal', 'battery_state': 'high', 'muffling': False}", "11": "{'smoke_sensor_status': 'normal', 'battery_state': 'high', 'muffling': False}", "13": "{'smoke_sensor_status': 'normal', 'battery_state': 'high', 'muffling': False}", "12": "{'smoke_sensor_status': 'normal', 'battery_state': 'high', 'muffling': False}", "17": "{'smoke_sensor_status': 'normal', 'battery_state': 'high', 'muffling': False}", "18": "{'smoke_sensor_status': 'normal', 'battery_state': 'high', 'muffling': False}", "20": "{'smoke_sensor_status': 'normal', 'battery_state': 'high', 'muffling': False}", "19": "{'smoke_sensor_status': 'normal', 'battery_state': 'high', 'muffling': False}", "NK": 1678406420337, "changedKey": "time_stamp", "message_id": "2023-09-25 15:02:20.288569"}

   Output:

     What is the most important insight in 100 words from this data? The 16 is  The Status of the smoke sensor is  normal. The Current state of the battery in the fire alarm device is  high. The State of muting or muffling in the fire alarm system is  False. The 22 is  The Status of the smoke sensor is  normal. The Current state of the battery in the fire alarm device is  high. The State of muting or muffling in the fire alarm system is  False. The 21 is  The Status of the smoke sensor is  normal. The Current state of the battery in the fire alarm device is  high. The State of muting or muffling in the fire alarm system is  False. The 15 is  The Status of the smoke sensor is  normal. The Current state of the battery in the fire alarm device is  high. The State of muting or muffling in the fire alarm system is  False. The 14 is  The Status of the smoke sensor is  normal. The Current state of the battery in the fire alarm device is  high. The State of muting or muffling in the fire alarm system is  False. The 11 is  The Status of the smoke sensor is  normal. The Current state of the battery in the fire alarm device is  high. The State of muting or muffling in the fire alarm system is  False. The 13 is  The Status of the smoke sensor is  normal. The Current state of the battery in the fire alarm device is  high. The State of muting or muffling in the fire alarm system is  False. The 12 is  The Status of the smoke sensor is  normal. The Current state of the battery in the fire alarm device is  high. The State of muting or muffling in the fire alarm system is  False. The 17 is  The Status of the smoke sensor is  normal. The Current state of the battery in the fire alarm device is  high. The State of muting or muffling in the fire alarm system is  False. The 18 is  The Status of the smoke sensor is  normal. The Current state of the battery in the fire alarm device is  high. The State of muting or muffling in the fire alarm system is  False. The 20 is  The Status of the smoke sensor is  normal. The Current state of the battery in the fire alarm device is  high. The State of muting or muffling in the fire alarm system is  False. The 19 is  The Status of the smoke sensor is  normal. The Current state of the battery in the fire alarm device is  high. The State of muting or muffling in the fire alarm system is  False. The Unix time at which the reading was collected is  1678406420337. The message id is  2023-09-25 15:02:20.288569.
     The message type is  2023-09-25 15:02:20.288569. The message is  2023-09-25 15:02:20.288569. The message is  2023-09-25 15:02:20.288569. The message is  2023-09-25 15:02:20.288569. The message is  2023-09-25 15:02:20.288569. The message is  2023-09-25 15:02:20.288569. The message is  2023-09-2

2. Weather data: Need to figure out why it gives blank output only for weather data.

   Input:

     {"dateutc": 1678488300000, "tempf": null, "humidity": null, "windspeedmph": null, "windgustmph": null, "maxdailygust": null, "winddir": null, "uv": null, "solarradiation": null, "hourlyrainin": null, "eventrainin": 258.012, "dailyrainin": null, "weeklyrainin": null, "monthlyrainin": null, "totalrainin": null, "battout": 1, "tempinf": 76.6, "humidityin": 48, "baromrelin": 30.224, "baromabsin": 29.698, "feelsLike": null, "dewPoint": null, "feelsLikein": 76.2, "lastRain": "2023-03-02T18:01:00.000Z", "NK": 55.4, "changedKey": "dewPointin", "message_id": "2023-09-25 15:04:13.835468"}

   Output:

     What is the most important insight in 100 words from this data? The Unix time at which the reading was collected is  1678488300000. The Outdoor Temperature (Fahrenheit) is  None. The Outdoor humidity is  None. The Wind Speed (average wind speed in MPH) is  None. The Wind gust (peak wind speed in MPH) is  None. The Maximum daily value of wind gust is  None. The Direction of Wind is  None. The UV Index is  None. The Solar radiation (Watt/m^2) is  None. The Last 10 minutes of rainfall, multiplied by six (10 minutes x 6 = 1 hour) is  None. The Total rainfall during the current event (in) is  258.012. The Rainfall since midnight (00:00) is  None. The Calendar week total and resets on Sunday morning at midnight is  None. The Calendar month total and resets on the first day of the month is  None. The Running total since the station was last powered up is  None. The Battery Indicator (1 -> Battery available) is  1. The Indoor Temperature (Fahrenheit) is  76.6. The Indoor Humidity is  48. The Relative Barometric Pressure is  30.224. The Absolute Barometric Pressure is  29.698. The Outside feels like temperature (Fahrenheit) is  55.4. The Outside Dew Point is  None. The Inside feels like temperature (Fahrenheit) is  76.2. The Time of last rain is  2023-03-02T18:01:00.000Z. The message id is  2023-09-25 15:04:13.835468.
     **blank space instead of output**

3. EQ data: Note here is that the standardization of mag key was not correct.

   Input:

     {"time": "2023-03-10 00:54:48.811953408", "NK": 1.1026406882929414, "changedKey": "mag", "message_id": "2023-09-25 14:48:03.111072"}`

   Output:

     What is the most important insight in 100 words from this data? The The time at which reading was collected is  2023-03-10 00:54:48.811953408. The Calendar month total and resets on the first day of the month is  1.1026406882929414. The message id is  2023-09-25 14:48:03.111072.
     The most important insight from this data is that the time at which reading was collected is 2023-03-10 00:54:48.811953408, which suggests that the data was collected on March 10, 2023. This is important because it provides context for the other data points in the dataset, such as the calendar month total and message ID, which can be used to understand the reading habits of the user on that particular day.

Sentimental analysis

• We have ventured on the ways to utilize sentimental analysis using GPT 3.5 for AI3 messages.
• We haven't implemented any of it in the code but there are some examples of prompts we tried by providing it the output from sentencification process and also the LLaMA responses to find what works best.

Examples

• Find the conversation on OpenAI here - openai.com/sentimental_analysis

• Other sample Q & A we tried are:

  1. Sample using LLM response:
    Please analyze the sentiment of the following message:  The most important insight from this data is that the device is not collecting temperature data. The last temperature reading was taken on March 2, 2023, at 18:01:00.000 Z, and the device has been inoperative since then. Additionally, the device is not providing other environmental data such as humidity, wind speed, or wind direction. The device is, however, tracking rainfall and the calendar week and month total rainfall. The device's battery indicator indicates that the device is powered up, but it is not functioning properly.
  
    Response: The sentiment in this message is "concerning" or "problematic."
  
  2. Sample using sentencified output:
    Please provide the important takeaway from this data in 100 tokens and then analyze sentiment of your response in 20 tokens: The time at which reading was collected is  2023-03-10 00:54:48.811953408. The Calendar month total and resets on the first day of the month is  1.1026406882929414. The message id is  2023-09-25 14:39:56.985513. The type is  SOS. The technology is  GPS. The coordinates is  The latitude is  37.5284945. The longitude is  -77.4493453.
  
    Response:
    Important takeaway: SOS signal sent via GPS with coordinates (37.5284945, -77.4493453) on 2023-09-25.
  
    Sentiment: Concerning and urgent information.