Assessing LLMs Responses in the Field of Domestic Sustainability: An Exploratory Study

In the next years, we must challenge climate change, and the urgency of adopting a more sustainable lifestyle has increased. Conversational Agents, such as Smart home Personal Assistants, have shown promise in fostering sustainable behaviors in domestic environments. However, traditional conversations with rule-based approaches in such agents face challenges in addressing users' questions in complex domains like sustainability. Large Language Models (LLMs) are a promising tool to overcome these limitations of their capability to answer open-domain questions. The final objective of this work is to compare the generative capabilities of four large language models in ecological sustainability to determine the most suitable LLM to be embedded into home assistants and create a hybrid model of conversational agent for environmental sustainability. We performed two evaluations. In the former, we constructed a set of trustable sources on the topic and analyzed the extent to which the themes covered in the text generated by the models appeared in it. The results do not show a statistical difference between the outputs of the candidate models, while qualitative analysis determined that ChatGPT, at the moment, is the optimal solution. In the second evaluation, we tested the responses generated by ChatGPT on a corpus of 167 questions from a sample of 75 people. Responses evaluation was performed by a team of experts (N=5) on fluency, coherency, consistency, accuracy, and reasoning. The results suggest that ChatGPT for generic questions on sustainability is quite reliable.