About the project

The Climate Committee's Report, "Transition to Low Emissions — Pathways for Climate Policy Towards 2050" (NOU 2023:25), received several hundred consultation responses. The Ministry of Climate and Environment requested a compilation of the responses, both in text, visual formats, and as aggregated data.The goal was to gain an overview of whether the content of the responses supported or opposed various recommendations, as well as to identify any errors or gaps in the report. Given the volume of responses and the need for efficiency, Menon was engaged to develop a method based on artificial intelligence (AI).

The project was carried out as a pilot to explore the use of large language models (LLMs) for text processing. We evaluated several models, including OpenAI’s GPT-4, Meta’s Llama 3, Google’s Gemini, and the Norwegian-trained NorMistral model. We chose GPT-4, as it balanced performance, usability, and quality. The model was used to summarize the consultation responses thematically and assess the stance on each recommendation: agree, disagree, or with pointing out errors/gaps. The results were structured and processed to create clear data files and visualizations.

We thoroughly validated the model's responses, focusing on contextual relevance, truthfulness, and response relevance. Through iterative testing and dialogue with the ministry, we ensured the analyses met the needs and reduced the risk of errors or hallucinations in the responses.

The results from the language models significantly contributed to streamlining and improving the quality of work with the consultation responses. The language models were useful for summarizing the responses, but also laid the groundwork for visualizing opinions and attitudes from various types of respondents. This helped the ministry’s advisors quickly get an overview of the responses and the views of different groups. The results of the project were used by the government during the preparation of a new parliamentary report on climate.

The experiences and methods from the project have been further developed, and Menon has created a prototype of a tool for a client that allows customers to summarize individual consultations and see connections between consultation responses across multiple consultations. The prototype is an example of how AI can make such analyses both efficient and repeatable in the future.

Contact information for the client can be shared upon request.