IFASS members first received two presentations on the possibilities and the limits of AI and then discussed how AI might possibly be employed in standard setting.

The first presentation was a general introduction into AI, which also includes machine learning and deep learning, and into generative AI, a subgroup of AI that consists of algorithms that use prompts or existing data to create new content. Of special interest to standard setting seemed to be the generative AI subgroup of large language models (LLMs) trained on high-volume data sets to generate, summarise and translate human-like text. Competencies of generative AI include translation, summarisation and generic content creation, as well as, to a degree, creativity. The competencies do not, however, include critical thinking and complex decision-making. Problems also include the need for (costly) constant training and "hallucination". 

The second presentation introduced the real-live example of an LLM trained to perform disclosure analysis and research on the question whether LLMs can be used to analyse sustainability reports. The results presented of an analysis of 11,000 sustainability reports to see whether the TCFD recommendations had been followed explained the possibility of requiring the LLM to include references to enable a review of the findings produced and the need for/possibility of iterative processes to refine findings. They also underlined the need for a careful review, consideration and evaluation of the results produced. Main conclusion of research across various LLMs was that many statements produced lack support and citations are inaccurate. Fluency often comes at the cost of accuracy.

In the discussion that followed the presentations the following points were raised:

• AI is here to stay, so use it and familiarise yourself with it. Not doing anything is not an option.
• Develop a strategy, governance and responsible practices when you use AI.
• Don't rely on AI without constantly reviewing the results.
• The more human expert input, the more accurate the results.
• LLMs need constant training and updating.
• LLMs can be usefully applied to repetitive, time-consuming tasks.
• LLMs can be used as intelligent search machines.

Regarding the work of standard setters it was suggested that training an LLM would be relatively easy as the literature is low in volume and very specific. It could also be used from an administrative perspective, for example for creating meeting minutes. While comment letter analysis by LLM would seem tempting, the results would always need to be reviewed, however searching for certain comments would be made much easier. Use of several LLMs trained to take different perspectives and then using a mediating model could lead to useful arguments that could trigger own thinking.

A final poll of the audience regarding the tasks the standards setters would consider using AI tools for led to the following results (most supported answers first):

• literature review of standard setting topics
• stakeholder survey results analysis
• stakeholder comment letter analysis
• disclosure and financial statement analysis
• mapping and checking the consistency of accounting or sustainability guidance and requirements
• assessing the pervasiveness of transactions across companies