Artificial intelligence models, including GPT-3, struggle to accurately interpret U.S. Securities and Exchange Commission (SEC) filings, according to a recent study. Despite the models’ proficiency in language tasks, they falter when asked to analyse complex financial documents. This limitation hinders their application in financial services, where understanding such documents is crucial.
The study tested several AI models, including OpenAI’s GPT-3, on their ability to extract key details from SEC filings. The models were asked to identify the filing company, the type of document, and the date. They also had to extract specific financial figures and understand the narrative in the management discussion section. In all tasks, the models performed poorly.
The researchers attributed this poor performance to the AI models’ lack of understanding of the structure and language of SEC filings. The models struggled with the dense, technical language and the non-linear structure of the documents. The researchers also noted that the models often misinterpreted the tone of the documents, reading neutrality as positivity.
To improve AI comprehension of SEC filings, the researchers suggested training the models on a more diverse range of financial documents. They also proposed creating a new model specifically designed to understand financial language and structure.
Go to source article: https://www.cnbc.com/2023/12/19/gpt-and-other-ai-models-cant-analyze-an-sec-filing-researchers-find.html