What Are the Current Limitations for AI in the Finance Industry?

The adoption of artificial intelligence has increased exponentially in the past few years, with many companies starting to work with AI to a certain extent. While there have been many in the finance industry that claims to utilize AI as part of everyday business operations, there are clear hurdles that need to be overcome if AI is to be more widely integrated into the industry and play a more productive role within it. Some of the major challenges for the wider adaptation of AI within the finance industry are-

1. Lack of usable financial data

2. The difficulty in AI transparency and explainability (XAI).

Being fully aware of the issues that hinder the adaptation of AI in finance, Qraft Technologies has been able to develop solutions that can fully take advantage of what AI can offer despite the challenges that exist.

It may seem like an oxymoron to suggest that there isn’t enough financial data out there for artificial intelligence to run on. However, this is most definitely the case.

According to the New York Times, the amount of time data scientists spend sorting out bad data, a process they refer to as ‘data wrangling’, amounts to somewhere between 50-80% of their entire time worked[1]. In a shocking study, IBM estimated that bad data cost US-based companies $3.1 trillion in 2016[2]. This is possible as an article in the Harvard Business Review suggests that it costs 10 times more[3] to complete work in which the data is defective or lacking in some form or another. Whether the data input is incomplete due to human error or negligence, the problem is rampant and difficult to address.

The ushering of the new age of data science has also given rise to ‘noisy data’. Noisy data refers to data points that have increased in number in the past decade but must be sorted out due to their poor value. Because so many data points now exist, metrics based on this noise have proliferated as well. Noisy data may occur from the simple fact that in this age of ‘big data, data is created just for data’s sake.

One illustration of the consequences of the lack of usable data can be shown with the difficulties in financial modelling. When creating functions that especially depict non-linear relationships, the main problem that arises is the problem of overfitting or underfitting.

Overfitting refers to the fact that relationships and correlations made with existing data (training data) may not hold when new data points (testing data) are introduced. It suggests low bias (because of its success with training data) but a high variance in results (as it has difficulty with testing data). Underfitting occurs when no verifiable clear relationship can be found with existing data. It signifies high bias and high variance in results.

So, any data cannot just be added to AI, nor can it train with insufficient data. How can we handle this situation? This is the point where financial experts can fill in the gaps. They can make inputs in adjusted forms with financial knowledge and insights to augment insufficient data through imputation, simulation, or any other method that makes sense. 

Qraft’s Kirin API[4] is the crucial first step that is used to tackle the issue of poor data. It can cut in time the number of hours spent pre-processing financial data, by ensuring the elimination of various biases that can skew data. It can easily sort data by specific investment universes, giving decision-makers an easy way to filter data on demand. The ability can allow human managers to easily see trends and insights that the data may show when formulating their strategies.

With this support, we can better utilize AI, Machine Learning, and Deep Learning. In other words, AI’s current role in asset management is to be a tool used by asset managers, helping them to realize their intuitive strategy optimally with scalability.

One of the biggest hurdles left for AI to overcome is the classic problem faced by many paradigm-shifting technologies- that of explainability.

This is especially true of developments concerning AI due to its complex nature. In its 2021 Business & Finance Outlook[5], the OECD highlighted “The explainability conundrum” as one of the major challenges faced by the deployment of AI in finance. It highlights the gap in technical literacy between many of those working in finance in being able to understand and explain to consumers, as well as the gap that exists for regulators and lawmakers that need to create the guidelines to make sure AI is ethically used.

Qraft’s Alpha Factory- composed of the Factor Factory[6] and Strategy Factory[7] can address both the issues of poor data and AI explainability, as it allows human input to fill in the gaps that exist in financial data, as well as give humans a say in portfolio construction. This would allow human managers to have an influence by selecting factors such as but not limited to investment universe, goals, horizon, and features. Human managers will therefore have the final say and thereby enable managers to answer for themselves the reasons as to why they made certain decisions, alleviating some of the concerns about AI explainability and accountability.

A complete black box will not be acceptable to many stakeholders, giving way to the necessity for more transparency. Some argue that the ‘black box’ created by AI is no different from that made by complex high mathematical financial models. This is true to an extent, but different, as some AI models cannot be explained at all. In some jurisdictions (Germany), AI models that are not explainable are prohibited.  But this must also be balanced with the need to protect IP and to be mindful of not constraining innovation with explainability restraints.

One solution comes from the OECD, by way of the OECD AI Principles[8], which in general promotes a more human-centric approach to AI, in which humans remain the final decision maker. Given the early stage of this promising technology, and the various stakeholders involved, it remains to be seen whether a comprehensive framework will be agreed to in the near future. However, some general framework will be needed if AI is to achieve its potential in the financial industry.

[1] Lohr, Steve. “For Big-Data Scientists, 'Janitor Work' Is Key Hurdle to Insights.” The New York Times, The New York Times, 18 Aug. 2014, https://www.nytimes.com/2014/08/18/technology/for-big-data-scientists-hurdle-to-insights-is-janitor-work.html.  

[2] Redman, Thomas C. “Bad Data Costs the U.S. $3 Trillion per Year.” Harvard Business Review, 4 Oct. 2017, https://hbr.org/2016/09/bad-data-costs-the-u-s-3-trillion-per-year. 

[3] Redman, Thomas C. “Assess Whether You Have a Data Quality Problem.” Harvard Business Review, 9 Mar. 2018, https://hbr.org/2016/07/assess-whether-you-have-a-data-quality-problem.  

[4] Kirin API - Developed by Qraft’s data scientists, integrates multiple vendors to provide both macroeconomic and company fundamentals with the correct point-in-time data.

[5] OECD (2021), OECD Business and Finance Outlook 2021: AI in Business and Finance, OECD Publishing, Paris, https://doi.org/10.1787/ba682899-en.

[6] Factor Factory – Qraft’s core AI technology that automatically finds factors that could bring excess returns. Factor Factory can produce at least 10 factors per day without any human intervention.

[7] Strategy Factory- extracts the investment strategy with a nonlinear asset price model by nonlinearly combining factors extracted automatically from the factor factory.

[8] “Artificial Intelligence, Machine Learning and Big Data in Finance.” OECD, OECD, 11 Aug. 2021, https://www.oecd.org/finance/artificial-intelligence-machine-learning-big-data-in-finance.htm.