As Artificial Intelligence (AI) grows in power and availability, it is becoming an essential tool for businesses. As well as its ability to automate repetitive and manual tasks, it also has the potential to offer intelligent insights to improve decision-making. 

For organisations to remain accountable and in control of these advancements, it is important for AI to be as transparent as possible. However, at present, not enough is being done to address the lack of understandability and explainability of AI, which is unsustainable in a business context. 

Lack of AI transparency

Lack of transparency in AI models is referred to as the ‘black box’ problem. Typically, the inner processes of machine learning (ML) models are obscured since only algorithmic inputs and outputs are visible. In particular, this is true of complex structures such as Deep Learning (DL) neural networks, which consist of many neural nets of multiple neurons. This is also the case with algorithms simultaneously optimising many variables to map geometric patterns.

As a consequence, it is difficult to discern how variables are being combined to reach conclusions. This results in the inability of the users of AI outputs and even model designers to understand or explain AI insights.

This lack of transparency is problematic due to many reasons, including:

• Social and ethical implications
  It is difficult for human operators to be accountable for AI-assisted decision-making when they do not understand the rationale behind insights. Additionally, an opaque system has the potential to mask bias and it will be difficult to detect the causes of errors. This is especially problematic when AI is used to make important and even life-impacting decisions in fields such as legal and healthcare. In NewVantage Partners’ 2022 Data and AI Executive Survey, only 44.1% of interviewed organisations reported that well-established data and ethics policies were in place, while just 21.6% believed that enough was being done to address this.
• Lack of scalability
  Without insight into how an AI system is producing its outputs, it will be difficult to investigate the reasons for errors or to predict algorithm performance when it is fed new data and variables. As a result, users of AI will struggle to make refinements to their systems and to scale them across different use cases to maximise benefits.
• Barrier to adoption
  Lack of understandability and explainability of AI solutions could pose a barrier for organisations hoping to adopt AI solutions or with solutions already underway. The decision-making risks associated with purchasing AI solutions can discourage and provide an obstacle to the adoption of AI solutions by business leaders. In fact, a 2022 Capital Economics report found lack of explicability and accountability to be a key barrier to adopting AI.

All of the above can lead to grave organisational costs, which might not only be purely monetary. There could be performance issues leading to operational inefficiencies and business disruptions, or even reputational damage and legal repercussions.

Increasing transparency

It is clear that there is work to be done to combat opacity in AI systems within organisations. An increasing field of research into ‘white box AI’ is emerging, which promotes a system that is understandable, justifiable and auditable by humans. This should offer insight into how models behave and which factors affect their decision-making.

From a technical standpoint, there are interpretability techniques which can create clearer observability of the inner workings of an AI system. One such way is to enforce model constraints, including limiting development to decomposable or linear models. Additionally, it is possible to adjust inputs to infer how they will affect model behaviour or to use additional explanation neural networks to probe.

However, it is insufficient for solutions to be understandable by technical experts alone. Since AI is used to carry out specific business functions within organisations, it is also necessary to remove barriers to visibility and accountability for wider stakeholders. As well as prioritising stakeholder education, this requires a culture of transparent AI, defined by the human actors involved.

A culture of transparent AI for sustainability

Transparent AI requires the active involvement of both technical and non-technical human stakeholders in the design and process of an AI journey. It is necessary to have a Human-In-The-Loop (HITL) framework in place, which incorporates meaningful human expertise into the system. Directly involving human stakeholders into the workflow provides stakeholders with increased insights into model functionality as well as introducing clear structures for accountability. 

In Nexus FrontierTech, we design our AI solutions to include a domain expert in the process, which has come to be known as the Expert-In-The-Loop (XITL) framework amongst our clients and stakeholders. With this, we incorporate expert feedback loops and use these to inform model improvements (read more about collaborative intelligence here). 

A transparent XITL architecture leads to sustainable and scalable AI initiatives. Traditionally, it can be difficult to upgrade these technologies in line with changing demands. However, when humans are an inherent part of the architecture, the technology will grow organically in line with requirements. Moreover, transparent models are more transferable, since it is easier to reuse and fine-tune them for different contexts across the business. As a result, companies are likely to realise greater returns from their AI initiatives.

To make this possible, it is necessary to have the correct tools available including User Interfaces (UIs) such as dashboards so that humans are provided with a visual way to understand and validate outputs. Companies can consider using an end-to-end AI platform to optimise efficient integration of human stakeholders into their initiatives.

Lack of transparency in AI systems can have serious ethical ramifications, impact the scalability of solutions, and present a barrier to organisational adoption of such technology. It is important to promote understandability and explainability of how AI models reach conclusions, especially in cases where important tasks and decisions are being handled. This is both a technical and a business consideration; as well as improving the interpretability of AI for developers, it is equally important for companies to have a coherent framework in place to allow visibility and feedback from all relevant stakeholders. With this focus on implementing transparent AI solutions, companies will succeed in retaining control of AI and developing sustainability for the business.