Designing trustworthy AI: Principles for effective human-AI teaming in finance
We share key human-AI collaborative design principles to help organizations safely deploy agentic AI in financial environments with trust, transparency, and control.
Deploying AI in high-stakes environments like finance — where accuracy, speed, and trust are critical — requires safe and effective human collaboration. Rather than pursuing fully automated processes, the objective is to implement Human-AI Teaming.
Human-AI Teaming is a socio-technical approach that identifies and designs specific tasks and interaction points where AI can best augment human expertise. This approach directly addresses the core industry challenges of accountability and trust. When an AI’s reasoning is opaque or its actions are irreversible, it cannot be a viable partner. This is especially important when working with agents, or advanced AI systems that leverage large language models (LLMs) to autonomously navigate and complete complex workflows.
The following principles outline design practices for an AI teammate that remains transparent, accountable, and subordinate to human expertise, ensuring users can reliably manage resources and maintain oversight.
Design principles for AI teammates
Transparent operations
To address the pervasive issue of the black box AI, the system’s operations must be transparent and easily interpretable. Rather than providing opaque, one-shot outputs, AI should clearly demonstrate reasoning through incremental, step-by-step processes. Users may not want to read lengthy text explanations, so visual formats, like graphs, decision trees, and process flow diagrams can help maintain engagement and make AI reasoning easier to understand.
A flow diagram to illustrate AI reasoning steps
This is especially important for agentic AI, which can follow its own processes autonomously and often in the background. Since these actions may not be immediately apparent to the user, clear visibility into the AI’s reasoning is critical to maintaining trust and accountability.
Shared context
AI systems should operate within the same context as their human team members using the same inputs, tools, and resources; their actions must be understandable and reproducible. When referencing technical or specialized information, AI should provide clear citations or links, enabling users to verify sources and trace conclusions. Context becomes especially critical in agentic AI systems, which can autonomously access and use external sources. Users must be able to see and operate tools or data sources the AI interacts with, be it APIs, databases, plugins, or even other agents. No tool should exist exclusively for AI to use.
Full agency and control
Recognizing that AI systems can and will make mistakes, it’s important to prioritize user control and agency. Users should be able to reverse or modify AI-driven actions, adjust instructions dynamically, and completely revert to manual mode. AI systems should always remain interruptible, allowing users immediate intervention during any step of a process to prevent errors or correct deviations. Critically, AI must never autonomously enact changes without explicit user confirmation, making sure that users check the work, and retain ultimate decision-making authority.
Unobtrusive presence
The presence of AI within workflows must be supportive rather than intrusive. Effective AI systems should integrate into the user’s natural workflow without causing disruption or distraction. Interaction with AI should be efficient, quickly invoked, and easily dismissed, with minimal impact on screen space and cognitive load. AI solutions can incorporate intuitive design patterns such as in-line overlays, side panels, or generative grids to display information without forcing users to switch tasks or tabs.
In-situ AI layouts
An unobtrusive design maintains workflow continuity, increases productivity, and enhances overall user satisfaction.
Financial AI you can trust
Implementing these overarching design principles ensures that financial AI acts as a true teammate. By combining transparency, shared context, user control, and unobtrusive integration, organizations can unlock the productivity and efficiency benefits of agentic AI without compromising accountability or trust. This balance is essential in financial services, where even small errors can have outsized consequences and confidence in decision-making is paramount.