The dual role of AI in financial services: a weapon and shield

AI is both a powerful tool for defenders and a potent weapon for attackers. Joseph Carson, Chief Security Evangelist & Advisory CISO at Segura, explores its dual role in financial services.

Artificial Intelligence (AI) is transforming financial services at a pace that few other technologies have matched. From customer service automation to real-time fraud detection, AI has become indispensable to banks, insurers, and fintechs. Yet the same technology that optimises efficiency and security is increasingly weaponised by cybercriminals to exploit trust, manipulate identities, and scale financial crime.

This dual role of AI as both a powerful tool for defenders and a potent weapon for attackers defines one of the most critical battlegrounds in today’s financial sector. To navigate it effectively, leaders must understand not only how AI is evolving but also how to manage its risks, ethics, and security implications, especially around identity.

The evolution of AI: from promise to peril

AI’s role in financial services has evolved dramatically over the past decade:

• Early AI (2000s–2010s): Machine learning was first used for credit scoring, algorithmic trading, and basic anomaly detection. The focus was narrow but powerful, spotting outliers in transaction data to flag potential fraud.
• Modern AI (2020s): The rise of deep learning and large language models (LLMs) has unlocked broader applications. AI now powers conversational banking assistants, personalised investment advice, real-time risk scoring, and automated compliance checks.
• Generative AI: The latest wave has introduced new opportunities and risks. Attackers now use AI to generate synthetic identities, create convincing phishing campaigns, and even clone voices or video for social engineering attacks.

From my experience in the field, AI has lowered the barrier for attackers, and today, an attacker simply needs a laptop and an internet connection. Even language-based defences are no longer sufficient. For example, relying on linguistic cues, such as assuming emails written in Estonian are safe from phishing, is no longer adequate. AI can generate convincingly localised content in seconds, making these traditional protections nearly obsolete.

This evolution highlights the paradox of AI: the same sophistication that allows defenders to protect customers can also be exploited to deceive them.

AI as a weapon: the dark side of innovation

In financial services, AI-driven attacks are growing in frequency and sophistication. Some of the most concerning trends include:

1. Deepfake scams and synthetic identities
   Criminals use AI to generate realistic fake voices, videos, and documents. Fraudsters can impersonate executives in real-time during video calls or create entirely synthetic customers to bypass onboarding checks.
2. Phishing and social engineering at scale
   AI-driven text generation produces highly personalised phishing emails free from the telltale grammar mistakes of older scams. Attackers can tailor messages to specific financial institutions, products, or even individual employees. From my experience, AI allows attackers to quickly analyse stolen datasets, extracting financial insights, social relationships, or behavioural patterns, which makes targeted phishing far more effective than ever before.
3. AI-powered negotiation and automation
   Attackers are now using AI chatbots to negotiate with victims or intermediaries in financial fraud cases, replacing humans in scenarios such as extortion or ransomware settlements. AI also automates iterative code improvements, enabling malware or fraud scripts to adapt and improve with minimal effort from human operators. These capabilities make even entry-level cybercriminals far more potent than in the past.
4. Fraudulent trading and market manipulation
   Attackers use AI to execute high-frequency trading manipulations or spread misinformation on social platforms to influence stock prices.

AI as a shield: defending the financial sector

Fortunately, AI is also the most effective defence against these threats. Banks and fintechs are deploying advanced AI to protect customers and infrastructure:

1. Real-time fraud detection
   AI analyses vast amounts of transaction data in milliseconds, identifying anomalies that human analysts would miss. These models continuously adapt to detect new fraud tactics.
2. Threat intelligence and SOC augmentation
   AI assists security operations centres (SOCs) by triaging alerts, correlating threat intelligence feeds, and even predicting emerging attack vectors.
3. Customer protection
   Virtual assistants powered by AI help customers detect fraud, report suspicious activity, and receive guidance faster than human-only call centres.

While AI enables stronger defences, I have observed firsthand that attackers can adapt faster than ever. This makes continuous monitoring, behavioural analytics, and identity-centric security essential, especially as AI becomes more autonomous in both attack and defence scenarios.

The rise of shadow AI in financial services

While sanctioned AI initiatives drive innovation, ’shadow AI’ poses a hidden risk. This refers to employees using unauthorised AI tools, such as generative AI chatbots or unvetted machine learning models, for work-related tasks.

In financial services, shadow AI is especially dangerous:

• Data leakage: Employees may inadvertently input sensitive customer or transaction data into unsecured AI platforms.
• Model poisoning: Using unverified AI outputs in risk assessments or compliance reports could introduce errors or bias.
• Identity risks: Attackers may exploit unsecured AI accounts or shadow deployments to gain unauthorised access.

Shadow AI reflects the tension between innovation and control. Leaders must balance empowering teams with maintaining governance and oversight. From my perspective, shadow AI is becoming a significant blind spot: attackers often exploit unsecured AI endpoints as entry points into larger systems, making identity security an even higher priority.

The ethics of AI in finance

AI’s growing role raises ethical challenges that cannot be ignored. In financial services, key concerns include:

1. Bias and fairness
   AI models trained on biased data risk unfair outcomes, for example, denying loans to certain demographics. Regulators are increasingly scrutinising fairness in AI-driven decision-making.
2. Transparency and explainability
   Financial institutions must ensure that AI-driven decisions (e.g., credit scoring, fraud detection) are explainable to customers and regulators. ’Black box’ models create compliance and trust issues.
3. Accountability
   Who is responsible when AI fails? Financial organisations must define governance structures to manage AI accountability and oversight.
4. Human-AI collaboration
   AI should augment, not replace, human decision-making in high-stakes areas like fraud investigations or lending. Maintaining a human in the loop is both an ethical and regulatory necessity.

Preparing for the dual role of AI: best practices

Financial leaders must adopt a proactive approach to balance AI’s promise and peril. Key best practices include:

1. Identity security as a foundation

Protecting AI begins with securing access. Identity security is critical because:

• AI models as assets: Generative AI systems and fraud detection engines are high-value targets for attackers. Compromise could mean stolen intellectual property, poisoned models, or manipulated outputs.
• Access controls: Implement least-privilege access to AI platforms. Ensure only authorised users can train, query, or deploy models.
• Multi-factor authentication (MFA): Require MFA for all AI-related accounts to prevent credential theft.
• Continuous monitoring: Use identity intelligence to detect suspicious logins or access patterns to AI systems.

From my experience, neglecting identity security around AI is one of the fastest ways organisations can turn their most valuable defences into liabilities.

2. Establish AI governance

Define clear policies for how AI can and cannot be used. This includes approved tools, risk assessment frameworks, and shadow AI reporting channels.

3. Invest in AI-powered defences

Just as attackers use AI to innovate, defenders must do the same. Continuous investment in fraud detection, behavioural biometrics, and SOC augmentation is essential.

4. Train employees to spot AI threats

Humans remain the weakest link. Regular training should cover AI-enabled phishing, deepfake awareness, and safe AI usage practices.

5. Build ethical AI frameworks

Create cross-functional committees, including compliance, IT, and business units, to evaluate bias, fairness, and explainability in AI systems.

6. Test resilience with red teaming

Simulate AI-driven attacks through red teaming to identify vulnerabilities in fraud detection, customer onboarding, and market surveillance.

7. Collaborate across the industry

Fraudsters don’t work in silos, and neither should defenders. Financial institutions should participate in industry threat intelligence sharing initiatives to stay ahead of AI-driven scams.

Privileged Access Management (PAM) for AI systems

In addition to general identity security, managing privileged accounts is essential when protecting AI platforms and critical financial systems. Privileged Access Management (PAM) ensures that accounts with elevated permissions, such as administrators, AI model trainers, or SOC operators, are tightly controlled, monitored, and auditable.

Key considerations include:

• Least privilege enforcement: Grant users only the access necessary to perform their roles. For AI systems, this means restricting access to model training environments, data repositories, and deployment pipelines. Attackers often target privileged accounts to manipulate models or exfiltrate sensitive information.
• Just-in-Time (JIT) access: Provide temporary privileged access when needed, rather than permanent elevated rights. This limits exposure if an account is compromised.
• Session monitoring and recording: Monitor privileged sessions in real-time and maintain detailed audit logs. This enables rapid detection of anomalous behaviour, such as unexpected queries or data downloads from AI systems.
• Credential vaulting: Store credentials for privileged accounts in a secure vault with enforced rotation policies. Hard-coded passwords or shared credentials are a common attack vector that AI-driven attacks can exploit.
• Integration with identity intelligence: Combine PAM with identity analytics to detect unusual privilege escalations or access patterns. For example, if an AI operator suddenly accesses a model they have never used before, automated alerts can trigger further investigation.

From my experience, PAM is often the first line of defence against AI-targeted attacks. Attackers increasingly attempt to compromise administrator-level accounts to manipulate AI outputs, access sensitive datasets, or pivot into other critical systems. Implementing PAM effectively ensures that even if attackers breach lower-level accounts, elevated privileges, and by extension, your most sensitive AI resources remain protected.

The future of AI in financial services

The financial sector is entering an era where AI will define competitive advantage but also determine resilience against fraud and cybercrime. The dual role of AI will continue to evolve:

• Attackers will refine deepfakes, synthetic identities, automated negotiations, and adaptive malware.
• Defenders will rely on AI for adaptive fraud detection, predictive intelligence, and customer protection.
• Regulators will demand higher standards of transparency, ethics, and accountability.

Ultimately, success will depend on balance: embracing AI’s value while securing its risks. Identity, trust, and resilience must be defended alongside innovation.

Conclusion

AI is no longer just another tool in financial services; it is the defining technology of the era. Its dual role as a weapon and a shield creates both unprecedented opportunities and risks.

The path forward requires vigilance, ethics, and above all, identity security. Protecting access to AI ensures that its benefits are not hijacked by adversaries. By investing in governance, defences, and industry collaboration, financial institutions can thrive in this new landscape, turning AI into a competitive advantage while keeping fraudsters at bay.

From my personal experience, the stakes are higher than ever: even small mistakes in AI governance or identity security can be exploited within minutes, making preparation and foresight essential. AI is both the battleground and the arsenal. How organisations manage it will define their success for years to come.

 

About author

Joseph Carson is an award-winning cybersecurity professional and ethical hacker with more than 30 years’ experience in enterprise security, specialising in blockchain, endpoint security, network security, application security & virtualisation, access controls, and privileged access management. Joe is a Certified Information Systems Security Professional (CISSP) and Offensive Security Certified Professional (OSCP), an active member of the cybersecurity community frequently speaking at cybersecurity conferences globally, often being quoted, and contributing to global cybersecurity publications. At the moment, Joseph is Chief Security Evangelist and Advisory CISO at Segura.