Here in India, it’s telling that global AI powerhouse, OpenAI has revealed that India has the second-highest number of users of ChatGPT, demonstrating the widespread use of Gen AI tools. For too long, the narrative around AI in cyber security has focused on its defensive capabilities. While AI is revolutionizing how organizations protect themselves – bringing unprecedented speed, accuracy, and automation – it’s crucial to acknowledge the other side of the coin. Cyber criminals are quickly embracing AI, using large language models (LLMs) and advanced agentic AI to craft more potent and elusive attacks.
Consider the rise of malicious LLMs like WormGPT and the more recent Xanthorox AI. These aren’t just theoretical threats; they’re platforms designed for nefarious purposes. WormGPT, based on the GPT-J model, was marketed as a “blackhat” alternative, offering features tailored for malicious activities and reportedly trained on malware-related data. While its creators have ceased operations, the genie is out of the bottle. We’re seeing a trend of offensive AI tools, including BurpGPT, PentestGPT, FraudGPT, and PassGPT, with plans for even more sophisticated models like “Evil-GPT-Web3.”
Xanthorox AI, emerging in Q1 2025, represents a significant leap. Unlike previous iterations, it’s an autonomous, modular system built from scratch, operating entirely offline for enhanced anonymity and resilience. Its five specialized AI models – Xanthorox Coder, V4 Model, Xanthorox Vision, Xanthorox Reasoner Advanced, and the Fifth Coordination Module – work in concert to automate malware development, reconnaissance, social engineering, and coordinated attacks without external oversight. This is not just an LLM; it’s an agentic AI, signifying a logical and concerning shift in the cyber criminal’s toolkit.
AI-Enhanced Phishing Breaking Brand Trust
The impact of these AI developments on brand protection is particularly acute in the realm of phishing attacks. Threat actors are already using prompt injection techniques to manipulate legitimate LLMs into generating compelling phishing content. This means phishing attacks are being launched faster, with greater frequency, and with an alarming degree of personalization.
In India, AI-powered phishing attacks have surged by 175% in the financial sector during the first half of 2024, with over 135,000 incidents reported—underscoring the rapid adoption of AI by cybercriminals targeting digital banking and e-commerce platforms.
A recent Check Point Research report highlights that Indian enterprises are now facing nearly twice the number of cyberattacks compared to the global average, with AI being a major enabler for attackers
In 2024, a staggering 67.4% of global phishing incidents involved AI tactics, with the finance industry among the top targets. This isn’t just about volume; it’s about sophistication. AI enables attackers to craft highly personalized and convincing campaigns, including spear-phishing, deepfakes, and advanced social engineering techniques.
India now ranks as the third-largest country globally for phishing attacks, with technology and finance sectors being the most targeted.
Emails are now Flawless, Fast, and Focused
One of the most immediate impacts is on phishing emails themselves. Gone are the days when grammatical errors and spelling mistakes were clear indicators of a scam. AI-generated emails are often indistinguishable from legitimate corporate communications. This speed of creation and personalization is significantly augmented by LLMs, allowing attacks to be launched and scaled rapidly.
Research from 2021, showed that even with older AI, spear-phishing emails generated by AI achieved a 60% click-through rate. Another more recent study from 2024 found that fully AI-generated phishing emails achieved a 54% click-through rate in a human subject study, a 350% increase over arbitrary phishing emails.
A chilling real-world example occurred in February 2024, when the Hungarian branch of a European retail company lost €15.5 million in a business email compromise (BEC) attack. The attackers used generative AI to create emails that perfectly mimicked the tone, style, and formatting of prior corporate correspondence, targeting financial staff with urgent money transfer requests. These emails were error-free and contextually accurate, bypassing traditional filters and highlighting the effectiveness of AI-enhanced BEC attacks.
In India, nearly 72% of organizations reported being targeted by AI-powered cyberattacks in the past year, with AI-generated phishing scams and deepfake impersonation in business emails among the most common threats.
Deepfakes Are Becoming the Ultimate Impersonation
Beyond sophisticated emails, deepfakes have added an entirely new dimension to phishing. These synthetic media, created using deep learning, can fabricate realistic images, audio, or video to impersonate individuals, fake voice messages, or simulate video calls. What was once limited to highly skilled individuals is now easily accessible, with Deloitte reporting that 25.9% of executives have experienced one or more deepfake incidents.
Consider these “in the wild” examples:
- UAE (2020) where a bank manager lost approximately $35 million after falling victim to an AI-driven phishing attack. Threat actors used deepfake voice technology to impersonate a company director, whose voice was cloned from publicly available audio samples. Spoofed emails from the “director” and a “lawyer” lent further legitimacy.
- Hong Kong (January 2024) where a multinational firm suffered a $25 million loss due to a deepfake video scam. Attackers used AI-generated deepfake videos to impersonate the company’s CFO and other employees during a video conference call, exploiting group dynamics to dispel a finance worker’s doubts.
- UK (May 2024) where in an unsuccessful but alarming attempt, attackers used a fake WhatsApp account, AI-generated voice cloning, and manipulated YouTube footage. They impersonated a CEO during a Microsoft Teams meeting, aiming to trick an agency leader into setting up a new business entity to solicit funds and personal details.
- In India, voice clones and deepfakes can now be produced for as little as ₹8, making them accessible to even low-resource attackers. These technologies have been weaponized for financial scams, impersonation, and even influencing elections.
AI Analysis Boosting Reconnaissance
AI’s role in phishing isn’t limited to content generation and deepfakes. AI-powered data analysis allows attackers to harvest and analyze vast datasets from social media, public records, and breached databases at unprecedented speeds. This facilitates highly targeted spear-phishing campaigns tailored to specific individuals or organizations.
AI can predict victim behaviors and optimize attack timing. For instance, AI can analyze communication patterns within an organization to determine the ideal moment to deploy a phishing email mimicking a CEO’s tone and style, significantly increasing success rates.
As AI models become more powerful, threat actors are effectively receiving the same upgrades for highly targeted reconnaissance. Advanced features in models like Grok 3 and ChatGPT 4.0 allow for rapid analysis of public information. This means AI can forecast high-value opportunities, crafting campaigns that exploit trends before organizations can implement defenses.
A notable incident in early 2024 involved a major Indian financial institution targeted by an AI-driven phishing campaign. Attackers used NLP and generative AI to craft spear-phishing emails mimicking the CEO’s writing style, incorporating official formatting and terminology gleaned from LinkedIn and corporate websites. These emails directed top managers to a fraudulent internal portal, granting attackers access to financial databases.
Overall, India lost ₹22,812 crore (about $2.78 billion) to digital frauds in 2024, with AI-powered phishing, deepfakes, and rural expansion driving nearly triple the losses compared to the previous year
An “in the wild” example from India in early 2024 saw a major financial institution targeted by an AI-driven phishing campaign that compromised sensitive customer data. Attackers used NLP and generative AI to craft spear-phishing emails mimicking the CEO’s writing style, incorporating official formatting and terminology gleaned from LinkedIn and corporate websites. These emails directed top managers to a fraudulent internal portal, granting attackers access to financial databases. In fact, there has been a 175% increase in financial phishing attacks in India during H1 2024, with AI playing a significant role.
We’re witnessing a significant shift, where artificial intelligence (AI) is not just a tool for defense but also a potent. Our latest ebook, “AI & Speed in Cyber,” delves into this critical shift, highlighting how AI is impacting brand protection, malware infections and the relentless surge of phishing attacks.
