In January 2024, a finance employee at the Hong Kong office of Arup — the global engineering firm — attended what appeared to be a routine video conference with the company’s CFO and other senior executives. The call was compelling: familiar faces, recognizable voices, normal business context. Over the course of the scheme, the employee executed 15 wire transfers totaling HK$200 million — approximately US$25.6 million. Hong Kong police reported that the other participants on the call were impersonated via deepfake. Police announced the case in February 2024; Arup subsequently confirmed in May 2024 via press statements to CNN, the Financial Times, and others that it was the victim.

This incident is not an outlier. It is a data point in a trend.

The Cost Collapse

Traditional impersonation attacks had a hard floor on quality. Forging someone’s voice or video well enough to deceive a trained employee required expensive studios, skilled actors, and post-production resources. That floor has collapsed.

Voice cloning today requires a sample of a few seconds — sometimes available from a conference recording, a podcast appearance, or a public YouTube video — and commodity compute time. Services offering voice cloning APIs are widely available, with free tiers sufficient to generate test clips. The barrier to producing a plausible clone of a CEO’s voice for a brief targeted phone call is now measured in minutes and dollars, not days and thousands.

Deepfake video generation has followed the same trajectory. Models capable of real-time face-swapping now run on consumer hardware. The quality threshold required to deceive someone in a live video call is lower than it appears: most corporate video calls have compression artifacts, variable lighting, and connection latency — conditions that mask generation artifacts. Attackers are not trying to produce Hollywood-quality deepfakes. They are trying to produce good enough fakes for a five-minute call in a degraded environment.

The Arup attack required deepfakes of multiple executives simultaneously, a level of sophistication that reflects dedicated adversarial effort. But the trajectory of capability means that tomorrow’s attackers will achieve similar results with fewer resources, less planning time, and lower technical expertise.

Voice Cloning in the Attack Chain

Voice cloning attacks occupy a specific position in the business email compromise (BEC) ecosystem. Traditional BEC relies on impersonating executives via email — spoofed headers, lookalike domains, social authority. Adding a voice call shifts the attack to a higher-trust channel.

The attack pattern: an email arrives requesting an urgent wire transfer or credential action, attributed to a senior executive. When the target calls the number in the email to verify — a common security practice — they reach an attacker using a real-time voice clone of the executive. The verification step, intended as a defense, becomes confirmation of the fraud.

This dynamic matters because out-of-band verification is a core recommended defense against BEC. Voice cloning doesn’t just enable new attacks — it degrades existing defenses.

Beyond corporate fraud, voice cloning scales to mass victimization. “Grandparent scams” — where attackers impersonate a grandchild in distress and request emergency wire transfers — have existed for decades. AI voice cloning allows these attacks to use a voice that genuinely resembles the real grandchild, synthesized from social media audio. The personalization threshold has dropped below what humans can reliably detect under emotional duress.

Deepfake Video: Impersonation in High-Fidelity Channels

The Arup incident illustrates a key capability escalation: deepfake video makes high-trust channels attackable. Video conference calls carry an implicit authenticity assumption that email cannot — seeing a familiar face activate pattern-recognition instincts that were calibrated on a world where real-time video manipulation was not possible.

Corporate executive impersonation is the high-value target, but the attack surface is broader. Know Your Customer (KYC) processes — used by financial institutions, cryptocurrency exchanges, and regulatory bodies to verify identity — frequently involve video submission of the applicant holding a document. Deepfake generation tools can synthesize video of a face (generated or stolen) meeting the liveness requirements of these checks. The attack uses AI-generated video to pass AI-assisted identity verification — a direct arms race between generation and detection.

The second application is creating entirely synthetic executive personas for social engineering. A fabricated LinkedIn profile with an AI-generated profile photo (from a generative model, not cloned from any real person), a manufactured employment history, and a small synthetic social network of connected profiles can establish enough apparent legitimacy to initiate targeted outreach. The “executive” that contacts a supplier about an urgent contract amendment doesn’t need to impersonate anyone real — it just needs to appear real enough to generate compliance.

LLM-Powered Spear Phishing

Traditional phishing is volumetric and generic. Spear phishing is targeted but expensive — it requires researching each target, crafting personalized content, and managing scale manually. LLMs change this economics.

The attack pattern is well-documented in threat intelligence reporting: scrape LinkedIn, company websites, social media, and public filings to extract the target’s role, reporting relationships, recent projects, professional interests, and organizational context. Feed this to a capable LLM to compose personalized phishing messages that reference authentic details, adopt plausible sender personas, and embed the request within an organizational context the target would recognize. Threat intelligence reporting has documented this approach, and security researchers have investigated the hypothesis: AI-personalized phishing messages may outperform generic templates in controlled evaluations, though published studies vary in methodology and real-world campaign data remains limited.

The resulting emails differ from generic phishing in both sophistication and specificity. They reference real projects, use correct internal terminology, and anticipate the target’s likely response patterns. The writing quality is uniform — no grammatical errors or syntactic awkwardness that trained users learn to notice.

Beyond single-email phishing, LLMs enable multi-turn relationship attacks: AI personas that conduct extended professional correspondence with a target over days or weeks, building trust and context before the malicious request is made. The attack converts patience into automation — a human attacker managing one long-con is replaced by an automated agent managing hundreds in parallel.

Detection: The Arms Race You’re Losing

Detection of AI-generated content — voice, video, or text — has attracted significant research and commercial investment. The fundamental problem is structural.

Detectors are trained on the output of current generation models. Generation models improve continuously. The detector trained on yesterday’s model artifacts performs poorly on tomorrow’s model outputs. This lag is a structural feature of the adversarial setting, not a temporary gap: the generator can adapt specifically to evade any known detector, while the detector can only generalize from examples it has already seen. Research in adversarial machine learning has characterized this as a fundamental asymmetry, though the exact bounds remain an open question.

Liveness detection in video KYC — asking users to blink, turn their head, or perform random gestures — was a meaningful defense when real-time video generation was slow and artifact-prone. Researchers have demonstrated that some liveness checks can be defeated with current generation tools, and security vendors have flagged this risk in their public advisories. The effectiveness of any specific implementation varies significantly — more robust deployments add device-binding, behavioral biometrics, and cryptographic attestation. But what was once considered a reliable baseline control is now an active area of adversarial research, and organizations that haven’t updated their threat models around KYC should.

Voice authentication systems — used by banks and call centers to replace password authentication — were calibrated against voice synthesis technology from several years ago. These systems can be fooled by modern voice clones, a problem that has been demonstrated in published research and in adversarial audits. Financial institutions that rely on “voiceprint” authentication without additional controls are operating on an assumption that no longer holds.

The human baseline for deepfake detection is even weaker than automated tools suggest. Human detection performance degrades under time pressure and emotional context — exactly the conditions that attackers engineer into their scenarios. A finance employee receiving an urgent request from what appears to be their CFO on a live video call is not optimizing for artifact detection. They’re processing an authority relationship under deadline pressure.

What Doesn’t Work

“Just look for the artifacts” is the most common recommended defense against deepfakes. It doesn’t scale. Generation quality is improving faster than human perceptual training. Artifacts that were reliable signals — unnatural blinking, skin texture inconsistencies, audio-visual synchronization lag — are features that current models are specifically optimized to eliminate. Training humans to detect artifacts from 2020 deepfakes does not prepare them for 2024 deepfakes.

“Verify through a different channel” breaks down when the adversary controls the verification channel. Voice cloning defeats phone verification. Deepfake video defeats video verification. If the attacker can spoof both modalities simultaneously — as the Arup case suggests — the remaining verification options require pre-established shared secrets or physical presence.

“AI can detect AI” is a research aspiration, not a reliable operational control. Commercial deepfake detection services exist, but their false positive and false negative rates in real operational conditions are not well characterized, and their accuracy against the most recent generation models is an active research question.

What Works

None of the available defenses are complete. That said, some approaches shift the economics meaningfully.

Pre-established verification protocols with rotating shared secrets. The most robust defense against voice and video impersonation is a one-time code or rotating phrase agreed upon through a separate, established channel before any high-stakes transaction. Reusable code words can themselves be compromised over time — the protocol must include rotation and ideally out-of-band confirmation. If the CFO’s policy is to always use a session-specific rotating code when authorizing large transfers, cloning their voice or video doesn’t help the attacker. This requires organizational policy and consistent enforcement.

Transaction authorization that doesn’t route through a single human. Multi-party authorization with independent verification channels reduces the blast radius of any single successful deception. Large wire transfers are a natural candidate for such controls regardless of the threat model — social engineering simply raises the urgency of implementing them.

Aggressive skepticism toward urgency as an attack signal. Virtually all high-value social engineering attacks rely on urgency — the request must be acted on immediately, before verification can happen. Organizational policies that treat urgency as a red flag, rather than a reason to bypass controls, interrupt this pattern. The policy itself is straightforward; the enforcement is the challenge.

Treating video and voice as equivalent to email for authentication purposes. This is a cultural shift as much as a technical one. The assumption that seeing someone on a video call provides meaningful authentication needs to be retired in high-value transaction contexts.

The Broader Threat Model

The AI-powered social engineering threat has a property that distinguishes it from most technical vulnerabilities: the target is human cognition, not software. A software vulnerability can be patched. Human trust mechanisms evolved for a world where real-time audio-visual impersonation was physically impossible and social proof had meaningful evidential value. Those mechanisms are not patchable.

Chesney and Citron laid out this problem in a widely-cited paper — “Deep Fakes: A Looming Challenge for Privacy, Democracy, and National Security” — first circulated in 2018 and published in the California Law Review (Vol. 107) the following year. They framed deepfakes as an epistemological threat rather than a technical one. The technical defenses are catching up slowly. The organizational and cultural adjustments required are proceeding even more slowly. And the capability of attackers is advancing faster than either.

The Arup incident is significant not only for the sophistication of the deepfake — simultaneous video impersonation of multiple executives — but because it demonstrated that this capability is now within reach of criminal actors with sufficient motivation. Public reporting establishes the scale of the transfers; the specific internal controls that were circumvented have not been disclosed by Arup. What is clear from the outcome is that the attack successfully exploited a high-trust channel — live video with apparently familiar colleagues — to generate compliance. The question for organizations is not whether this capability will be used against them, but when, and what assumptions about trust and verification they are prepared to abandon in response.

The AI is not the target here. It is the weapon. Defending against it requires accepting that several properties of high-trust channels — voice recognition, visual recognition, real-time presence — no longer provide the authentication guarantees organizations have been relying on.