Megatrend · Cybersecurity & Digital Trust
Once AI can actually act for you, a single email can tell it to betray you
We used to worry that AI would give a wrong answer. Now AI is starting to *do* things for us — read email, pay bills, open files, send data. And once it can act, tricking it into doing something it shouldn't becomes a brand-new attack surface, born alongside AI itself. This is a story of two battlefields: protecting AI from being fooled, and using AI to fight attackers who are using AI too.
01What is it? (two sides of one coin)
Imagine you hire a brilliant personal assistant. It reads your email, books your tickets, pays your bills, logs into company systems on your behalf. That ability is hugely useful — but it also means that if someone sends a "fake note" fooling the assistant into thinking "the boss said to wire money to this account," and it believes them, the damage happens instantly. That's the heart of this node.
AI Security & Agent Guardrails is a security field that was "born" alongside the era of large language models (LLMs) and AI agents. It splits into two sides you have to understand together.
- Side 1 — securing AI itself: the AI becomes a new target with weaknesses older systems never had — "tricking it with text" (prompt injection), unlocking the rules (jailbreak), poisoning its training data (data poisoning). And the most dangerous case is when AI becomes an agent that can actually act — a hijacked agent doesn't just answer wrong, it *does* the wrong thing
- Side 2 — AI vs AI (attack vs defense): attackers use AI to write slicker phishing emails, fake voices and video (deepfake), and find vulnerabilities faster — while defenders use AI to speed up the work in the security operations center (SOC) and keep up with the game
LLM (Large Language Model) = an AI model that understands and generates language, like the one behind ChatGPT · Agent = an LLM wired up to "act" — call tools, search the web, run code, hit databases, pay money · Prompt = the instruction text you feed the model. The catch: AI struggles to tell "a command from the boss" apart from "text from an outsider" — and that's the root of the vulnerability.
On the megatrend map, this node sits under Cybersecurity & Digital Trust, defined in plain terms as "securing the AI itself — preventing prompt injection, attack-testing models (red-teaming), and limiting permissions / putting guardrails around agents." It's the youngest member of the Cybersecurity family, and still has more startups than giants.
02Why it matters — an attack surface born with AI
Normal cybersecurity is about protecting "code" — code that runs exactly as written. But AI isn't code. It's a system that "interprets language" and decides for itself. That opens a new class of weakness traditional firewalls never met. And industry standards like OWASP have already ranked these weaknesses officially.
In the 2025 edition of OWASP Top 10 for LLM apps, the number-one threat for two years running is prompt injection — sneaking "secret commands" into the email, document, or web page the AI reads, so it follows an outsider instead of its owner. And in late 2025, OWASP went so far as to publish a separate Top 10 just for agentic AI, because once a model can do work on its own, the "blast radius" of a single weakness gets enormous.
This isn't just theory. In June 2025, researchers disclosed a vulnerability called EchoLeak (CVE-2025-32711) in Microsoft 365 Copilot — the first zero-click prompt injection in a real, production AI system, with a CVSS severity score as high as 9.3. Just sending one well-crafted email (with the victim not clicking anything at all) fooled Copilot into pulling internal files and shipping the data out to the attacker's server. Microsoft has patched the hole, but it proved this kind of threat is genuinely "real and tangible."
The other side of the coin is growing just as fast — attackers weaponizing AI. In 2025, AI-driven cyberattacks are estimated to have broken past 28 million incidents, up roughly 72% from the year before. AI-generated phishing emails fooled about 54% of people into clicking, versus around 12% for traditional phishing — and deepfake incidents jumped year over year.
Put simply: we've just opened the door for AI to work in place of people across organizations worldwide, and on the other side attackers have just gotten AI tools to speed up their attacks. This node is the "armor lining" that has to exist for the whole AI wave to be trustworthy enough to actually use.
03How it works — the agent on a leash
The core of a guardrail is a simple idea: don't trust the input, and don't let an action happen on its own. Instead of letting an AI agent take a command and act on it directly, you put a "checkpoint" on both the way in and the way out — just like a good assistant needs a manager's sign-off before doing anything big.
The first checkpoint (input) works like a "firewall for AI" — it reads every piece of text headed into the model and catches patterns that look like attempts to embed a secret command or unlock the rules. The second checkpoint (action) matters most in the agent era: before AI "does" anything with real consequences — wire money, delete a file, send email outside the org — it has to pass a permission check. The principle is least privilege (give the smallest permissions necessary) plus human-in-the-loop (a person signs off on high-risk work).
Red-teaming = hiring a "red team" (people or AI) to attack your own model every way they can, to find the holes before attackers do — like a fire drill · Guardrail = a "fence" that constantly filters both what goes in (dangerous inputs) and what comes out (answers or actions that shouldn't happen). These two are the main tools of the whole field.
What makes this problem especially hard to solve is that an AI model is built to "obey language" by nature. So there's no clean line between "a trusted command" and "text that happens to have a command embedded in it." Putting up guardrails isn't a one-time patch — it's layered defense that has to be updated constantly.
04Connections — what it ties into
This node is the newcomer in the Cybersecurity & Digital Trust family, whose siblings each guard a different checkpoint. But AI Security is the one that cuts across all of them — because AI is working its way into every system.
- Ties directly to Identity & Access Management: when an agent acts for a person, the big question becomes "who is this agent, and what is it allowed to do?" — giving rise to the idea of agent identity (an AI's identity) that has to be managed like an employee. That's why CrowdStrike launched an identity system covering people, machines, and AI agents
- Feeds work to Security Operations (SOC): the "AI vs AI" side means using AI as a helper in the operations center, sifting millions of alerts down to the real ones
- Born out of Artificial Intelligence: especially Agentic AI (AI that acts on its own) and Foundation Models (giant base models) — the more these two spread, the more demand for guardrails grows. It's a "shadow follows the body" relationship
Put another way: IAM asks "who can get in?" · SOC asks "is anything abnormal?" · and AI Security asks a new question no one asked before — "is this model being fooled right now, and if it's about to act, can we stop it in time?" It doesn't replace the old checkpoints; it fills the one that goes missing when AI sits at the center of every system.
05Where it stands now + who's playing
Right now this field is in a "just exploded" phase — many firms are calling 2025 the year AI security became a "non-negotiable" for organizations. Money is starting to flow in for real, and the giants are buying up startups to bolt this capability onto their own platforms.
The market is still small next to cybersecurity as a whole, but it's growing abnormally fast. The security market for agentic AI is projected to grow from about $1.65B in 2026 to about $13.5B in 2032 — a CAGR of roughly 42%. Meanwhile the broader GenAI cybersecurity market is pegged at about $8.7B (2025) heading to about $35.5B (2031).
The cast of "players" here is different from other fields — many of the real experts are still private companies / startups not on the stock market. The giants who are public usually enter by "buying" this capability rather than building it:
- Palo Alto Networks bought Protect AI (a Seattle AI-security startup) in April 2025 for what several sources estimate at around $650–700M, folding model-scanning, red-teaming, and runtime defense into a new platform called Prisma AIRS
- Cisco bought Robust Intelligence (August 2024) to add AI-model testing and defense to its Security Cloud
- CrowdStrike bought Pangea (an AI-security startup) and launched a new identity system that covers AI agents
Pure-play startups driving the innovation are starting to post real numbers too — HiddenLayer has raised about $56M in total (a $50M Series A led by Microsoft's M12 fund) and hit roughly $35M in revenue in 2025. Lakera and Protect AI come up often as pioneers of guardrails and red-teaming for LLMs.
A key note for investors: there's still no large "pure-play AI security stock" on the market — so investing in this theme mostly means investing in cybersecurity giants that are "bolting on AI security" (PANW, CRWD, CSCO), not in companies that do only this.
06The road ahead
The first direction is agents spreading until guardrails become mandatory. As organizations start turning real AI agents loose (paying bills, answering customers, writing and deploying their own code), every company will have to answer "how do we control our agents?" — so AI firewalls and agent identity should become a standard layer, the way firewalls are part of every organization today.
The second direction is consolidation into the big platforms. The round of acquisitions in 2024–2025 (Cisco→Robust Intelligence, Palo Alto→Protect AI, CrowdStrike→Pangea) sends a clear signal that the cybersecurity giants intend to "own" this layer. Many specialized startups will likely get bought into platforms rather than grow into large independent companies.
The third direction is the AI race on both sides accelerating. When attackers use AI to strike faster and cheaper, defenders have to use AI in the SOC to keep up — moving toward an "autonomous SOC" where AI helps triage and respond to threats semi-automatically. 2025 data shows organizations using AI defenses detect threats faster and catch breaches months sooner. That's the economic reason pushing the whole industry to "arm up with AI" on both sides.
07Challenges & risks
Let's be straight: this is a field that's still brand-new, not yet settled, and carries a whiff of hype. Plenty of vendors have just slapped an "AI security" label on existing products, with real effectiveness yet to be proven — investors and users have to tell the real thing apart from the marketing.
The first technical risk is that prompt injection may be a problem that "can't be fully solved", because it's baked into the nature of a model built to obey language. OWASP itself admits there's no single method that blocks it 100% — you can only cut the risk with layered defense. Incidents like EchoLeak warn that even a world-class company's product can still slip up.
The second risk is that a guardrail that's too tight makes the AI useless. If you set the checkpoints so strict that everything needs a human sign-off, the agent loses the whole value of "working in place of a person." The real challenge is finding the balance between safety and agility — and there's no proven formula yet.
The third risk is an asymmetric fight. Attackers only need one hole; defenders have to plug every one. And when both sides speed up with AI, the game gets faster still — defenders who can't keep up are at an even bigger disadvantage than before.
In short: we're opening an era where AI doesn't just "talk" but "acts" for us in real systems — and every new ability brings a new attack surface. AI Security & Agent Guardrails is the field born to answer one question — how far can we trust AI to act, while still stopping it in time when it gets fooled? This is still the opening chapter, but one the whole AI industry can't do without.