AI Is Coming: Meet the Startups Building Cyber Defenses for the Age of AI

There’s a dark side to AI that’s underreported. It’s not deepfakes, mass surveillance, job displacement, or AI takeovers but rather every CISO’s (Chief Information Security Officer’s) worst nightmare. Since the launch of ChatGPT in 2022, employees have been feeding code, emails, and sensitive documents into generative AI tools, opening the floodgates to data leaks and security risks.

‍

And it’s not just third-party tools like ChatGPT that are raising concerns. Businesses are also developing their own AI tools using internal company data, which presents new challenges for CISOs. For example, an internal AI chatbot trained on Slack conversations could inadvertently expose private discussions if it has security vulnerabilities. As the CEO of Advsersa AI, an AI Security startup, put it: “The PC sparked the first cybersecurity revolution, followed by the cloud and cloud security. Now, we’re entering the era of AI, and AI security is the natural next step.”

‍

In this blog, we’ll map out the rapidly growing AI security ecosystem, highlighting where innovation is happening now and predicting where it’s headed next.

Overview

It’s no surprise that AI security innovation is gaining momentum. To address risks, companies like Apple, Samsung, and JPMorgan initially responded by banning AI tools altogether. However, as AI adoption accelerates, it has become clear that such blanket bans can stifle innovation.

‍

Instead of avoiding AI, organizations are now focusing on securing it. This has given rise to a new wave of AI security startups tackling various AI security challenges, from providing visibility into AI usage within organizations to securing AI models before and after deployment.

‍

For example, Protect AI offers both pre- and post-deployment scanning to safeguard large language models (LLMs), while Aurascape.ai helps businesses monitor AI usage across the firm and enforce security policies. The impact of these solutions is already evident. HiddenLayer, for instance, partnered with a global financial firm to uncover vulnerabilities in its fraud detection AI, saving the company millions. In 2024, Cisco acquired the AI security startup Robust Intelligence for $400 million to offer AI security solutions to its corporate customers, showing an appetite for products in the space.

‍

The market map below gives an overview to the layers of AI security that innovators and companies are producing. We’ll walk through each area to explain the issues and emerging solutions.

‍

‍

1. Visibility, governance, and control

‍

The first step in securing AI is gaining visibility into how employees interact with AI tools: identifying which models they use, what data they input, and what they receive in return. Once CISOs have a clear view of AI usage within the organization, they can implement control measures (also known as governance) to manage AI access and usage. Governance applies to both third-party AI tools (like ChatGPT) and internally developed AI tools (such as an in-house chatbot).

‍

Internal AI Models

For internally built AI, Role-Based Access Control (RBAC) is essential to restrict access to sensitive data. This is a security method that gives access to systems, applications, and data based on a user's role, ensuring resource access is only given to those who need it for their job functions.

‍

For example, if a company builds an AI-powered chatbot trained on internal Slack conversations, it’s crucial to ensure the security of the data layer that contains the private Slack conversations. This involves attaching metadata to the chat data so that only authorized users can access specific information – you don’t want just anyone viewing private Slack conversations.

‍

A key innovation in this space is policy-as-code, which allows teams to define security policies using code rather than relying on manual processes (like writing directly to configuration files). Policy-as-code frameworks can include rules, conditions, and automated protections, such as blocking unauthorized access when breaches occur. CISOs can use programming languages like Python to define policies, making security faster and less error-prone.

‍

Backward compatibility is also essential. New AI security tools must seamlessly integrate with existing enterprise system like Salesforce to prevent accidental data leaks.

‍

External AI Services

For third-party tools like ChatGPT, similar policy enforcement principles can apply. CISOs can set controls specifying which tools employees are allowed to use, define the types of data that can be input to the tools, and establish safeguards to prevent data leaks, often using policy-as-code or comparable frameworks.

‍

While most AI security tools today focus on human interactions with AI, the scope is quickly evolving. APIs and AI agents are starting to access AI models too, so very soon we expect AI security solutions to secure these non-human AI interactions as well.

2. Pre-production model selection, scanning, and posture management

The next layer of AI security focuses on selecting the right AI model and preparing it for deployment within an organization. As of 2025, the landscape of AI models is rapidly expanding, with platforms like Hugging Face offering 1.5M models and 40,000 new models added every week. However, choosing the right model isn’t just about performance; it’s also about evaluating factors like bias and security.

‍

Most AI models are trained on datasets sourced from the internet, making them susceptible to biases and misinformation. These biases can result in reputational damage and even legal repercussions. A prominent example occurred in 2024 when Google’s Gemini model sparked controversy by generating racially diverse images, even when users explicitly requested images of a specific race.

‍

To mitigate such risks, many enterprises are investing in AI auditing and compliance. Similar to how companies use tools like Drata and Vanta to achieve SOC 2 certification, organizations are increasingly adopting AI model certification solutions to ensure their AI models are free from bias and toxicity. Additionally, businesses are partnering with vendors that assess AI models for security risks and ethical concerns, providing documentation for regulators and customers.

‍

Another emerging trend is firms using AI to detect high-risk employee behaviors that could compromise security. For example, Dune Security’s platform identifies risky actions, such as clicking on phishing links or visiting unsecured websites, and flags them as potential threats.

‍

Continuous Vulnerability Scanning

Continuous vulnerability scanning is gaining momentum as a component of AI security. Unlike traditional software, AI models cannot be effectively scanned using existing Endpoint Detection and Response (EDR) tools, which often fail to detect AI-specific threats such as payload attacks, prompt engineering, and exposure of personally identifiable information (PII).

‍

For example, the AI model DeepSeek has demonstrated several vulnerabilities identified by AI security startups like Holistic AI and Hidden Layer. Certain prompts can inadvertently cause DeepSeek to reveal its training data, while structured prompts can trap the model in an endless loop — a phenomenon known as Adversarial Chain of Thought. A red-teaming report by Enkrypt AI revealed that DeepSeek was 11 times more likely to generate harmful content, 4 times more toxic, and 4 times more likely to produce insecure code compared to leading models like GPT-4 and Claude-3. Consequently, new tools are emerging that provide continuous scanning of AI models, identifying vulnerabilities and highlighting areas that require remediation.

‍

Currently, most solutions focus solely on protecting and identifying vulnerabilities within the AI model itself. However, the next wave of innovation is expected to go beyond model protection, aiming to secure the entire AI ecosystem. This would include the application, its data flows, containerized services, vector databases, and other integrated infrastructure components.

3. Post-production query and response monitoring:

So, you've chosen your model, scanned for vulnerabilities, tested its performance, and are ready to deploy it to real users. But you’re not done yet. Once an AI system goes live, it becomes a prime target for attackers seeking to exploit its weaknesses in various ways.

‍

Let’s break down a key list of attack vectors, though this list is not exhaustive.

‍

1. Jailbreaking a model involves tricking it into generating harmful, unethical, or restricted content by bypassing its built-in safeguards. Attackers exploit weaknesses in how LLMs interpret prompts to override security controls. One common method is role-playing attacks, where an attacker instructs the AI to "act as an evil AI" or simulate a restricted scenario, deceiving it into producing content it would normally block. Another technique is character obfuscation, which uses spaces, special characters, or ASCII tricks to confuse content moderation filters and evade detection.
2. SQL Injection Attacks is another well-known web security threat that also applies to AI models interfacing with databases. Suppose an AI-powered chatbot or tool is given direct access to a database without proper input sanitization. In that case, an attacker can insert malicious SQL queries to bypass authentication and access confidential data.
3. Remote Code Execution (RCE) Attacks can impact AI models that support code generation or execution, such as AI-powered coding assistants. If a model is configured to execute code provided by a user, an attacker could inject malicious scripts into a prompt, trick the model into running the scripts on the host system, and gain control over the underlying infrastructure of the AI application. For example, a coding assistant that allows executing Python snippets could be manipulated to run a command that grants the attacker remote access to a system.

‍

Many solutions are emerging to defend against these attack vectors. One promising approach is the AI firewall, which acts like a security guard for your AI models.

‍

AI Firewall

Just as Web Application Firewalls (WAFs) monitor and protect web applications from harmful traffic, an AI firewall monitors and filters the data flowing in and out of an AI model to detect suspicious activity or malicious inputs that could compromise the model. One example is Robust Intelligence’s AI Firewall, which validates model inputs and outputs in real time, protecting against various threats, including prompt injection, PII extraction, and hallucination. Robust Intelligence’s AI firewall is integrated with MongoDB Atlas Vector Search, allowing MongoDB’s 40,000+ customers to use any commercial or open-source AI solutions safely.

‍

Robust Intelligence’s AI security solutions have gained significant traction, with its products being used by JPMorgan Chase, Expedia, Deloitte, Cisco, and even the U.S. Department of Defense. The adoption highlights the growing need for reliable AI security solutions today, and we expect the demand to grow.

‍

Looking ahead

As AI security evolves, some key questions are shaping its future:

1. Who will lead AI security? Startups are leading the charge in innovation today, but it remains to be seen whether they will maintain their dominance or if established security giants will catch up.
2. Where will AI security live? Today, many AI security solutions reside at the software and application layer. However, future innovations may shift toward the platform, infrastructure, or hardware level. It remains to be seen which of these layers will ultimately dominate.
3. Will AI security emerge as a stand-alone product? Early conversations with CISOs and founders suggest that AI security may be distinct enough to emerge as its own category, just like Identity & Access Management, Network Security, Application Security, Cloud Security. Whether this will ultimately happen remains to be seen.

‍

AI is not slowing down, and neither are the threats. The future belongs to those who build defenses as fast as AI evolves.

‍

We’d love to hear from you. If you’re interested in AI security or have insights on where the space is headed, please connect.

‍

📩 Vedant Agrawal – vedant.agrawal@premjiinvest.com

📩 Shivani Singh – shivani@av.vc

‍

Written together with Alumni Ventures, including Shivani Singh (Senior Associate) and Kshiteej Prasad (Alumni Venture Scout)

‍