5 Questions Every CISO Should Ask Their AI Gateway Vendor

The AI gateway market has exploded. Every vendor claims "enterprise-grade security," "SOC 2 compliance," and "complete data control." Most of these claims are either true in a narrow technical sense, meaningless in practice, or aspirational.

As a CISO evaluating AI governance infrastructure, your job is to cut through the marketing and understand what you're actually buying. These five questions will do that.

Question 1: Where Does My Data Live, and Who Can Access It?

Why it matters: "Data residency" is a marketing term. What you need to know is the complete data flow: where each byte of your AI traffic is stored, for how long, and who — including the vendor's own employees — can read it.

What you're looking for:

• Data processed and stored exclusively within your chosen cloud region (or your own infrastructure)
• No vendor access to request/response content, even for "support purposes"
• Clear data retention policies with configurable limits
• The ability to verify these claims through contract terms and technical architecture documentation

Red flags:

• Vague answers like "we use industry-standard security"
• Logging pipelines that send data to the vendor's central analytics infrastructure
• Support teams that can access your traffic to diagnose issues (this means they can read your data)
• Terms of service that grant the vendor rights to use your data for model improvement

Good answer: "Your traffic is processed in [region] and written to [storage location] that you control. Our staff have no access to request content. You can verify this through our network architecture diagram and the fact that our support team works only with anonymized telemetry you explicitly share."

Question 2: Who Holds the Encryption Keys?

Why it matters: Encryption at rest is table stakes. The real question is key ownership. If the vendor holds the keys, they can decrypt your data — and so can their cloud provider, and so can law enforcement with a valid subpoena served to the vendor without your knowledge.

What you're looking for:

• Customer-managed encryption keys (CMEK) — your keys, not the vendor's
• Support for your existing KMS (AWS KMS, Azure Key Vault, HashiCorp Vault, HSM)
• Crypto shredding capability: when you delete a key, all associated data becomes permanently inaccessible
• Key rotation support without data re-encryption downtime

Red flags:

• "We encrypt all data at rest" without specifying who controls the keys
• No support for external KMS — keys managed exclusively by the vendor
• Inability to explain what happens to your data if you delete your account and your keys

Good answer: "You manage your own encryption keys via [KMS options]. We never see your plaintext key material. Destroying your key makes all your data cryptographically inaccessible. We support key rotation with zero downtime using key versioning."

Question 3: Can You Prove Your Audit Logs Haven't Been Tampered With?

Why it matters: Audit logs are only useful for compliance and forensics if you can trust them. A vendor who controls both the data and the audit log can, in principle, modify both. Regulators increasingly expect tamper-evident audit trails, not just audit logs.

What you're looking for:

• Cryptographically signed audit events (Ed25519 or similar)
• Hash-chained records: each event includes a hash of the previous event
• Customer-controlled signing keys: the vendor can't forge signatures on records they didn't generate with your key
• Verifiable chain integrity: you can run a verification tool that will detect any tampering

Red flags:

• "Our audit logs are stored in [cloud service] which is highly durable" — durability is not tamper-evidence
• No cryptographic signing, just database records with timestamps
• Vendor-controlled signing keys (they can re-sign modified records)
• No tooling for independent chain verification

Good answer: "Every audit event is signed with an Ed25519 key that lives in your KMS — we can't forge signatures without your key. Events are hash-chained: modifying any record breaks the chain for all subsequent records. We provide a verification tool you can run independently to confirm chain integrity."

Question 4: What Happens to Your Data in a Breach or Vendor Insolvency?

Why it matters: Vendors get breached. Startups run out of money. You need to know what your data exposure looks like in both scenarios, and whether you can recover your data and audit history without the vendor's cooperation.

What you're looking for:

• On-premise deployment option: if the software runs in your infrastructure, a vendor breach doesn't expose your traffic data
• Data export capabilities: complete export of all your data in a standard format
• Audit log portability: your compliance records should outlive any single vendor
• Contractual data destruction obligations if you terminate

Red flags:

• No on-premise option — all processing happens in the vendor's infrastructure
• Proprietary data formats that require the vendor's tools to read
• Audit logs only accessible through the vendor's portal
• Unclear contractual obligations around data after termination

Good answer: "Our on-premise Docker/Kubernetes deployment means your traffic never leaves your infrastructure. Your audit logs are standard JSON/CSV and can be exported at any time. We have no hold over your compliance records. Our termination clause requires we delete all your data from our systems within 30 days and provide written confirmation."

Question 5: How Does This Handle AI-Specific Threats?

Why it matters: Traditional security controls protect against traditional attacks. AI systems have a new threat surface: prompt injection, tool misuse, jailbreaks, and model manipulation. Your AI gateway needs to address these, not just OWASP Top 10.

What you're looking for:

• Prompt injection detection in requests and in tool results (RAG outputs, web search results, etc.)
• Function call / tool use security: schema validation, per-tool rate limiting, allowlist enforcement
• PII detection and redaction in AI traffic specifically
• Agentic security controls for multi-step, autonomous workflows

Red flags:

• Security features described entirely in terms of standard WAF/API gateway capabilities
• No mention of prompt injection, tool security, or agentic attack vectors
• PII detection that only covers requests, not responses or tool results
• No concept of per-session tool allowlists or agent policy enforcement

Good answer: "We intercept and validate every tool call before execution. Our ToolGuard layer enforces schema validation, per-tool rate limits, and parameter sanitization. We scan tool results for prompt injection before returning them to the model. For agentic workflows, you can define per-session tool allowlists so agents can only use the tools appropriate for their task."

---

Using These Questions in Your Evaluation

Run these questions in a technical deep-dive with the vendor's engineering team, not just their sales team. Sales representatives will give you marketing answers. Engineers will either confirm capability or reveal gaps.

For each question, ask for:

1. A written answer that can be included in your vendor risk assessment
2. A reference customer in your industry who can speak to the capability
3. A technical demonstration (not a slideshow) of the specific feature

The vendors who can answer all five questions with specificity, technical detail, and verifiable claims are the ones worth serious consideration. The rest are selling you a story.

---

Ready to put SharkRouter through these questions? Request a technical briefing with our engineering team — we'll answer all five with live demonstrations.