Giskard Review (2026): Features, Pricing & Verdict

What Is Giskard?

Giskard is an open-source Python library — with an accompanying enterprise platform, the Giskard Hub — for testing and evaluating AI systems, with a specific focus on LLM agents and RAG applications alongside traditional machine-learning models. It sits in the same broad quality-and-safety space as evaluation platforms like Braintrust and Patronus and red-team scanners like garak and PyRIT, but it stakes out a distinct position: Giskard frames AI evaluation as software testing. The core insight behind it is that LLMs fail in ways traditional software doesn’t — hallucinations, prompt injection, adversarial manipulation, bias — and that developers under pressure to ship often push models into production with hidden vulnerabilities because they don’t know where to start, what to test for, or how to write effective tests for these novel risks. Giskard’s answer is to automate that testing: point it at your model (it operates black-box, needing only an API endpoint, so it doesn’t have to know your foundation model or vector database), and it will automatically scan for a wide range of issues, generate domain-specific test cases, and produce shareable reports and reusable test suites you can drop into your development process and CI/CD pipeline. Crucially, it’s designed for continuous use across the whole lifecycle: before deployment, it provides quantitative KPIs to confirm an agent is production-ready; after deployment, it continuously detects new vulnerabilities that emerge once the application is live. To find those vulnerabilities it draws on multiple sources — internal company knowledge (like a RAG knowledge base) to catch hallucinations, external cybersecurity monitoring to identify new threats, and a library of internal prompt templates distilled from client experience. It’s genuinely open (Apache 2.0), it’s research-grade (Giskard’s R&D team publishes work like the Phare multilingual benchmark and a striking 650,000-story study of stereotypes across 23 models, with funding from the European Commission and collaborations including Google DeepMind), and it’s built to be accessible to data scientists and AI engineers rather than only security specialists. Within this site’s Machine Learning & MLOps category, Giskard is the open-source AI-testing-and-red-teaming specialist of the cluster: the tool to reach for when you want to proactively find and fix quality and security failures — especially in RAG systems — as a repeatable, automated part of how you build.

Core Features

RAGET: automated RAG evaluation with component-level attribution

The feature that earns Giskard a genuinely distinct place in the toolkit is RAGET, the RAG Evaluation Toolkit — and for anyone shipping a retrieval-augmented assistant, it addresses the single hardest part of the job. RAG applications are notoriously difficult to evaluate because a failure can originate at any stage of the pipeline: retrieval might pull the wrong context, the generator might ignore good context, the knowledge base might be incomplete, or the rewriting and routing logic might misfire — and a simple end-to-end accuracy score tells you something is wrong without telling you where. RAGET tackles this on two fronts. First, it removes the biggest practical barrier to testing RAG at all: instead of hand-authoring an evaluation set, it reads your knowledge base and automatically generates a synthetic test set — realistic questions, reference answers, and the supporting context needed to judge them — so you have a domain-specific evaluation dataset grounded in your own documents without the manual labour. Second, and more importantly, it evaluates your pipeline with component-level attribution: it measures retrieval accuracy (did it find the right chunks?), context relevance (are the retrieved chunks actually useful?), answer correctness (does the response match the reference?), and hallucination rate (did the model make things up?), and it’s designed to attribute failures to specific components of the RAG pipeline rather than just scoring the system as a black box. That component-level diagnosis is the capability that makes Giskard complementary to — rather than overlapping with — the offensive red-team scanners: where tools like garak and PyRIT exist to attack a model and see what breaks, RAGET tells you whether your retrieval-augmented application answers correctly and precisely where it breaks down. It’s the kind of feature that, once a team building a RAG assistant has it, becomes hard to give up, because it turns the murky, frustrating problem of “our chatbot sometimes gives wrong answers and we don’t know why” into a structured, attributable, repeatable evaluation. For RAG evaluation specifically, RAGET is one of the strongest offerings available in any open-source tool.

The LLM Scan and autonomous red teaming

Giskard’s second pillar is proactive vulnerability detection, delivered through its automatic Scan and its red-teaming capabilities. The LLM Scan (invoked as simply as giskard.scan) is an automated vulnerability detector that performs an in-depth assessment of your domain-specific LLM system — chatbots, QA systems, RAG models — rather than merely benchmarking a foundation model in the abstract. It combines two kinds of detectors: heuristics-based detectors that use known techniques and patterns to test for vulnerabilities not specific to your model, and LLM-assisted detectors that use a separate LLM to probe your system for issues specific to your business case. Across those it surfaces the failure modes that matter most for production LLMs: hallucinations, prompt injection, sensitive-information disclosure, harmfulness, bias and stereotypes, and robustness problems. The genuinely useful part is what happens next: the scan results aren’t a dead-end report but a starting point — you convert the issues it finds into actionable tests with a single call, saving them as a reusable test suite that you can re-run on every iteration and against new model versions to guard against regressions, and integrate directly into your CI/CD pipeline. That “scan → test suite → CI” loop is what makes Giskard feel like software testing rather than a one-off audit. Layered on top is red teaming: Giskard’s scanner uses autonomous agents that conduct dynamic, multi-turn conversations, escalating attack intensity when initial probes are blocked and adapting their strategies in real time across a library of 40-plus probes — which catches conversational vulnerabilities that single-turn testing misses entirely. The scanner works with essentially any model — OpenAI, Anthropic, Hugging Face, Azure OpenAI, local models via Ollama, and, through a LiteLLM integration, any supported provider — with a simple model-wrapper function handling the rest. It’s worth an honest calibration here: as security specialists note, Giskard’s prompt-injection and harmfulness detectors are a valuable first-pass signal, but the Scan is a scan, not the exhaustive probe library of a dedicated tool like garak or the orchestrated, sustained campaigns of PyRIT — so for a deep, comprehensive adversarial red-team engagement you’d complement Giskard with a specialist. As an automated, integrated, developer-friendly first line of defence that turns findings into permanent tests, though, it’s excellent.

Traditional ML testing, the enterprise Hub and the v3 rewrite

Beyond LLMs, Giskard retains a capability many newer LLM-only tools lack: it tests traditional tabular machine-learning models too. The ML testing module supports classification, regression and ranking models built with the usual libraries (scikit-learn, XGBoost, CatBoost and others), automatically detecting performance issues, data leakage, bias and robustness problems — which makes Giskard a single testing framework for teams that run both classical ML and modern LLM workloads, rather than forcing them to adopt separate tools for each. For organisations that need more than a local library, the enterprise Giskard Hub adds the collaboration and governance layer: real-time team collaboration with shared workspaces and collaborative annotation, role-based access control, centralised dataset management with versioning of test cases, custom failure categories beyond the standard security and business-logic issues, and — critically for its target market — continuous red teaming with automated, scheduled scanning and monitoring that detects new vulnerabilities as they emerge in production. The Hub is squarely aimed at regulated sectors like banking and insurance, where AI failures carry serious compliance and reputational risk, and it reflects that with a context-aware, Policy-as-Code framework that includes ready-to-use EU AI Act and OWASP Top 10 for LLMs compliance packs — a genuinely differentiated compliance story. The most important thing to understand about Giskard’s current state, though, is the v3 transition. Giskard v3 is a complete, from-scratch rewrite announced in 2026, motivated by two real problems with v2: heavy dependency management (v2 leaned on scikit-learn, PyTorch, pandas and more, which caused “dependency hell”) and a tabular-centric architecture poorly suited to modern multi-turn conversational agents. v3 replaces the monolith with a modular, package-based ecosystem — giskard-checks (composable testing, from simple assertions to dynamic multi-turn scenarios) and giskard-agents (an agent-workflow framework) — that is data-structure-agnostic and lightweight, installing only what you need. It’s a genuinely promising direction. But it’s currently in beta/pre-release, and the flagship Scan and RAGET features have not yet been migrated to v3 (they still rely on the v2 engine), while v2 itself is no longer actively maintained. That transition state is the single biggest practical caveat for a new adopter today: you’re choosing between a modern-but-still-beta v3 whose headline features are mid-migration, and a mature-but-frozen v2 — and it’s the main reason Giskard, for all its strengths, sits where it does rather than higher.

Scored Categories

Pricing

Frequently Asked Questions