115 controls · 9 dimensions · Grounded in NIST, AWS, Microsoft & Google frameworks

9 Dimensions of AI Production Readiness

Every AI workflow ShipSmith assesses is scored against these 9 dimensions. Each dimension maps to a set of controls grounded in established industry frameworks.

Scan Your Repo, Free →
01

Data Foundation

The AI is only as good as what it's fed. Most production failures trace back here.

Google MLOps Maturity ModelAWS ML LensNIST AI RMF
14 controls

What we check for

Data flows through a repeatable pipeline, not manual exports
PII in training data is identified and handled
Ground-truth labels exist, so accuracy can be measured
Data freshness is monitored, not assumed
02

Model & Architecture

Is the model matched to the job — and can you show why it was chosen?

Google MLOps Maturity ModelAWS GenAI LensMicrosoft Responsible AI
15 controls

What we check for

System prompts are version-controlled, not buried in application code
Model choice is benchmarked against the real use case
An upgrade path is documented for when the model is deprecated
RAG retrieval strategy is tuned for the use case, not left on defaults
03

Evaluation & QA

The dividing line between a production system and a demo.

Microsoft Responsible AI (Fairlearn)AWS GenAI LensNIST AI RMF
15 controls

What we check for

An eval dataset measures accuracy before incidents, not after
Evals gate deployment
Hallucination rate is measured and tracked
Adversarial test cases are part of the suite
04

Observability & Monitoring

You can only improve what you can see — and instrumentation is the easiest layer to defer under deadline.

AWS GenAI LensAWS ML LensMicrosoft Responsible AI
14 controls

What we check for

LLM calls are logged, so failures can be debugged
Cost per request is visible
Error-rate spikes trigger alerts
Business stakeholders have a dashboard
05

Resilience & Production Engineering

Does this system survive the real world, or only the demo environment?

Google MLOps Level 1-2AWS ML LensAWS GenAI Lens
13 controls

What we check for

A fallback keeps the feature up when the LLM is down
Retry logic absorbs transient 429s before users see them
Load testing happens before scale, not after
A rollback procedure covers bad prompt deployments
06

Security & Compliance

Rarely the fun part of shipping — and the first thing an enterprise buyer asks about. We carry this half so it's ready when they ask.

OWASP LLM Top 10AWS GenAI LensMicrosoft Responsible AI
14 controls

What we check for

API keys live in a secrets manager, not environment files
Prompt-injection mitigations are in place
Agentic tool calls are sandboxed against irreversible actions
An incident-response plan covers AI-specific failures
07

Cost Management

Unchecked AI costs are one of the most common reasons companies kill promising AI projects.

AWS GenAI LensGoogle MLOps Level 1
11 controls

What we check for

Token budgets cap what a runaway agent can spend
Spend spikes trigger alerts
Model routing sends simple tasks to cheaper models
A cost model projects spend at 10x usage
08

Adoption & Change Management

Most of what makes AI succeed is people and process — the part that lives outside the code, and the easiest to lose track of as you scale.

NIST AI RMFMicrosoft Responsible AI
13 controls

What we check for

Every workflow has a named owner to call when something breaks
Users have a channel to report AI errors
Adoption is tracked, so you know the workflow is actually used
Leadership has visibility into how the system is performing
09

AI Governance & Risk Management

The organisational scaffolding that determines whether AI is run responsibly at scale.

NIST AI RMFAWS GenAI LensMicrosoft Responsible AI Standard v2
6 controls

What we check for

A written AI policy defines acceptable use
An impact assessment runs before deployment
Third-party vendors' data-retention policies are known and reviewed
A named AI risk owner carries the risk, not just the product owner

See how your AI workflows score across all 9 dimensions.

Free for your first workflow. No credit card required.

Scan Your Repo, Free →