115+ items · 9 dimensions · Grounded in NIST, AWS, Microsoft & OWASP frameworks

AI Workflow Readiness Checklist

The checklist our team runs through before any AI system goes to production. Below are the top 5 controls in each dimension — a preview of the full 115+ item assessment.

85–100%
Production-ready
65–84%
Fragile, proceed with caution
40–64%
High risk, significant gaps
<40%
Not production-ready

Want ShipSmith to run this automatically?

Connect your repo and we'll score every AI workflow against these 115+ controls. No manual work required.

How the scan works

01

Connect your repo

~2 min

Point ShipSmith at your codebase. No config, no prep call. It reads the code directly.

02

We discover & score

Automated

ShipSmith finds every AI workflow (including the shadow workflows that pile up when teams move fast) and scores each against these 115+ controls across all 9 dimensions.

03

Get your report

Minutes

A readiness grade per workflow, your weakest areas ranked, and a plain-English fix list, plus guidance to close each gap.

Scan Your Repo, Free →

No credit card required · Free for your first workflow

Grounded in established standards

Our checklist is not proprietary. It synthesises six established industry frameworks into a single actionable assessment. Each item is traceable back to a specific control or best practice in at least one of these sources.

NIST AI RMF

2023

Four core functions: GOVERN, MAP, MEASURE, MANAGE. The gold standard for AI risk management as an organisational discipline.

Risk ManagementView framework

AWS GenAI Lens

2025

Best practices for LLM/RAG/agent systems: excessive agency controls, prompt catalog governance, multi-agent tracing, bias drift monitoring.

GenAI ArchitectureView framework

AWS ML Lens

2025

Model registry gating, data poisoning protection, environment parity, and feature attribution drift monitoring.

ML OperationsView framework

Microsoft RAI

2022

Fairness evaluation (Fairlearn), model cards, error cohort analysis, content safety layers, and Discover-Protect-Govern deployment framework.

Responsible AIView framework

Google MLOps

2021

Three maturity levels (0→2) covering training-serving skew, feature stores, continuous training pipelines, and ML pipeline component testing.

MLOps MaturityView framework

OWASP LLM Top 10

2023

The ten most critical security risks for LLM applications, including prompt injection (#01), excessive agency (#08), and supply chain vulnerabilities (#05).

LLM SecurityView framework
01

Data Foundation

Google MLOps Maturity ModelAWS ML Lens (MLOPS06)NIST AI RMF MAP 2.1
Top 5 of 14
01Data sources are inventoried and documented
02Data quality has been assessed. Error rates, completeness, freshness are measured
03A data pipeline exists to feed the AI system (not manual exports)
04The pipeline handles failures gracefully
05Data freshness is monitored. Stale data is flagged
+9 more controls in this dimension — assessed in full when you run ShipSmith on your codebase.
02

Model & Architecture

Google MLOps Maturity ModelAWS GenAI Lens (GENREL04)Microsoft Responsible AI Standard
Top 5 of 15
01Model choice was selected based on benchmarking against the actual use case
02The model in production is the right size for the task
03System prompts are version-controlled and reviewed like code
04Prompt changes go through a review/approval process before hitting production
05Retrieval strategy (for RAG) has been benchmarked, not just defaults
+10 more controls in this dimension — assessed in full when you run ShipSmith on your codebase.
03

Evaluation & QA

Microsoft Responsible AI (Fairlearn)AWS GenAI Lens (GENOPS01)NIST AI RMF MEASURE 2.5-2.6
Top 5 of 15
01A curated eval dataset exists (real inputs paired with expected outputs)
02The eval dataset is large enough to be statistically meaningful (50–100+ examples)
03Evals run automatically before any deployment
04Evals block deployment if scores drop below a defined threshold
05LLM-as-judge scoring is used for open-ended outputs
+10 more controls in this dimension — assessed in full when you run ShipSmith on your codebase.
04

Observability & Monitoring

AWS GenAI Lens (GENOPS02-GENOPS03)AWS ML Lens (MLOPS06)Microsoft Responsible AI (Accountability)
Top 5 of 14
01Every LLM call is logged with input, output, token count, latency, model, cost
02Multi-step workflows have span-level tracing
03Cost per request is tracked and attributed per user/workflow/team
04Latency is monitored at p50, p90, p95, not just average
05Error rates are tracked and alert on anomalies
+9 more controls in this dimension — assessed in full when you run ShipSmith on your codebase.
05

Resilience & Production Engineering

Google MLOps Level 1-2AWS ML Lens (MLREL04)AWS GenAI Lens (GENOPS05)
Top 5 of 13
01Model fallbacks are implemented: if primary model fails, a fallback handles it
02Retry logic is implemented with exponential backoff for transient errors
03Retryable errors (429, 503) are distinguished from non-retryable ones (400)
04Circuit breakers are in place: if provider error rate exceeds threshold, traffic shifts to fallback
05The system degrades gracefully when the LLM is unavailable
+8 more controls in this dimension — assessed in full when you run ShipSmith on your codebase.
06

Security & Compliance

OWASP LLM Top 10AWS GenAI Lens (GENSEC02-GENSEC06)Microsoft Responsible AI (Privacy & Security)
Top 5 of 14
01API keys are stored in a secrets manager, not in env files or code
02Prompt injection risks are mitigated. System context separated from user input
03User inputs are sanitized before being used in retrieval or tool calls
04Outputs are validated post-generation for PII leakage and policy violations
05Agentic tool execution is sandboxed with appropriate permissions
+9 more controls in this dimension — assessed in full when you run ShipSmith on your codebase.
07

Cost Management

AWS GenAI Lens (GENOPS05)Google MLOps Level 1
Top 5 of 11
01Token budgets are defined per request type
02Monthly AI spend is monitored with budget alerts
03Cost per unit of business value is calculated and tracked
04Model routing is used: simple tasks go to cheap models, complex to expensive
05Batch inference is used for offline/non-latency-sensitive workflows
+6 more controls in this dimension — assessed in full when you run ShipSmith on your codebase.
08

Adoption & Change Management

NIST AI RMF GOVERN 5.1Microsoft Responsible AI (Transparency)
Top 5 of 13
01There is a named owner responsible for the AI system's business outcomes
02The specific daily workflows that changed are documented
03Users who interact with the system were involved in its design
04Training and onboarding for the system exists and has been completed
05There is a feedback channel for users to report AI errors
+8 more controls in this dimension — assessed in full when you run ShipSmith on your codebase.
09

AI Governance & Risk Management

NIST AI RMF (GOVERN 1.1-6.1)AWS GenAI Lens (multiple BPs)Microsoft Responsible AI Standard v2
Top 5 of 6
01A written organisational AI policy exists (covering acceptable use, prohibited use cases, risk tolerance, and review cadence) owned by a named executive
02An AI impact assessment is conducted before deploying any new AI capability, documenting who is affected, what harms could occur, and what mitigations are in place
03A named AI risk owner exists who is distinct from the product/business owner and has authority to pause deployment
04The feedback-to-improvement loop is a documented, owned process, specifying who reviews errors, on what cadence, and what triggers a prompt update vs. retraining vs. architecture change
05A model deprecation policy exists, defining minimum performance floor, sunset timeline, data deletion obligations, and knowledge transfer requirements
+1 more controls in this dimension — assessed in full when you run ShipSmith on your codebase.

Skip the manual checklist.
Let ShipSmith score your workflows automatically.

Connect your repo and ShipSmith discovers every AI workflow and scores it against these 115+ controls. Most scans finish in minutes. Free for your first workflow.

Scan Your Repo, Free →Book a call →