What if your paint shop could catch a failing tank and reroute production around it automatically? SPARK does more than flag the problem. It classifies the anomaly, reschedules the line, and gives maintenance an exact fix before defective bodies reach the end of the line.
Smart Paint Shop Anomaly Response and Knowledge (SPARK) is an agentic AI system that detects quality anomalies in a paint shop and responds to them automatically, without waiting for an engineer to step in. It monitors the pre-treatment and electrodeposition tanks in near real time, using three machine learning (ML) models on an Amazon SageMaker AI multi-container endpoint to detect an anomaly and classify its type. After an anomaly is found, two AI agents respond in parallel. One computes a new schedule that routes production around the affected tank, and the other writes a root cause analysis (RCA) report with suggested fixes. Deterministic guardrails validate every recommendation before it takes effect, and operators review each incident on the dashboard before maintenance acts.
We tested SPARK on a single-Region deployment that processed 12 tanks at once. In that setup, the full cycle from detecting an anomaly to notifying an operator completed in a median of 55 seconds. The infrastructure as code (IaC) for the deployment is available in the GitHub repository.
In this post, we walk through the architecture, the ML pipeline, and the agent design behind SPARK, and we point you to a sample code repository so you can deploy it in your own account.
The problem
Automotive paint shops run each vehicle body through a sequence of chemical treatment tanks, first for pre-treatment steps such as degreasing, rinsing, and zinc phosphating, and then for electrodeposition. Each step is held to a precise duration. A tank can degrade through wrong pH, depleted zinc, or temperature creep. The body then comes out with paint adhesion failures, corrosion under the coat, or damage severe enough to require a full strip and rework.
Plants manage this reactively today. Engineers watch dashboards by hand, catch anomalies through visual inspection or end-of-line quality checks, and then spend hours rescheduling jobs and writing RCA reports. A single missed anomaly can stall the whole line, ruin hours of in-progress bodies, and delay delivery.
The core challenges:
- Multivariate, time-dependent anomalies – An anomaly shows up not in one sensor reading but across correlated signals over time.
- Slow manual rescheduling – Under time pressure, engineers must judge which jobs can move to equivalent tanks and which must be held.
- Delayed RCA documentation – Engineers write reports from memory hours later, which loses diagnostic accuracy.
- No closed loop – These problems compound because detection, rescheduling, and documentation run as disconnected steps with no automated handoff between them.
What SPARK does
SPARK links three processes into one autonomous pipeline:
- Detects anomalies early – ML models monitor 12 tanks (8 pre-treatment, 4 electrodeposition) and flag deviations before they hit production output.
- Reschedules dynamically – On detection, a master production schedule (MPS) agent finds affected jobs, reroutes them to healthy tanks, and holds jobs-per-hour (JPH) targets while limiting mean time to repair (MTTR) impact on first build output (FBO).
- Recommends fixes – A knowledge graph of equipment, standard operating procedures (SOPs), and maintenance history surfaces root cause hypotheses and fixes in parallel.
Success metrics
SPARK targets four manufacturing key performance indicators (KPIs). The following figures describe intended behavior rather than measured results from a production line.
- First build output (FBO) – FBO is the percentage of vehicle bodies that pass quality the first time without rework. Because anomalies are classified before a body enters a degraded tank, fewer bodies need rework, and every incident reports the projected FBO delay in minutes.
- Jobs per hour (JPH) – Dynamic rescheduling keeps the line moving through degraded-tank events, and the validator enforces a configurable throughput floor of 45 JPH in our reference deployment on every applied schedule.
- Mean time to repair (MTTR) – An RCA report with SOP-backed remediation steps reaches the technician before they arrive at the tank, rather than hours afterward.
- Uptime – Early classification prevents the cascading failures that cause extended shutdowns. Uptime is one contributor to overall equipment effectiveness (OEE), and SPARK improves the availability component directly.
Solution architecture
The SPARK architecture is a three-tier, fully serverless system built on AWS.
Device tier: Plant shop floor
At the device tier, physical equipment generates the operational data that feeds the entire system. Pre-treatment tanks, electrodeposition tanks, programmable logic controllers (PLCs), and sensors continuously capture the critical process parameters: temperature, pH, conductivity, free acid, bath solids, zinc concentration, titanium activity, and accelerator redox potential.
Edge tier: Plants
Those sensor readings flow into the edge tier, where edge computing infrastructure processes the data close to the source before sending it to the cloud. A virtual machine runs an MQTT broker that receives two streams of readings from the shop floor: PT signals from the pre-treatment tanks and ED signals from the electrodeposition tanks. Each reading is one record from one tank at one point in time, and both streams share the same structure. AWS IoT Greengrass provides the edge runtime, and Greengrass Stream Manager buffers the readings locally to minimize data loss during intermittent connectivity.
The readings then cross the edge-to-cloud boundary. Greengrass Stream Manager pushes the combined PT and ED readings, referred to in this post as telemetry, into Amazon Kinesis Data Streams in the enterprise tier. At the same time, it archives the same readings unprocessed in an Amazon Simple Storage Service (Amazon S3) data lake for historical analysis and model retraining.
ML training pipeline
Data from Amazon Kinesis Data Streams flows into Amazon SageMaker Pipelines, which orchestrates the full machine learning lifecycle of preprocessing, training, testing, and retraining. After validation, each model is registered in the Model Registry for version control and governance.
ML inference: Multi-container endpoint
From the Model Registry, the three models are deployed side by side to a single Amazon SageMaker AI multi-container endpoint. The endpoint uses Amazon SageMaker AI managed endpoint auto scaling to adjust instance capacity with load. The three models divide the work rather than vote on it:
| Model | Framework | Purpose |
| Isolation Forest | sklearn 1.2 | Scores each feature vector (one row of correlated sensor values from a chemical treatment tank) to catch statistical outliers |
| LSTM Autoencoder | PyTorch 2.0 | Reconstruction error is the anomaly signal, catching temporal pattern breaks and gradual drift |
| XGBoost Classifier | XGBoost 1.7 | Classifies the anomaly into one of 9 known failure modes |
Table 1: ML models deployed on the Amazon SageMaker AI multi-container endpoint
Together, these three models provide a multi-dimensional view of tank health in a single inference call: classification, drift detection, and anomaly scoring. The Isolation Forest and LSTM catch the anomaly, and XGBoost identifies which anomaly class it is.
Data and orchestration
XGBoost makes the call. When it classifies a reading as anything other than normal, the stream processor function on AWS Lambda marks the tank as degraded. The Isolation Forest and LSTM scores ride along as diagnostic context rather than blending into that decision. The same function writes the status to Amazon DynamoDB and publishes a TankAnomalyDetected event to Amazon EventBridge, because DynamoDB doesn’t natively emit change events. An Amazon EventBridge rule then starts an AWS Step Functions workflow that invokes two AI agents in parallel.
AI agents: Amazon Bedrock AgentCore runtime
Two specialized AI agents run in parallel on Amazon Bedrock AgentCore runtime, a capability of Amazon Bedrock AgentCore. The agents are built with the Strands Agents SDK and powered by Anthropic Claude Haiku 4.5. Each agent reaches its data through tools over Amazon Bedrock AgentCore Gateway, a capability of Amazon Bedrock AgentCore.
MPS Rescheduling Agent: This agent reads the affected jobs and tank health from DynamoDB, scans for functionally equivalent healthy tanks, computes a reschedule that maintains the JPH target, and writes the validated plan back to DynamoDB.
RCA Agent: This agent pulls sensor and maintenance history from DynamoDB, traverses the Amazon Neptune knowledge graph for causal chains, retrieves the relevant SOP from Amazon Bedrock Knowledge Bases, and writes an RCA report with remediation steps back to DynamoDB, where the dashboard reads it.
The knowledge layer
Both agents reason over a shared knowledge layer of three components. The Neptune graph provides structural knowledge, meaning what is physically upstream of an anomaly and what causes it, while the knowledge base provides procedural knowledge, meaning the documented steps to fix it.
- Amazon Neptune knowledge graph: The graph connects 12 tank vertices and 33 anomaly-type vertices with two kinds of directional edges. FEEDS_INTO edges encode the physical process sequence from PT-01 through ED-04, and CAUSED_BY edges encode causal chains between anomalies. SOP vertices and maintenance records complete the picture, so when the RCA Agent investigates a rinse contamination, it traces upstream to the degreasing tank that caused it.
- Amazon Bedrock Knowledge Bases: All 33 SOP markdown documents are chunked and embedded with Amazon Titan Text Embeddings V2 into a vector index.
- Amazon OpenSearch Serverless: This service holds the vector index behind the knowledge base and supports semantic retrieval of the right remediation procedure at analysis time.
The two anomaly counts differ by design. The XGBoost classifier recognizes 9 anomaly classes as the ML label space, and the Neptune graph expands those into 33 per-tank anomaly-type vertices, which are the same classes instantiated on specific tanks, each with its own SOP and causal chain.
Frontend: Operator experience
The frontend layer delivers a secure, real-time operator experience. AWS WAF provides web application firewall protection, Amazon Cognito handles operator authentication and authorization, and Amazon CloudFront serves the React single-page application as static files through a global content delivery network (CDN).
Operators see live telemetry from every tank, real-time health status across the line, the rescheduled production plan from the MPS Agent, and the full RCA report with remediation steps.
Technical stack
| Layer | Services / Frameworks |
| IaC | AWS Cloud Development Kit (AWS CDK), 9 stacks |
| IoT Ingestion | Amazon EventBridge Scheduler, Amazon Kinesis Data Streams, Lambda |
| ML Detection | Amazon SageMaker AI multi-container endpoint: Isolation Forest (sklearn), LSTM Autoencoder (PyTorch), XGBoost |
| Orchestration | Amazon EventBridge, AWS Step Functions |
| AI Agents | Amazon Bedrock AgentCore runtime, Amazon Bedrock AgentCore Gateway (MCP), Strands Agents SDK, Anthropic Claude Haiku 4.5 |
| Knowledge | Amazon Neptune graph DB, Amazon Bedrock Knowledge Base, Amazon OpenSearch Serverless |
| Storage | DynamoDB (5 tables), Amazon S3, Amazon Timestream |
| Dashboard | Amazon API Gateway REST + WebSocket, Amazon CloudFront, React + Vite + Tailwind |
| Authentication / Configuration | Amazon Cognito User Pool, Amazon SSM Parameter Store |
Table 2: SPARK technical stack by layer
Cost attribution
Each anomaly event triggers two agents that call Anthropic Claude and Amazon Titan foundation models, so model usage scales with the number of anomalies the line produces. SPARK optionally creates Amazon Bedrock application inference profiles so that every Anthropic Claude and Amazon Titan Text Embeddings invocation carries cost allocation tags, which makes AI spend visible and filterable in AWS Cost Explorer.
| Profile | Capability tag | Component tag | Model |
| spark-mps-agent | production-scheduling | mps-agent | Anthropic Claude Haiku 4.5 |
| spark-rca-agent | rca | rca-agent | Anthropic Claude Haiku 4.5 |
| spark-kb-embeddings | rca | kb-embeddings | Amazon Titan Text Embeddings V2 |
Table 3: Amazon Bedrock application inference profiles for per-workflow cost attribution
With one Cost Explorer filter on the capability tag, you can answer questions such as what RCA costs per month, agent and embeddings combined.
Sample RCA report output
When the RCA Agent completes its investigation, it produces a structured report:
Recurrence risk is computed deterministically from actual anomaly history in DynamoDB rather than from large language model judgment, so the value stays consistent across reports.
Sample code repository
For the sample code and deployment instructions, refer to the accompanying GitHub repository. The repository includes the nine AWS CDK stacks, the ML training pipeline, both agent definitions, the Amazon Neptune graph schema, the 33 SOP documents, the React operator dashboard, and step-by-step deployment and cleanup instructions.
Conclusion
In this post, you saw how to connect quality detection and production scheduling in an automotive paint shop. By combining real-time ML classification with AI agents that reason over a knowledge graph and production constraints, SPARK detects an anomaly, reschedules production, and documents the root cause before a single vehicle body is affected. The three models divide the work: the Isolation Forest and the LSTM autoencoder provide anomaly and drift evidence, and XGBoost names the exact anomaly class. The two agents divide the response: the MPS Rescheduling Agent keeps production moving behind a deterministic validation guardrail, and the RCA Agent tells maintenance exactly what to fix, with a recurrence risk computed from real anomaly history rather than model judgment.
To deploy SPARK in your own account and explore the full architecture, visit the GitHub repository. For guidance on building agentic AI systems for automotive manufacturing, contact your AWS account team.

