Amazon EMR on EKS now supports Apache Spark troubleshooting agent
AI-powered root cause analysis and code fixes for failed Spark jobs now available across all EMR deployment options, including EKS.
View original announcement →Visual Summary
graph TD
A{{Spark Troubleshooting Agent}}:::announced
B((Data Engineer)):::external
C(Amazon EMR on EKS):::compute
D(SageMaker Unified Studio MCP Server):::compute
E([Root Cause Analysis]):::feature
F([Code Recommendations]):::feature
G(Spark History Server):::storage
H(AWS CloudTrail):::storage
I([MCP AI Assistants]):::feature
B ==>|"troubleshoot with AI"| A
A -->|"analyzes jobs"| C
A -->|"connects to"| D
D -->|"extracts logs"| G
A -->|"provides"| E
A -->|"generates"| F
A -.->|"logs actions"| H
B -.->|"uses via"| I
I -->|"invokes"| A
classDef announced fill:#ff9900,stroke:#ec7211,color:#fff,font-weight:bold
classDef compute fill:#e3f2fd,stroke:#1565c0,color:#1565c0
classDef storage fill:#e8f5e9,stroke:#2e7d32,color:#2e7d32
classDef feature fill:#fff3e0,stroke:#e65100,color:#e65100
classDef external fill:#f5f5f5,stroke:#616161,color:#616161
What's New
Amazon EMR on EKS now supports the Apache Spark troubleshooting agent, completing coverage across all three EMR deployment options (EC2, Serverless, and EKS). Data engineers can diagnose failed Spark jobs using natural language prompts, receiving automated root cause analysis and PySpark code recommendations without manually sifting through distributed logs or the Spark History Server. The agent is accessible directly from the EMR on EKS console via a "Troubleshoot with AI" button, or through any MCP-compatible AI coding assistant such as Kiro, Claude Code, or Cursor.
How It Works
- Feature Extraction: When a Spark job fails, the agent connects to the EMR Persistent UI for the EKS job and automatically collects Spark event logs, executor timelines, query plans, configuration settings, and error stack traces to build a comprehensive context profile.
- GenAI Root Cause Analysis: The agent correlates extracted telemetry with an AI model and a Spark knowledge base to identify the precise failure cause—such as executor out-of-memory errors, data skew, resource contention, configuration mistakes, or connectivity failures.
- Code Recommendations: If a code-level issue is detected, the agent automatically generates specific PySpark (or Scala) code fixes; users can also request code suggestions on demand without triggering a full analysis.
- MCP Architecture: The system uses a three-component architecture: an MCP-compatible AI assistant in the developer's environment, an MCP Proxy for AWS handling secure authentication, and the Amazon SageMaker Unified Studio Remote MCP Server that hosts the specialized Spark troubleshooting tools.
- Console Integration: A "Troubleshoot with AI" option appears directly on failed jobs in the EMR on EKS console, requiring no additional setup for console-based access.
- Security: All agent operations are strictly read-only, authenticated via IAM roles, and every action is logged in AWS CloudTrail for full auditability.
Why It's Important
- Reduces Mean Time to Resolution: Diagnosing distributed Spark failures traditionally requires manually correlating logs across dozens of executors, Spark History Server UI, and cluster metrics—a process that can take hours; the agent compresses this to minutes.
- Lowers the Skill Barrier: Junior data engineers and analysts who lack deep Spark internals expertise can now get expert-level diagnostic guidance through simple natural language questions.
- Unified Coverage Across EMR: With EKS support added, the agent now covers every EMR deployment model, meaning teams using mixed or migrating deployments have a consistent troubleshooting experience regardless of the underlying infrastructure.
- Actionable, Not Just Diagnostic: The agent goes beyond identifying problems by providing concrete code changes, reducing the feedback loop between diagnosis and fix validation.
- No Additional Cost: The agent is included at no extra charge as part of Amazon EMR, making it immediately accessible without budget justification.
- Ecosystem Integration via MCP: Support for the Model Context Protocol means the agent can be embedded into existing developer workflows (Cursor, Claude Code, Kiro) rather than requiring engineers to context-switch to a separate tool.
How It's Different
- Full EMR Parity: Unlike earlier releases that only covered EMR on EC2 and EMR Serverless, this launch extends the same AI troubleshooting capability to EMR on EKS, making it the only AWS-native tool that spans all three EMR deployment modes.
- MCP-First Design: The agent exposes its capabilities as an MCP server, allowing any compatible AI coding assistant to invoke Spark diagnostics programmatically—a more flexible integration model than point-in-time console tools or static runbooks.
- Read-Only and Auditable by Design: Unlike ad-hoc scripts or third-party observability tools that may require write permissions or external data egress, this agent is strictly read-only and fully CloudTrail-logged, satisfying enterprise security and compliance requirements.
- Contextual Code Generation: Most log analysis tools surface errors; this agent additionally generates corrected PySpark/Scala code snippets tied directly to the identified root cause, closing the gap between diagnosis and remediation.
- Native AWS Integration: Because the agent connects directly to EMR Persistent UI and SageMaker Unified Studio infrastructure, it requires no additional log exporters, third-party agents, or data pipeline setup compared to external APM solutions.
When to Prefer It
- Intermittent or Hard-to-Reproduce Failures: When a Spark job fails sporadically and the error is buried deep in executor logs across many pods, the agent's automated log aggregation and correlation is far faster than manual investigation.
- Memory and Resource Tuning: When jobs fail with OOM errors or exhibit data skew, the agent can pinpoint the specific stage, executor, or partition causing the issue and recommend configuration or code changes.
- Teams Migrating to EMR on EKS: Organizations moving workloads from EMR on EC2 or Serverless to EKS who are less familiar with Kubernetes-specific failure modes will benefit from the agent's ability to abstract infrastructure complexity.
- Developer Workflow Integration: When data engineers prefer to stay in their IDE (Cursor, Claude Code, Kiro) rather than switching to the AWS console, the MCP server integration allows troubleshooting without leaving the coding environment.
- Compliance-Sensitive Environments: When security policies require all diagnostic tooling to be read-only and fully audited, the IAM + CloudTrail model makes this agent a safe choice compared to tools requiring broader permissions.
- High-Velocity Data Pipelines: In production environments where failed jobs have downstream SLA impacts, the speed of AI-assisted root cause analysis helps on-call engineers resolve incidents faster.
Availability
- GA Status: Generally available as of July 10, 2026; no preview or beta caveats mentioned.
- Supported Platforms: EMR on EKS (new), EMR on EC2, EMR Serverless, AWS Glue, and Amazon SageMaker Notebooks.
- Supported Languages: Python (PySpark) and Scala Spark applications.
- Supported Workload States: Only failed Spark workloads are supported; the agent does not analyze running or completed-successfully jobs.
- Regional Availability: Available in 16 regions co-located with SageMaker Unified Studio: us-east-1, us-east-2, us-west-2, ca-central-1, eu-north-1, eu-west-1, eu-west-2, eu-west-3, eu-central-1, ap-northeast-1, ap-northeast-2, ap-southeast-1, ap-southeast-2, ap-south-1, and sa-east-1.
- Pricing: No additional cost; included as part of Amazon EMR. Standard EMR resource charges apply when re-running jobs to validate fixes.
- Prerequisite: EMR Persistent UI must be configured for EMR on EC2 and EMR on EKS workloads; SageMaker Unified Studio availability is required in the target region.
