Run Interactive Workloads on Amazon EMR Serverless with Spark Connect
Develop PySpark interactively in VS Code or Jupyter against serverless Spark—no cluster management, with per-session cost tracking.
View original announcement →Visual Summary
graph TD
A{{EMR Serverless Spark Connect}}:::announced
B((Developer IDE)):::external
C(SageMaker Unified Studio):::compute
D([Persistent Spark Context]):::feature
E([Per-Session Cost Tracking]):::feature
F(Spark Driver & Executors):::compute
G([Spark UI Monitoring]):::feature
H(Amazon S3):::storage
I([IAM Access Control]):::feature
B ==>|"connects via"| A
C ==>|"native integration"| A
A -->|"provisions"| F
A -->|"enables"| D
A -->|"provides"| E
F -->|"reads/writes"| H
A -.->|"observability"| G
A -->|"governs"| I
classDef announced fill:#ff9900,stroke:#ec7211,color:#fff,font-weight:bold
classDef compute fill:#e3f2fd,stroke:#1565c0,color:#1565c0
classDef storage fill:#e8f5e9,stroke:#2e7d32,color:#2e7d32
classDef feature fill:#fff3e0,stroke:#e65100,color:#e65100
classDef external fill:#f5f5f5,stroke:#616161,color:#616161
What's New
Amazon EMR Serverless now supports interactive sessions via Apache Spark Connect, allowing data engineers and scientists to develop and run PySpark applications directly from local IDEs (VS Code, PyCharm, Jupyter) or managed notebooks in Amazon SageMaker Unified Studio without managing Spark infrastructure. The feature introduces a persistent Spark context that spans notebook cells and scripts, blending local Python execution with remote Spark operations on serverless compute. Available starting with EMR release 7.13, it also delivers granular per-session cost visibility and real-time observability through the Spark UI and EMR console.
How It Works
- Client-server architecture via Spark Connect: The Spark Connect protocol decouples your local PySpark client (IDE or notebook) from the Spark driver process running on EMR Serverless, so application code and Spark infrastructure operate independently.
- Session lifecycle management: When you invoke
StartSession, EMR Serverless provisions a dedicated Spark driver and executors; each session receives a unique endpoint URL and authentication token used to establish the connection. - Persistent Spark context: The session remains alive across multiple cells, scripts, and queries until explicitly terminated or an idle timeout is reached, eliminating the cold-start overhead of restarting Spark between operations.
- Local + remote execution blend: Local Python logic executes in your client environment while DataFrame transformations and SQL operations are serialized and dispatched to the remote Spark driver, then results are returned to the client.
- IAM-based access control: New fine-grained IAM permissions (
emr-serverless:StartSession,GetSessionEndpoint,TerminateSession, etc.) govern who can create, inspect, and terminate sessions, plusiam:PassRolefor the execution role. - Observability stack: Real-time monitoring is available via the Spark UI (accessible through
emr-serverless:GetResourceDashboard), historical job tracking via the Spark History Server, and session management through the EMR console, API, CLI, or SDK. - SageMaker Unified Studio integration: Managed notebooks within SageMaker Unified Studio connect natively to EMR Serverless interactive sessions, providing a fully integrated data and ML development environment without manual endpoint configuration.
Why It's Important
- Eliminates the infrastructure management tax for interactive workloads: Previously, running interactive Spark sessions required provisioning and maintaining persistent EMR clusters; serverless removes that burden while retaining the full Spark engine.
- Dramatically shortens the inner development loop: Developers can iterate on PySpark logic cell-by-cell against production-scale data without redeploying batch jobs, reducing the time from idea to validated code.
- Enables true IDE-native Spark development: Setting breakpoints, stepping through PySpark code, and inspecting DataFrames in VS Code or PyCharm against live remote data was not previously possible with EMR Serverless.
- Granular cost attribution per session: Per-session cost and usage visibility allows teams to track spending at the individual developer or experiment level, enabling chargeback models and tighter cost governance.
- Bridges the gap between exploration and production: Incremental job development in an interactive session on the same EMR Serverless runtime used for batch production reduces environment parity issues and accelerates promotion to production.
- Unified data + AI development in SageMaker Unified Studio: Combining EMR Serverless Spark compute with SageMaker's AI/ML tooling in a single interface supports end-to-end workflows from raw data exploration to model training without context switching.
How It's Different
- vs. EMR Serverless batch jobs: Batch jobs require a full job submission and cluster teardown per run with no persistent context; interactive sessions maintain a live Spark driver across multiple operations, enabling iterative, stateful development.
- vs. EMR on EC2 / EMR on EKS interactive notebooks: Those options require you to provision, size, and manage persistent clusters or node groups; EMR Serverless auto-scales and bills only for actual compute consumed per session.
- vs. Amazon Managed Service for Apache Flink or Glue interactive sessions: AWS Glue interactive sessions offer a similar serverless notebook experience but are tied to the Glue runtime and catalog; EMR Serverless Spark Connect uses the full EMR Spark runtime (7.13+) with broader library and configuration compatibility.
- vs. self-hosted Spark Connect on Kubernetes: Running Spark Connect on self-managed Kubernetes requires cluster provisioning, networking, and security configuration; EMR Serverless handles all of this and adds native AWS IAM authentication and console observability.
- vs. Databricks interactive clusters: Databricks provides a similar interactive notebook-to-Spark experience but within a proprietary platform; EMR Serverless Spark Connect delivers comparable interactivity within the AWS-native ecosystem with standard open-source tooling.
- Per-session endpoint isolation: Each session gets its own endpoint URL and auth token, providing stronger isolation between concurrent users or experiments compared to shared notebook kernels on a single cluster.
When to Prefer It
- Ad hoc data exploration on large datasets: When analysts need to run exploratory queries against S3-resident data at scale without provisioning a cluster, interactive sessions provide on-demand Spark compute with no idle cost.
- Iterative PySpark job development: When building a new ETL pipeline, developers can write and test individual transformation steps cell-by-cell, inspecting intermediate DataFrames before assembling the final production job.
- IDE-based debugging of complex Spark logic: When a PySpark job produces unexpected results, engineers can use VS Code or PyCharm breakpoints to step through DataFrame operations against real data rather than relying on log analysis alone.
- Multi-user notebook environments with cost isolation: When multiple data scientists share an EMR Serverless application and per-user cost tracking is required, the per-session billing granularity enables accurate chargeback.
- Unified AI/ML and data engineering workflows in SageMaker Unified Studio: When teams want to perform feature engineering with Spark and then immediately feed results into SageMaker model training within a single project environment.
- Pre-production validation before batch job deployment: When a batch PySpark job needs to be validated against production-scale data before scheduling, an interactive session on the same EMR Serverless runtime provides high-fidelity testing without a separate cluster.
- Organizations standardizing on serverless compute: When an organization wants to eliminate all persistent Spark cluster management, interactive sessions extend the serverless model to cover the previously cluster-dependent interactive development use case.
Availability
- General Availability (GA): Spark Connect on Amazon EMR Serverless is generally available as of June 9, 2026, with EMR release 7.13 and later.
- Regional availability (EMR Serverless): Available in all AWS Regions where Amazon EMR Serverless is currently supported.
- Regional availability (SageMaker Unified Studio integration): The SageMaker Unified Studio notebook experience is available in 15 regions: US East (N. Virginia), US East (Ohio), US West (Oregon), Europe (Ireland), Europe (Frankfurt), Europe (London), Europe (Paris), Europe (Stockholm), Asia Pacific (Tokyo), Asia Pacific (Seoul), Asia Pacific (Singapore), Asia Pacific (Sydney), Asia Pacific (Mumbai), Canada (Central), and South America (São Paulo).
- Pricing model: Billed on the standard EMR Serverless consumption model (vCPU-hours and GB-hours for driver and executor resources); per-session cost visibility is provided in the EMR console for granular tracking.
- Client compatibility: Supported from any PySpark client including VS Code, PyCharm, Jupyter notebooks, and SageMaker Unified Studio managed notebooks; requires the PySpark library compatible with Spark Connect on the client side.
- Considerations and limitations: Certain Spark features that rely on direct driver-side execution (e.g., RDD APIs, SparkContext-level operations) may have limitations under the Spark Connect client-server model; refer to the EMR Serverless Interactive Sessions User Guide for the full list.
