← Back to all announcements
★★★☆☆ 07/07/2026

Amazon SageMaker now supports data lineage in IAM-based domains

IAM-based SageMaker domains now get full data lineage with column-level tracking, interactive graphs, and a new event deletion API.

View original announcement →

Visual Summary

graph TD A{{SageMaker Data Lineage IAM Domains}}:::announced B(Amazon EMR):::compute C(AWS Glue):::compute D(SageMaker Visual ETL):::compute E(SageMaker Notebooks):::compute F([Interactive Lineage Graph]):::feature G([Column-Level Lineage]):::feature H([DeleteLineageEvent API]):::feature I((Data Engineers)):::external B ==>|"Spark lineage events"| A C ==>|"Spark lineage events"| A D -->|"lineage events"| A E -->|"lineage events"| A A -->|"visualizes"| F A -.->|"Event Timestamp mode"| G A -->|"manages"| H I -->|"queries & publishes"| A classDef announced fill:#ff9900,stroke:#ec7211,color:#fff,font-weight:bold classDef compute fill:#e3f2fd,stroke:#1565c0,color:#1565c0 classDef storage fill:#e8f5e9,stroke:#2e7d32,color:#2e7d32 classDef feature fill:#fff3e0,stroke:#e65100,color:#e65100 classDef external fill:#f5f5f5,stroke:#616161,color:#616161

What's New

Amazon SageMaker Unified Studio has extended OpenLineage-compatible data lineage support to IAM-based domains, closing a gap that previously limited this capability to IAM Identity Center-based domains only. The feature captures lineage events from Apache Spark jobs running on Amazon EMR, AWS Glue, SageMaker Visual ETL, and notebooks, providing an interactive visual graph of data movement from source to consumption. A new DeleteLineageEvent API is also introduced, enabling programmatic removal of previously published lineage events across both domain types.

How It Works

  • OpenLineage event capture: Lineage events are automatically captured from Apache Spark jobs running on Amazon EMR, AWS Glue, SageMaker Visual ETL, and notebooks, conforming to the OpenLineage open standard specification.
  • Interactive lineage graph: Data flow is rendered as a directed graph with nodes representing datasets, jobs, and job runs, connected by edges that show upstream and downstream dependencies.
  • Two graph traversal modes: Query Graph mode performs fast server-side traversal for efficient lineage path retrieval; Event Timestamp mode uses client-side recursive expansion to expose detailed column-level lineage data.
  • Configurable graph depth: Users can control how many hops (1–5 levels) of upstream and downstream nodes are displayed, allowing both high-level and granular views.
  • Dataset-only view: A toggle hides job and job run nodes, simplifying the visualization to show only data asset relationships.
  • Programmatic API access: The PostLineageEvent API accepts OpenLineage run event payloads; GetLineageEvent, ListLineageEvents, QueryGraph, GetLineageNode, and ListLineageNodeHistory support retrieval and traversal; the new DeleteLineageEvent API enables removal of specific published events.
  • IAM-based authentication: In IAM-based domains, lineage operations are authorized using standard AWS IAM roles and policies, without requiring IAM Identity Center federation.

Why It's Important

  • Broader domain coverage: Organizations that deployed SageMaker Unified Studio with IAM-based authentication—common in enterprise environments with existing IAM governance structures—can now access data lineage without migrating to IAM Identity Center.
  • Data governance and compliance: End-to-end lineage tracking helps organizations satisfy regulatory requirements (e.g., GDPR, CCPA, HIPAA) by providing auditable records of how data originates, transforms, and is consumed.
  • Faster root-cause analysis: Column-level lineage in Event Timestamp mode allows data engineers and analysts to pinpoint exactly where data quality issues were introduced in a pipeline.
  • Impact analysis: Before modifying a dataset or pipeline, teams can visualize downstream dependencies to assess the blast radius of a change, reducing unintended breakages.
  • OpenLineage interoperability: Adherence to the OpenLineage standard means lineage events from third-party tools can be ingested and managed within the same graph, avoiding vendor lock-in.
  • Event lifecycle management: The new DeleteLineageEvent API gives teams control over lineage data hygiene, enabling removal of erroneous or test events from production lineage graphs.

How It's Different

  • IAM-native support vs. Identity Center requirement: Unlike the prior state where data lineage required IAM Identity Center federation, this update enables the same capability using native IAM roles, making it accessible to a wider set of existing AWS deployments.
  • OpenLineage standard vs. proprietary formats: SageMaker's lineage implementation uses the open OpenLineage specification rather than a proprietary schema, enabling integration with any OpenLineage-compatible tool (e.g., Apache Airflow, dbt, Marquez).
  • Column-level granularity: Event Timestamp mode exposes field-to-field column lineage, which goes beyond dataset-level tracking offered by many competing lineage tools.
  • Dual visualization modes: The combination of a fast server-side Query Graph mode and a detailed client-side Event Timestamp mode gives users flexibility that single-mode lineage tools lack.
  • Integrated multi-service capture: Lineage is captured natively across EMR, Glue, Visual ETL, and notebooks within a single unified graph, whereas standalone lineage tools typically require separate connectors and manual stitching.
  • Programmatic event deletion: The DeleteLineageEvent API is a differentiating capability not universally available in managed lineage services, providing explicit control over the lineage event store.

When to Prefer It

  • IAM-only environments: Use this feature when your SageMaker Unified Studio domain is configured with IAM-based authentication and migrating to IAM Identity Center is not feasible or desired.
  • Multi-engine Spark pipelines: Ideal when data flows across multiple AWS compute services (EMR, Glue, Visual ETL) and you need a single unified lineage graph rather than siloed per-service tracking.
  • Regulated industries: Prefer this when your organization must demonstrate data provenance and transformation history for compliance audits in sectors such as financial services, healthcare, or government.
  • Data quality troubleshooting: Use Event Timestamp mode and column-level lineage when debugging data discrepancies that require tracing a specific field's transformation history across multiple job runs.
  • Large-scale data mesh architectures: Leverage the configurable graph depth and dataset-only view when managing complex data mesh topologies where visualizing full job-level detail would be overwhelming.
  • OpenLineage ecosystem integration: Choose this when you already use OpenLineage-compatible tools (Airflow, dbt, Marquez) and want to centralize lineage management within AWS without abandoning existing instrumentation.
  • Lineage data lifecycle management: Use the DeleteLineageEvent API when running automated pipelines that generate test or ephemeral lineage events that should not pollute production lineage graphs.

Availability

  • General Availability: This feature is generally available (GA) as of July 7, 2026; it is not a preview or beta release.
  • Supported regions (15 total): US East (N. Virginia), US East (Ohio), US West (Oregon), Europe (Ireland), Europe (Frankfurt), Europe (London), Europe (Paris), Europe (Stockholm), Asia Pacific (Tokyo), Asia Pacific (Seoul), Asia Pacific (Singapore), Asia Pacific (Sydney), Asia Pacific (Mumbai), South America (São Paulo), and Canada (Central).
  • Domain type support: Available for both IAM-based and IAM Identity Center-based SageMaker Unified Studio domains.
  • Pricing: No separate pricing is listed for data lineage; it is included as part of Amazon SageMaker Unified Studio—consult the SageMaker pricing page for domain and API call costs.
  • Lineage sources: Lineage capture is supported for Apache Spark jobs on Amazon EMR, AWS Glue, SageMaker Visual ETL, and notebooks; other compute sources are not explicitly listed as supported.
  • Graph depth limitation: Configurable depth is bounded between 1 and 5 hops; deeper traversals beyond 5 levels are not supported in a single query.

Tags

Servicessagemaker-unified-studio
Typenew-feature
Conceptsmlopsdata-analytics
Use Casesenterprise
GeographyGlobal

Related Resources

AI Radar AWS

AWS AI/ML news — curated, researched, explained

An automated intelligence platform that curates, researches, and analyzes AWS AI/ML/GenAI announcements daily. Every report is backed by real research — the system reads linked blog posts and documentation to provide accurate, in-depth analysis.

How Each Report Is Generated

  1. Collection — Daily monitoring of the AWS "What's New" RSS feed
  2. Filtering — AI-powered relevance detection for AI/ML/GenAI topics
  3. Taxonomy Tagging — LLM-based classification across 6 dimensions
  4. Importance Scoring — Point-based system with tag bonuses (1-5 stars)
  5. Research Phase — Follows links to blog posts and documentation
  6. Report Generation — Claude Sonnet produces structured 6-section analysis
  7. Visual Summary — Claude Opus generates Mermaid diagrams for key items
  8. Publishing — Static website rebuilt and deployed via CloudFront

Features

  • Faceted filtering by service, type, concept, and more
  • Multi-dimensional taxonomy with 80+ tags across 6 dimensions
  • Geographic availability badges (Global, APJ, EMEA, AMER) with filtering
  • Timeline visualization of announcement volume
  • PDF export for offline reading
  • Mermaid visual summaries for key announcements
  • Daily automated updates — no manual curation
What makes this different: Each report involves a dedicated research phase where the system reads linked blog posts and AWS documentation pages. This produces analysis that goes beyond the original announcement text.

Technology

Built with Python, AWS Lambda, Amazon Bedrock (Claude Sonnet 4.6, Opus 4.6, Haiku 4.5), S3, CloudFront, WAF, EventBridge, and CDK.

Open Source

This project is open source. Fork it, customize it for your needs, and deploy your own instance.
📦 github.com/bbonik/ai-radar-aws

How Importance Scoring Works

Each announcement receives a point score based on multiple factors. The total score maps to a 1-5 star rating:

1★ < 2 pts 2★ ≥ 2 pts 3★ ≥ 3.5 pts 4★ ≥ 5 pts 5★ ≥ 6.5 pts

Point Breakdown

FactorPointsWhen
Core AI service (Bedrock, AgentCore, SageMaker AI)+4Service named in title
Key AI service (SageMaker, Kiro, QuickSight)+2Service named in title
Other AI-related service+1Default
Blog post link+3Link to aws.amazon.com/blogs/
GitHub samples link+2Link to github.com/aws*
Documentation link+1Link to docs.aws.amazon.com/
New model+1.5Tagged as "new-model"
New service+1Tagged as "new-service"
New feature+0.5Tagged as "new-feature"
Anthropic / OpenAI provider+2Provider explicitly mentioned
Instance / notebook announcement-2Hardware/capacity, not feature
Performance / pricing / security-0.5Incremental updates
Region expansion to APJ+1Expands to Asia Pacific
Region expansion (non-APJ only)-1.5Only expands to other regions

Geographic Relevance Badges

Each announcement card shows a small badge indicating whether the feature is available in your region:

🌐 Global Available in all regions
🌏 APJ Asia Pacific
🌍 EMEA Europe / Middle East / Africa
🌎 AMER Americas (US, Canada, South America)
No badge Geography unknown
How geography is detected: The system detects ALL geographies mentioned in each announcement. If the text mentions specific regions (Tokyo, Frankfurt, Oregon, etc.), the corresponding geography badges are shown. If it says "all regions" or is a new feature with no region specified, it gets the Global badge. Geography is also filterable — click a geo chip to see only announcements available in that region.