AWS Entity Resolution launches support for incremental Machine Learning based matching workflows
Save 95% on processing time by matching only new records instead of reprocessing your entire dataset.
View original announcement →Visual Summary
graph TD
A{{Incremental ML Matching}}:::announced
B((Enterprise Users)):::external
C([New Records Detection]):::feature
D([ML-Based Matching]):::feature
E(AWS Entity Resolution):::compute
F(Resolved Entity Graph):::storage
G([Match Confidence Scoring]):::feature
H(Historical Base Records):::storage
I([95% Faster Processing]):::feature
B ==>|"submits new records"| A
A -->|"identifies delta"| C
C -->|"compares against"| H
C -->|"applies models"| D
D -->|"generates"| G
G -->|"integrates into"| F
A -->|"powered by"| E
H -->|"up to 1B records"| F
A -.->|"delivers"| I
classDef announced fill:#ff9900,stroke:#ec7211,color:#fff,font-weight:bold
classDef compute fill:#e3f2fd,stroke:#1565c0,color:#1565c0
classDef storage fill:#e8f5e9,stroke:#2e7d32,color:#2e7d32
classDef feature fill:#fff3e0,stroke:#e65100,color:#e65100
classDef external fill:#f5f5f5,stroke:#616161,color:#616161
What's New
AWS Entity Resolution has launched General Availability support for incremental Machine Learning (ML) based matching workflows, enabling enterprises to process only newly added records rather than reprocessing entire datasets. This enhancement delivers a 95% reduction in processing time, handling 1 million incremental records in under one hour. The feature scales to support up to 50 million incremental records against datasets containing up to 1 billion historical base records.
How It Works
- Incremental ML matching workflows maintain a persistent state of previously processed and matched records, allowing the system to compare only net-new records against the existing resolved entity graph rather than recomputing all pairwise comparisons from scratch.
- When a new workflow run is triggered, AWS Entity Resolution identifies records added since the last successful run, applies its ML-based matching models to those incremental records, and integrates the results into the existing match output.
- The underlying ML models leverage probabilistic and deterministic matching techniques trained on entity attributes (such as names, addresses, and identifiers) to determine match confidence scores, and the incremental architecture ensures these models are applied selectively and efficiently without sacrificing match quality across the full historical dataset.
Why It's Important
- Prior to this launch, even a single new record addition forced a full dataset reprocessing cycle that could consume up to two days and cost thousands of dollars, making continuous or near-real-time entity resolution economically and operationally impractical for large enterprises.
- This bottleneck effectively excluded AWS Entity Resolution from use cases requiring frequent data refreshes, such as customer data platforms, fraud detection pipelines, and real-time identity graphs.
- By reducing processing time by 95% and dramatically cutting infrastructure costs, this feature makes AWS Entity Resolution viable for high-frequency, production-grade workloads at billion-record scale, unlocking new categories of enterprise use cases that were previously infeasible on the platform.
How It's Different
- The previous ML-based matching workflow in AWS Entity Resolution operated exclusively as a full-refresh batch process, requiring the entire dataset to be reprocessed on every run regardless of how many new records were added.
- This is fundamentally different from the new incremental approach, which introduces stateful workflow execution and delta processing.
- Compared to third-party entity resolution solutions or custom-built alternatives that enterprises were forced to adopt as workarounds, the incremental ML workflow is natively integrated into the AWS ecosystem, eliminating the need for external orchestration, custom deduplication logic, or separate change-data-capture pipelines.
- The combination of ML-grade matching accuracy with incremental efficiency is a meaningful architectural advancement over both the prior AWS capability and many competing solutions.
When to Prefer It
- Incremental ML matching workflows are the preferred choice whenever entity resolution must be performed on a recurring or continuous basis against a large, growing dataset.
- Ideal scenarios include customer master data management where new customer records are ingested daily or hourly, fraud detection systems requiring up-to-date identity graphs, healthcare patient matching pipelines with ongoing record additions, and retail or e-commerce platforms consolidating customer identities across channels in near real-time.
- This feature is particularly well-suited when the incremental record volume per run is significantly smaller than the total historical dataset, which is precisely the condition under which the efficiency gains are most pronounced.
- Organizations that previously avoided AWS Entity Resolution due to cost or latency constraints on full-refresh workflows should now re-evaluate it as a primary solution.
Availability
- Incremental ML-based matching workflows are now Generally Available (GA) and accessible in all AWS Regions where AWS Entity Resolution is currently supported.
- The feature supports incremental batch sizes of up to 50 million new records per workflow run, applied against historical base datasets of up to 1 billion records.
- There are no announced preview or opt-in requirements; customers can begin using the feature immediately through the AWS Entity Resolution console, API, or SDK by configuring an incremental ML matching workflow as described in the updated user guide.
- No specific regional exclusions were noted in the announcement beyond the general regional availability of the AWS Entity Resolution service itself.
