Salesforce Data Warehouse Integration: Incremental ETL
Updated: 3 days ago
Incremental Salesforce ETL pulls only the records that changed since the last run — using a stored watermark like SystemModstamp — instead of re-extracting an entire object every cycle. Enterprise IT teams build it with four pieces working together: an incremental start date per job, schema drift handling that extends target tables automatically, disciplined API limit management, and a scheduler that fits your Salesforce data warehouse integration into existing batch windows. Get those four right and warehouse replication stays reliable at scale.
What Incremental Salesforce ETL Actually Means
Full extraction pulls every row from every selected Salesforce object on every run. That approach works for a pilot, when a handful of objects and a few thousand rows barely register against any API allocation. It breaks down once an org holds millions of Account, Opportunity, and custom-object records, because each full pull burns API allocation, saturates network bandwidth, and stretches load windows into hours instead of minutes. Incremental data replication solves this by tracking a high-water mark — typically LastModifiedDate or SystemModstamp — and querying only records touched since that mark. The warehouse converges on the same end state as a full load, but at a fraction of the compute and API cost, which is why almost every serious Salesforce integration effort eventually migrates from nightly full loads to incremental design.
Sesame Software's platform makes this concrete with a job-level setting called Set Job Incremental Start Date, which resets the starting point an ETL job step uses for its next run. Admins can rewind a single job to backfill a gap without re-running every other job step in the warehouse, and they can confirm the current watermark before troubleshooting a sync that looks stale.
A Step-by-Step Framework for Salesforce Data Warehouse Integration
Enterprise teams that get incremental ETL right tend to follow the same five-step pattern, whether they build it themselves or configure a platform like Sesame Software to run it for them.
Step 1: Pick the Right Incremental Key Per Object
Most Salesforce standard objects expose SystemModstamp, which updates on any field change, including automated and workflow-driven updates that LastModifiedDate can miss. Custom objects need the same audit fields enabled explicitly. Standardize on one field per object and document the choice, because a mismatched key is the single most common cause of silently missed updates in Salesforce data synchronization.
Step 2: Set the Initial Load, Then Switch to Delta Queries
Run one full load to seed the warehouse, capture the watermark timestamp at that moment, and switch every subsequent run to a delta query filtered on the incremental key. This is where an explicit incremental start date matters: if a job step fails partway through, you need to reset the start date to the last confirmed watermark rather than guess at how far the sync got.
Step 3: Handle Schema Drift Before Salesforce Ships It
Salesforce orgs change constantly — admins add fields, teams create custom objects, and managed packages introduce their own schemas. A brittle pipeline hardcodes column lists and breaks the moment a field is renamed or added. Sesame Software's Data Warehouse Builder addresses this by auto-creating and updating target schemas as source objects change, so a new custom field lands in the warehouse without a manual DDL change. Enterprise teams should still review new fields for sensitivity and business relevance on a cadence, since automatic schema handling extends structure, not judgment.
Step 4: Reconcile Deletes, Merges, and Restores
An incremental query alone will never see a deleted record, because the record is gone from the source before the next sync runs. Sesame Software's platform tracks this state with a DELETE_FLAG on affected rows, a core concept in its ETL job design, so warehouse consumers can distinguish "still active" from "removed since last sync" without a full reconciliation pass against the live Salesforce org. Build the same discipline into a merge scenario: when two Salesforce records are combined into one, treat the surviving record as an update and the merged-away record as a delete, so downstream reports do not silently keep counting a record that no longer exists on its own.
Step 5: Schedule, Monitor, and Budget API Calls
An incremental job that runs too frequently on a large org can still exhaust daily API limits; one that runs too rarely lets the warehouse drift stale, which defeats the point of near real-time data synchronization in the first place. Configure a schedule — Sesame Software supports its own internal job scheduler alongside Windows Task Scheduler and Linux crontab — that matches your change volume, then watch the Job Monitor for insert, update, and error counts on every run so a quiet dashboard reflects a healthy sync rather than a silently broken one. Deliberate API limit management here, not raw frequency, is what keeps a Salesforce ETL pipeline sustainable long after the initial rollout, and it is also what separates a resilient database synchronization strategy from one that quietly degrades as record volume grows.
Budgeting Salesforce API Limits for Sustainable Sync
API limit management starts with query selectivity. SOQL filtering and selection criteria let a job step request only the fields and records a downstream use case actually needs, instead of every column on every object. Sesame Software's Salesforce connector applies this filtering at the object level during replication, which keeps each incremental run proportional to what changed rather than to org size.
It also pays to separate extraction from consumption. Once Salesforce data lands in the warehouse, BI tools, dashboards, and ad hoc SQL reporting query the replica directly — standard SQL tools, views, and stored procedures work against the warehouse without touching the Salesforce API a second time. That single design choice is often the biggest lever in API limit management: a hundred analysts running reports against a Snowflake or SQL Server replica cost zero additional Salesforce API calls, because only the sync job talks to Salesforce at all.
Common Mistakes That Break Incremental Salesforce ETL
Trusting LastModifiedDate everywhere. Some automated updates do not always touch it the way SystemModstamp does — verify per object rather than assuming.
Skipping the delete pass. Teams that only query "what changed" and never check "what disappeared" end up with warehouses that overstate active record counts.
Hardcoding schema. A field rename in Salesforce should not require an engineering ticket to fix a broken nightly job.
Running incremental jobs on a fixed schedule with no monitoring. A silently failing job step looks identical to a quiet day unless someone is watching insert and error counts.
How Sesame Software Simplifies Salesforce Data Warehouse Integration
Sesame Software has spent 30+ years and 15 patents building enterprise data replication, and incremental Salesforce ETL is where that experience shows up most directly. The platform's incremental start date controls, automatic schema handling, DELETE_FLAG-based reconciliation, and built-in job scheduling give IT teams a no-coding-required path to reliable data warehouse automation — without a custom pipeline that a departing engineer takes the institutional knowledge for. Sesame Software is SOC 2 Type II certified, which matters when the warehouse holds the same regulated Salesforce data your compliance team already audits.
The cost of getting this wrong is not abstract. Enterprise downtime runs roughly $9,000 per minute, and a broken overnight sync that a team does not catch until a stale morning dashboard is downtime by another name. Reliable incremental data replication is what keeps that number off your desk.
Talk to a Data Expert to see how Sesame Software's near real-time replication and API-efficient sync patterns fit your Salesforce data warehouse integration project: sesamesoftware.com.
FAQ: Incremental Salesforce ETL and Data Warehouse Integration
What are the tools helpful for Salesforce data warehouse integration?
Most enterprise teams rely on either a hand-built pipeline using Salesforce's REST or Bulk APIs plus a scheduler, or a dedicated platform that combines datasource connections, warehouse configuration, and job scheduling in one place. Evaluate candidates the way you would any other Salesforce data integration tools shortlist: ask how each handles schema drift, how transparent its API limit management is, and whether it is really built as dedicated Salesforce ETL tools or a general-purpose connector with Salesforce bolted on. A platform approach like Sesame Software's typically wins on maintenance cost once schema drift and API limit management become recurring problems rather than one-time setup tasks.
What is ETL in Salesforce?
ETL — extract, transform, load — in a Salesforce context means pulling records and metadata out of Salesforce objects, applying any transformations or filtering the destination requires, and loading the result into a target database or data warehouse such as SQL Server, PostgreSQL, or Snowflake. Salesforce ETL differs from generic ETL mainly in its API constraints and its need for careful incremental key and schema-drift handling.
What's the difference between a replica and an archive?
A replica is a near real-time, continuously synchronized copy of Salesforce data used for reporting and integration. An archive is data already moved and stored long-term elsewhere, typically for retention, audit, or cost reasons. Incremental Salesforce ETL builds and maintains a replica; archiving is a separate retention decision layered on top of it.
Do you need near real-time replication and scheduled snapshots?
Most enterprise Salesforce data synchronization projects need both. Scheduled incremental syncs (every few minutes to hourly, depending on API budget) keep the warehouse current, while a history-tracking table alongside each core table preserves point-in-time snapshots for audit and compliance without a separate archiving project.
Which databases can you sync Salesforce data into?
Salesforce data can be synchronized into most major relational and cloud-hosted databases, including SQL Server, Oracle, PostgreSQL, and cloud warehouses such as Snowflake, both on-premises and in the cloud. Sesame Software supports this range of destinations natively, so the choice of target database does not constrain the incremental ETL design.
Related Resources
