Databricks Concurrency: How Delta Lake Handles Concurrent Writes

Two pipelines. One Delta table. Both start at the same time. What happens? 

This is a common situation in Databricks. 

A streaming job may continuously ingest data while a batch pipeline performs a MERGE. At the same time, a CDC pipeline or another notebook may update the same Delta table. 

Parallel processing improves throughput, but it also introduces concurrency challenges. Poorly designed concurrent workloads can result in transaction conflicts, failed jobs, duplicate processing, metadata conflicts, and performance problems. 

Delta Lake addresses these challenges using ACID transactions, transaction snapshots, and Optimistic Concurrency Control (OCC). 

Let’s understand how it works and what engineers should do when concurrency issues occur. 

What Is Concurrency? 

Concurrency means multiple workloads access or modify shared data at overlapping times. 

A typical Databricks environment may look like this: 

Batch ─────┐ 
Streaming ─┼──► Delta Table 
CDC ───────┘ 
 

All three workloads may interact with the same table. 

However: 

Same table does not automatically mean conflict. 

For example: 

Job A → Customer 1001 
Job B → Customer 2001 

is different from: 

Job A → Customer 1001 
Job B → Customer 1001 

The possibility of a conflict depends on what the transactions read, modify, delete, or change. 

How Delta Lake Handles Concurrency 

Traditional databases often use pessimistic locking to coordinate concurrent transactions. 

Delta Lake primarily uses Optimistic Concurrency Control (OCC) instead. 

The idea is simple: 

Let transactions work independently and check for conflicts when they commit. 

OCC Flow 

Read Snapshot 
     ↓ 
  Process 
     ↓ 
Prepare Changes 
     ↓ 
Commit Validation 
     ↓ 
┌─────────────┐ 
│             │ 
No Conflict  Conflict 
│             │ 
↓             ↓ 
Commit       Fail 
               ↓ 
          Retry / Handle 
 

This approach allows multiple workloads to operate concurrently while protecting transactional consistency. 

Understanding the Transaction Snapshot 

Suppose a Delta table is currently at Version 125. 

Two jobs start at nearly the same time. 

Both read Version 125 and process their data independently. 

Job A finishes first: 

Version 125 → Job A → Commit → Version 126 
 

Job B then tries to commit. 

Delta validates Job B against the newer table state. 

If the transaction is still valid, it can be committed. 

If a relevant conflict is detected, the transaction fails. 

This snapshot-based validation is the key concept behind Delta Lake concurrency. 

A Practical MERGE Example 

Consider a customer table: 

customer_id | name | city 
————|——|——— 
1001        | Amit | Ahmedabad 
 

Now two jobs run concurrently. 

Job A 

Wants: 

Customer 1001 → Ahmedabad 

Job B 

Wants: 

Customer 1001 → Mumbai 
 

Both use a MERGE: 

MERGE INTO customer AS target 
USING customer_updates AS source 
ON target.customer_id = source.customer_id 
 
WHEN MATCHED THEN 
  UPDATE SET 
    target.name = source.name, 
    target.city = source.city 
 
WHEN NOT MATCHED THEN 
  INSERT *; 
 

Both jobs may initially read the same table version. 

If Job A commits first, the table moves to a newer version. 

When Job B attempts to commit, Delta checks whether Job A’s changes conflict with Job B’s transaction. 

If they conflict, Job B can fail with a concurrency exception. 

Important distinction 

This should not simply be described as: 

“Delta silently overwrites one update.” 

Instead, distinguish between: 

Application-level lost update: 

An application writes stale data without properly handling concurrent changes. 

Delta transaction conflict: 

Delta detects a conflicting transaction and prevents that transaction from committing. 

The failed pipeline may still need to retry, reconciliation, or business-specific logic. 

What Actually Causes a Concurrency Conflict? 

The important question is not: 

“Are two jobs using the same table?” 

Ask instead: 

“Are their operations overlapping?” 

Different records 

Job A → Customer 1001 
Job B → Customer 2001 

Potentially independent. 

Same record 

Job A → Customer 1001 
Job B → Customer 1001  

Greater potential for conflict. 

Update + Delete 

Job A → UPDATE Customer 1001 
Job B → DELETE Customer 1001 

Again, the operations overlap. 

Data + Metadata 

Job A → MERGE INTO customer 
Job B → ALTER TABLE customer ADD COLUMN … 

This can result in a metadata/schema conflict. 

Therefore, concurrency analysis should consider: 

• Data overlap 

• Files affected 

• Read/write behavior 

• Deletes 

• Schema changes 

• Table metadata changes 

Common Concurrency Errors 

Error	What it generally indicates	Typical action
ConcurrentAppendException	Concurrent changes affected relevant files	Check overlap; retry if safe
ConcurrentDeleteReadException	Data being read was deleted	Refresh state and retry
ConcurrentDeleteDeleteException	Concurrent operations affected overlapping data	Coordinate workloads
MetadataChangedException	Table metadata changed during the operation	Coordinate schema/metadata changes
ConcurrentTransactionException	Concurrent streaming transaction issue	Check streaming/checkpoint configuration
ProtocolChangedException	Table protocol changed	Coordinate protocol changes

Do not automatically retry every exception. 

First determine what caused the conflict. 

Idempotency: The Key to Safe Retries 

Concurrency errors often lead to retries. 

But a retry can create duplicate records if the pipeline is not idempotent. 

Common techniques include: 

• Stable business keys 

• MERGE 

• Source deduplication 

• Batch IDs 

• Streaming checkpoints 

• Deterministic transformations 

For example: 

MERGE INTO customer AS target 
USING customer_updates AS source 
ON target.customer_id = source.customer_id 
 
WHEN MATCHED THEN 
  UPDATE SET * 
 
WHEN NOT MATCHED THEN 
  INSERT *; 
 

However: 

MERGE alone does not automatically make a pipeline idempotent. 

The source must also be correctly deduplicated and the business key must uniquely identify the logical record. 

Retry Strategy 

A good retry strategy should be deliberate. 

Recommended practices: 

• Retry only transient concurrency failures. 

• Limit the number of retries. 

• Use exponential backoff. 

• Refresh the latest state before retrying. 

• Ensure the operation is idempotent. 

• Log retry attempts. 

• Do not endlessly retry schema or logic errors. 

Streaming + Batch 

A common production architecture is: 

Streaming → Bronze → Silver 
                    ↑ 
                Batch / CDC 
 

Streaming and batch workloads can coexist, but their responsibilities should be clear. 

For example: 

• Streaming: continuous ingestion 

• Batch: controlled transformations or historical corrections 

• CDC: applying source-system changes 

Avoid having many independent workloads continuously rewrite the same “hot” table unless there is a strong business reason. 

Also ensure streaming queries are correctly configured with their checkpoints. 

Pipeline Orchestration 

Sometimes the best solution to concurrency is simply to avoid unnecessary concurrency. 

Suppose: 

Pipeline A → Load Customer 
Pipeline B → Transform Customer 
 

If Pipeline B depends on Pipeline A, don’t unnecessarily run them together. 

Instead: 

Pipeline A 
    ↓ 
Complete 
    ↓ 
Pipeline B 
 

This leads to a simple rule: 

Parallelize independent workloads. Serialize dependent workloads. 

Retries are useful for transient conflicts, but orchestration is often a better solution when pipelines are logically dependent. 

Small Files: A Different Problem 

Concurrent workloads can create many small files. 

For example, several jobs may each generate hundreds of small Parquet files. 

Over time: 

Delta Table 
├── file_001 
├── file_002 
├── file_003 
├── … 
└── file_5000 
 

This can cause: 

• Slower queries 

• More metadata 

• Higher processing overhead 

• Longer maintenance operations 

But remember: 

Small files are primarily a performance problem, not a transaction-conflict problem. 

Depending on the workload, techniques such as optimized writes, auto compaction, and OPTIMIZE can help manage file layout. 

Best Practices 

For production Databricks workloads: 

1. Know Your Writers 

Understand which jobs read and write each Delta table. 

2. Avoid Unnecessary Concurrent Writes 

If multiple pipelines do not need to update the same table simultaneously, don’t make them compete. 

3. Design for Idempotency 

Use business keys, MERGE, deduplication, batch IDs, and checkpoints. 

4. Retry Intelligently 

Use limited retries, backoff, and state refresh. 

5. Reduce Data Overlap 

Use appropriate predicates and data layout to minimize unnecessary conflicts. 

6. Control Schema Changes 

Coordinate schema and metadata changes with production workloads. 

7. Separate Streaming and Batch Responsibilities 

Give each workload a clear role. 

8. Orchestrate Dependencies 

If B depends on A: 

A → B 

rather than allowing both to compete. 

9. Manage Small Files 

Use appropriate optimization and compaction techniques. 

Troubleshooting Checklist 

When a concurrency error occurs, follow this sequence: 

Identify all writers 
        ↓ 
Check DESCRIBE HISTORY 
        ↓ 
Check overlapping execution times 
        ↓ 
Determine data/file overlap 
        ↓ 
Check schema/metadata changes 
        ↓ 
Is retry safe? 
        ↓ 
Retry / Serialize / Redesign 
 

A useful starting point is: 

DESCRIBE HISTORY customer; 
 

Then classify the problem: 

Transaction conflict → investigate overlap and retry if safe. 

Metadata conflict → coordinate schema/metadata changes. 

Duplicate processing → improve idempotency. 

Performance issue → investigate small files and table layout. 

Key Takeaways 

The most important thing to remember is: 

Same table does not automatically mean conflict. 

The real question is what the concurrent transactions are reading, modifying, deleting, or changing. 

Delta Lake primarily uses Optimistic Concurrency Control rather than traditional database-style locking. 

The basic model is: 

Read Snapshot 
      ↓ 
   Process 
      ↓ 
Commit Validation 
      ↓ 
No Conflict → Commit 
Conflict    → Retry / Reconcile 
 

For production pipelines, combine Delta’s transaction guarantees with: 

• Idempotent design 

• Safe retry strategies 

• Controlled schema changes 

• Appropriate data layout 

• Streaming checkpoints 

• Pipeline orchestration 

• Small-file management 

Final Thought 

Concurrency is not something to eliminate. 

It is something to design for. 

The goal is simple: 

Run independent workloads in parallel, coordinate dependent workloads, and make every retry safe. 

That is how you build reliable and scalable Databricks pipelines.