Stop the DBU leak: Optimizing Databricks Spark performance from AQE to Liquid Clustering

Summary: Databricks workloads tend to get more expensive as the data grows. A job that ran well at a smaller scale may start using more Databricks Units (DBUs) and take longer to finish. In many cases, the extra compute goes into shuffles, uneven joins, or a partitioning setup that no longer works well for the data.

To avoid six-figure cloud bill overruns, organizations need to move beyond static Spark performance tuning. Databricks’ Adaptive Query Execution (AQE) and Databricks Liquid Clustering bring runtime and storage optimization together, creating a more automated approach to Databricks performance optimization.

Two layers, one problem: Runtime vs. storage optimization

Spark performance optimization isn’t only about how a query runs. How the data is stored also affects how much work Spark has to do.

• Adaptive Query Execution (AQE): Adjusts the query plan while it is running, using the data Spark sees at that point.
• Liquid Clustering: Organizes data on disk so queries can avoid reading files they don’t need.

Databricks Liquid Clustering reduces the amount of data scanned, while AQE ensures the scanned data is processed with maximum efficiency at runtime. Full Databricks performance optimization requires addressing both.

Adaptive Query Execution: Letting Spark think at runtime

Spark usually builds a query plan before execution begins, based on the statistics available at the time. Adaptive Query Execution (AQE) gives Spark room to adjust that plan once the query is running. This takes some of the manual work out of Spark performance tuning.

• Partition coalescing: After a shuffle, AQE can combine smaller partitions instead of running a large number of small tasks.
• Join strategy changes: If filtering leaves one side of a join small enough, AQE can switch from a Sort-Merge Join to a Broadcast Hash Join.
• Skew handling: When one partition carries much more data than the others, AQE can split the work into smaller pieces rather than leaving one task to hold up the job.

The result is less time spent adjusting shuffle partitions by hand, fewer jobs held up by data skew, and better control over DBU usage.

Liquid Clustering: Retiring static partitioning

For a long time, Delta Lake tables were organized using Hive-style partitioning and Z-Ordering. That works, but it can become harder to manage as tables grow and access patterns change. Liquid Clustering offers another way to organize the data and lets you change clustering keys without rewriting the entire table:

SQL

-- Enable Liquid Clustering on a table
CREATE TABLE transactions (
transaction_id STRING, account_id STRING, amount DECIMAL(18,2)
) CLUSTER BY (account_id);
​
-- Modify clustering keys without rewriting data
ALTER TABLE transactions CLUSTER BY (account_id, transaction_id);

CLUSTER BY AUTO: Self-tuning tables

Driven by Predictive Optimization in Unity Catalog, Databricks removes manual column selection entirely:

SQL

CREATE TABLE transactions ( ... ) CLUSTER BY AUTO;

Databricks uses past query activity, such as filters and joins, to decide how the data should be clustered. It can also adjust the clustering keys and run maintenance tasks such as OPTIMIZE and VACUUM as needed.

Legacy Spark tuning vs. Modern Databricks platform

Together, AQE and Liquid Clustering replace several manual Spark performance tuning tasks with runtime and workload-driven optimization.

Enterprise use cases beyond retail

• Financial services: Fraud workloads can slow down when a small number of merchant accounts account for a large share of the activity. AQE helps distribute that work, while CLUSTER BY AUTO can adjust the table layout based on how analysts are querying the data.
• Healthcare and life sciences: Genomic data may be queried in different ways as research needs change. Databricks Liquid Clustering makes it possible to change clustering keys without having to rewrite the full table.
• Manufacturing and Industrial IoT: A sudden jump in telemetry data can put extra pressure on streaming workloads. AQE can adjust execution as the workload changes, helping with Spark performance optimization when data volumes fluctuate.

FinOps and DBU governance: Making savings stick

Keeping DBU costs under control also means knowing where the compute is going.

1. Start with a baseline: Record DBU usage for each pipeline before turning on AQE or Liquid Clustering. This gives you something to compare against later.
2. Use tags: Tag jobs and clusters so DBU usage can be traced back to the right team or business unit.
3. Check system tables: Use system.operational.predictive_optimization to see how automatic maintenance is running.
4. Set alerts: Flag unusual DBU usage at the job level before it turns into a larger cost issue.

This makes Databricks DBU optimization easier to track over time instead of treating it as a one-off performance exercise.

Jade Global’s delivery methodology

Jade Global’s Intelligent Data Solutions (IDS) practice partners with enterprises to modernize Spark workloads through a 3-phase framework:
 

• Assess: Audit Spark jobs, profile data skew, and benchmark DBU spend.
• Implement: Migrate tables to Delta Lake with AQE and CLUSTER BY AUTO.
• Optimize: Implement FinOps guardrails and continuous DBU tracking.

The framework connects Spark performance optimization with storage efficiency and DBU governance, helping teams address performance and cost together.

Accelerate your Spark optimization with Jade Global

AQE and Databricks Liquid Clustering address architectural barriers that drive up compute costs and cause pipeline delays. By deploying native auto-optimization in Databricks, you bring runtime performance, storage efficiency, and governance together in the Lakehouse.

As a trusted Databricks Strategic Partner, Jade Global combines deep platform expertise with domain-specific integration capabilities across Retail, Healthcare, Finance, and Manufacturing.

Ready to improve Databricks performance and control cloud spend? Connect with Jade Global to schedule a personalized Databricks Performance & DBU Optimization Assessment with our Data & AI experts.

Frequently Asked Questions:

How to use CLUSTER BY AUTO in Databricks?
Define the table with CLUSTER BY AUTO. It automatically adapts clustering keys based on query history, reducing manual Spark performance tuning.
​
How does Liquid Clustering reduce Databricks DBU costs?
Databricks Liquid Clustering reduces unnecessary data scans, helping answer how does Liquid Clustering reduce Databricks DBU costs while supporting Databricks DBU optimization.
​
How to stop high DBU consumption in Databricks?
Track DBU usage, enable AQE and Liquid Clustering, and set consumption alerts. Together, these improve Databricks performance optimization.
​
What is the Difference between Legacy Spark Tuning and Liquid Clustering?
The difference between Legacy Spark Tuning and Liquid Clustering is static versus workload-driven optimization. Liquid Clustering reduces manual partition management and supports ongoing Spark performance optimization.