Databricks adds SQL features to speed up lakehouse migrations

Databricks has expanded its SQL capabilities to support direct migrations of legacy database procedures to the Databricks Lakehouse. By mapping traditional PL/SQL and Teradata BTEQ scripts directly to Databricks SQL Scripting, the platform allows organizations to preserve their existing business logic without rewriting code in Python or Spark. Databricks reports that these new features can slash migration timelines by 50% to 75% for complex stored procedures, enabling existing SQL teams to continue maintaining their own workflows.

The update introduces native cursor support in Databricks Runtime 18.1, allowing developers to use standard commands like OPEN, FETCH, and CLOSE. Legacy cursor attributes like %NOTFOUND are replaced by a CONTINUE HANDLER FOR NOT FOUND, while loop labels and LEAVE replace EXIT WHEN. For temporary tables, session-scoped CREATE TEMP TABLE commands serve as direct replacements, though developers must manually drop tables before recreating them because CREATE OR REPLACE TEMP TABLE is not yet supported. The platform's procedural toolkit now fully supports control flows including IF/ELSE, WHILE, FOR, LOOP, REPEAT, LEAVE, ITERATE, and SIGNAL/RESIGNAL.

To handle transactional integrity, Databricks uses BEGIN ATOMIC ... END blocks to manage automatic commits and rollbacks. This setup introduces row-level conflict detection, a significant shift from the table-level locking used by competitors like Oracle and Snowflake. To use this feature, practitioners must enable the catalogManaged table feature on existing Delta tables using a TBLPROPERTIES alter statement. Once deployed, these migrated procedures are registered in Unity Catalog, granting them column-level lineage and access controls that legacy schemas often lacked.