Migrate from Databricks

This guide covers moving a Databricks environment to LakeSail.

Before you begin

• Complete the Quickstart, so you have a cluster that can run jobs.
• Scan your code as Check Your Spark Code describes, to find Spark features that Sail does not support.

Migration sequence

Work through the migration in this order:

1. Choose what happens to the catalog and data
2. Verify table access
3. Transfer notebooks
4. Recreate jobs
5. Configure job settings
6. Update external tools and consumers
7. Validate and cut over

As you move each notebook and job, make the code changes for LakeSail.

Repeat steps 2 through 7 for each group of related workloads. A group should include any upstream jobs and downstream consumers that must change together.

1. Choose what happens to the catalog and data

Connecting a catalog lets LakeSail find your tables. To read and write the table files, the compute profile also needs S3 access: the network's workspace bucket, or an external data target granted to the profile. Plan that access for every path below.

Choose one of three paths:

1. Keep Unity Catalog when Databricks and LakeSail must both work with the same tables.
2. Copy the tables into the workspace bucket when you are leaving Databricks entirely.
3. Use a catalog outside Databricks when Databricks uses AWS Glue, Apache Polaris, or another external catalog.

Path 1: Keep Unity Catalog during the migration

Use this path when Databricks and LakeSail must both work with the same tables, for example during a staged migration.

For each catalog and schema in scope:

1. Enable external data access on the Databricks metastore. In Catalog Explorer, open Metastore from the gear menu and turn it on in the Details tab.
2. Create a Databricks personal access token for a user or service principal with EXTERNAL USE SCHEMA and the required catalog, schema, and table privileges. Schema owners do not have EXTERNAL USE SCHEMA by default. Store the token as a managed secret.
3. Connect a Unity Catalog catalog for Delta tables, or an Iceberg REST catalog pointed at the Unity Catalog Iceberg endpoint for supported Iceberg access.
4. Register the S3 location of each table as an external data target, and grant it to the compute profile. The prefix must cover the table's metadata, data files, and the location for new files. Follow every step on Connect External S3 Data, including the bucket policy. Objects encrypted with SSE-KMS are not supported.
5. Test access as described in Verify table access.

Unity row filters and column masks do not apply

LakeSail reads the table files directly, so a workload sees every row and column of a table it can reach. Inside LakeSail, roles control who can run what.

Managed Unity Catalog tables

Managed tables are not supported yet. If you have managed tables, either:

• Deep clone each one into an external location you control, then use path 1 for the clone.
• Move them to another catalog with path 2.

Path 2: Copy the tables into the workspace bucket

Use this path when you are leaving Databricks entirely. You copy each table into an AWS Glue catalog you connect in LakeSail, with its data in the workspace bucket.

1. Set up path 1 so LakeSail can read the source tables. If LakeSail cannot read a table, for example a managed table or a table in Databricks default storage, deep clone it to an external location you control first.

2. Connect an AWS Glue catalog, and add it to the same compute profile as Unity Catalog.

3. Create a database whose location is under the workspace bucket's catalogs/ folder:

   CREATE DATABASE glue.sales LOCATION 's3://<workspace-bucket>/catalogs/sales/';

4. Copy each table:

   CREATE TABLE glue.sales.orders USING delta AS SELECT * FROM unity.sales.orders;

   glue and unity are the names you gave the catalogs in LakeSail. Use USING iceberg for Iceberg tables. The copy starts a new table, so table history and time travel do not carry over.

5. Test access as described in Verify table access.

6. Point writers at the new table, and keep the source until the migrated workloads pass validation.

7. Remove Unity Catalog from the compute profile and from any job with its own catalog list, then revoke its Databricks token.

Path 3: Use a catalog outside Databricks

Use this path when Databricks uses a catalog outside Databricks for your tables, for example AWS Glue or Apache Polaris.

If Databricks and LakeSail both write to the same Iceberg tables, check that Sail supports the Iceberg features those tables use.

1. Connect the catalog and add it to the compute profile.
2. Register the S3 location of each table as an external data target, and grant it to the compute profile. The prefix must cover the table's metadata, data files, and the location for new files. Follow every step on Connect External S3 Data, including the bucket policy. Objects encrypted with SSE-KMS are not supported.
3. Test access as described in Verify table access.

2. Verify table access

Test a representative table from every storage location and table type used by the workload.

Start with discovery and a read:

SHOW TABLES IN catalog_name.schema_name;

SELECT COUNT(*)
FROM catalog_name.schema_name.table_name;

Then test the operations the workload uses, for example:

• Read the latest and an older snapshot or version.
• Read the newest partition.
• Write to a temporary table or isolated target.
• Run an append, overwrite, merge, or schema change.

If an operation fails, check that Sail supports it on the Delta Lake or Iceberg feature page.

A read that fails with Access denied can mean the path is outside every granted prefix, a prefix path does not end with /, the grant lacks the access the read needs, the bucket policy is missing the profile's statements, or the objects use SSE-KMS encryption, which is not supported. See Troubleshooting for every check.

3. Transfer notebooks

LakeSail notebooks are marimo notebooks. LakeSail cannot import Databricks source exports or .ipynb files, so convert each notebook first:

1. Export it from Databricks as an IPython notebook (.ipynb).
2. Convert it locally with marimo convert notebook.ipynb -o notebook.py.
3. Remove magics, widgets, and dbutils calls, as Code changes for LakeSail describes.
4. In Notebooks, click Create Notebook and choose the file under Start from a marimo file. To replace the file of an existing notebook, stop the notebook and click Import on its details page. The file can be up to 1 MiB.

Only the member who creates a notebook can open its editor.

Remove any assumptions about DBFS, cluster IDs, or workspace files.

4. Recreate jobs

Update each task's code as Code changes for LakeSail describes, then move it according to how it runs today:

• Move a SQL query task to a SQL job. Limitation: Sail binds parameters only in SELECT queries. A SQL job whose statement writes data, such as INSERT or CREATE TABLE AS, fails when it declares parameters.
• Move a Python script to a Python file job. A file job runs a single .py file. Package shared modules as a wheel, and add it to the libraries of the job or its compute profile.
• Move a Python wheel task to a wheel job with the same package and entry point.
• Move a notebook task to a notebook job. Transfer the notebook first, as Transfer notebooks describes, then click Run as job on it. Each run uses the last saved version of the notebook.

Databricks job permissions do not move. Each LakeSail job belongs to a team, and team and organization roles decide who can edit and run it. See Teams and access.

LakeSail does not connect to Git. Upload each job's .py file or wheel as a job file, and set its environment variables on the job or its compute profile.

5. Configure job settings

After a job runs manually, match the settings of the Databricks job it replaces:

1. Pause the job, then convert the Databricks Quartz schedule to a standard five-field cron expression. A paused job's schedule does not fire until cutover, and manual runs still work.
2. Preserve the time zone.
3. Set the concurrency policy and missed-schedule policy. For Databricks max_concurrent_runs, use the allow policy with Max concurrent runs. Runs started manually or through the API are never blocked by the limit, but they count towards it, so they can delay scheduled runs.
4. Set the timeout with the job's timeoutSeconds field in the API. It counts from when the run reaches Running, so starting the runner and installing libraries count towards it. See Timeout.
5. Set Automatic restarts. These restart a run after a lost pod, a platform error, an out-of-memory failure in Sail, or when your code raises SailRetryableError or exits with code 75. An error in your own code is not restarted. See Automatic restarts. To re-run a failed, timed-out, or cancelled run by hand, as with a Databricks repair run, click Retry Run. It runs the whole job again, because a LakeSail job has one task.
6. Convert widgets and task parameters to job parameters. For the time a scheduled run is for, read LAKESAIL_LOGICAL_TIME instead of a Databricks dynamic value reference. SQL jobs cannot read it.
7. Recreate failure and timeout notifications.
8. Move required secret values into managed secrets, not into source code or job definitions.

Databricks Spot and instance pool settings have no equivalent. Workers use on-demand capacity, sized by the compute profile.

6. Update external tools and consumers

Check every system that calls Databricks or reads its outputs.

Databricks Connect

Code that uses Databricks Connect connects to a LakeSail session instead. Uninstall databricks-connect, and install PySpark 3.5 or later (pip install pyspark). Issue a connection token, and set SPARK_REMOTE to the sc:// URL it shows. Tokens expire, and a session closes after its maximum duration (7 days by default; organization settings can change it), so fetch a new token rather than storing one.

dbt

Move projects from dbt-databricks to dbt-sail. Recheck adapter-specific materializations, incremental models, quoting, and session properties.

BI tools and SQL warehouses

LakeSail does not provide a direct replacement for a Databricks SQL warehouse. Keep the current serving layer or write results to the system that serves the BI tool.

Downstream data consumers

Update consumers only when the catalog name, table name, storage location, schema, or freshness contract changes.

7. Validate and cut over

Before you cut over, check that the LakeSail job produces the results you expect. Compare, for example:

• Row counts and key totals.
• Null, duplicate, and rejected-record counts.
• Latest partitions and watermarks.
• Schemas, including nullability and column order where consumers depend on them.
• Snapshot or version behavior for Delta and Iceberg tables.
• Runtime, retries, timeouts, and failure behavior.

When the results match:

1. Stop or pause the Databricks writer.
2. If you copied tables into the workspace bucket, bring over any data Databricks wrote since you copied them. For example, run INSERT INTO glue.sales.orders SELECT * FROM unity.sales.orders WHERE updated_at > '<time of the copy>', using a column that records when each row was written.
3. Set the LakeSail job to Active so its schedule starts running. When asked about missed runs, choose Skip Them: Databricks already ran them.
4. Observe at least one complete production cycle.
5. Point downstream consumers at the migrated output if its location changed.

If you need to return to the Databricks job after cutover, pause the LakeSail job first, so the two jobs never write to the same table at the same time.

Before you delete the Databricks job, export any run history you need to keep. Run history does not move to LakeSail.

Code changes for LakeSail

Make these changes to each notebook, Python script, and wheel as you move it in Transfer notebooks and Recreate jobs.

Scan your code with the Spark-to-Sail scanner to find Spark features that Sail does not support. See Check Your Spark Code.

Get the Spark session

Databricks provides an implicit spark. In LakeSail, create it yourself with SparkSession.builder.getOrCreate():

• In a Python file job, call it at the top of the file.
• In a wheel, call it inside the entry point, not at module level.
• In a notebook, it returns the notebook's session.

Call spark.stop() when Spark work ends. Use try and finally so the session closes if your code raises an error:

from pyspark.sql import SparkSession

spark = SparkSession.builder.getOrCreate()
try:
    ...  # your existing Spark code
finally:
    spark.stop()

SparkContext and RDDs are not available. See The Spark session.

Replace dbutils

• Replace dbutils.secrets.get with secret-backed environment variables on the job or compute profile.
• Replace dbutils.widgets with job parameters. A Python job receives each one as --name=value. A SQL job reads it as :name.
• Replace dbutils.fs table operations with DataFrame reads and writes.
• Replace raw object operations with the cloud provider SDK when they are still required. In a notebook, that client does not get the profile's S3 access, so give it its own credentials through a managed secret.
• Replace dbutils.notebook.run with separate jobs, run in order through the API.
• Replace dbutils.notebook.exit with a normal return or process exit.
• Replace display(df) with df.show() in jobs. Return the DataFrame from an interactive notebook cell.

For example, replace:

password = dbutils.secrets.get("production", "database-password")

With an environment variable supplied from a secret:

import os

password = os.environ["DATABASE_PASSWORD"]

Remove notebook-only commands

• Replace %sql with a marimo SQL cell whose engine is spark, spark.sql(...), or a SQL job.
• Replace %pip with libraries on the compute profile.
• Replace %run with Python imports from a wheel, or separate jobs.
• Replace %sh with Python code or a packaged dependency.
• Remove # MAGIC and Databricks notebook marker lines from exported source.

What cannot move yet

LakeSail does not support these yet. Rewrite them in PySpark or SQL, or keep them on Databricks for now:

• RDD and SparkContext workloads.
• Lakeflow Declarative Pipelines (formerly Delta Live Tables).
• Lakeflow Jobs with more than one task. A LakeSail job runs a single task, so task dependencies, conditions, and values passed between tasks have no equivalent.
• JAR, Scala, and R jobs.
• Structured Streaming with unsupported sources, sinks, or checkpoint behavior.
• Databricks-only SQL commands and table features.
• SQL tasks that run a .sql file, an alert, or a dashboard.
• Init scripts that install operating-system packages or change the host.

For details, see Migrate from Spark, Supported SQL features, and Delta Lake feature support.

If you need one of these, email support@lakesail.com with a description of the workload.

Next steps

• Connect a catalog
• Connect external S3 data
• Copy data to the workspace bucket
• Manage secrets
• Defining jobs
• Scheduling
• Runs and debugging
• Notifications
• Notebooks
• Personal access tokens