Defining Jobs

A job is a reusable, versioned workload. This page covers the static definition: what a job is, the three job types, and how drafts and versioning work.

For scheduling, see Scheduling. For runs and debugging, see Runs & Debugging.

Prerequisites

Jobs run on a cluster with a compute profile. Before you can save a job:

• A cluster must have Readiness set to Ready. A Deployed status alone does not guarantee that work can run. See Clusters.
• A compute profile must target that cluster. Each network provides a default Glue catalog automatically. To use other data, connect a catalog and add it to the profile or job.

Create a job

1. Open Jobs in the sidebar and click Create Job.
2. Give it a Name and assign it to a Team. Team and organization roles determine who can edit and trigger the job.
3. Pick the Job type (SQL, Python, or Notebook) and fill in the source (below).
4. Select a compute profile.
5. (Optional) Customize the profile's catalogs for this job.
6. (Optional) Add job-specific Sail environment variables. These override profile variables with the same name.
7. Click Save.

Saving creates the first published version of the job.

SQL jobs

Use SQL for analytics and ETL transformations. There are two source variants:

• Saved query: reference an existing query by ID. The query text is resolved at run time, so edits to the query flow through to the next run of the job.
• Snapshot: paste inline SQL text into the job. You can optionally track the origin (e.g. a Git path) for drift detection.

A file variant pointing at SQL in workspace storage is reserved in the API but not supported yet.

Pick saved query when the SQL is shared across jobs; pick snapshot when the SQL belongs to exactly one job and shouldn't drift.

LakeSail creates and stops the Spark session for SQL jobs.

Python jobs

Use Python when you need the Python ecosystem, such as ML or third-party libraries. Two source variants:

• Python wheel: run a wheel by package name and entry point. Best for reusable code that's versioned in a package registry.
• Python file: run a single .py workspace file or S3 URI. Best for scripts and ad-hoc work.

The Spark session

Python jobs use Spark Connect. You can use the DataFrame and SQL APIs. SparkContext and RDDs are not available. See Check Your Spark Code.

LakeSail sets SPARK_REMOTE for each Python job. Use SparkSession.builder.getOrCreate() to connect to Sail.

Python file, wheel, and notebook jobs should call spark.stop() when Spark work ends. Use try and finally to close the session if your code raises an error.

LakeSail requests Max nodes worker nodes once when cluster compute starts. Karpenter can remove unused worker nodes after the workload's prewarm hold and the configured consolidation delay. Prewarming also has a deadline that includes scheduling and node launch, so late-starting capacity may be held for less time. This deadline does not limit Spark execution. A later session can need new nodes.

SparkSession.builder.getOrCreate() reuses the active or default session in the same Python process. It creates a session if neither exists. Use the same spark variable throughout the job.

In cluster mode, SparkSession.builder.create() creates a separate Spark session and requests another worker pool. Max nodes limits workers per client session. It does not limit the total worker count for the workload. Three active Spark sessions with Max nodes set to 4 can use twelve worker nodes. These sessions share the driver. Each worker runs on a separate node.

Workers use available nodes for the workload. LakeSail adds nodes when more capacity is needed. Available capacity and AWS quotas still apply.

Python files

Get the session at the start of your file:

from pyspark.sql import SparkSession

spark = SparkSession.builder.getOrCreate()
try:
    df = spark.table("production.sales.orders")
    df.show()
finally:
    spark.stop()

Python wheels

Use the standard PySpark API in your entry point:

from pyspark.sql import SparkSession

def main():
    spark = SparkSession.builder.getOrCreate()
    try:
        spark.sql("SELECT 1").show()
    finally:
        spark.stop()

To run either example outside LakeSail, start a session and issue a connection token. Set SPARK_REMOTE to the sc:// URL shown for that session.

Arguments and parameters

Both variants take two kinds of input on the command line:

• Arguments are fixed. Every run of the version passes them, in order. Use them for how the code is invoked, such as --mode incremental or a positional path.
• Parameters change per run.

Your code receives the arguments first, then each parameter the run passed as --name=value. Read them with argparse as usual:

import argparse

parser = argparse.ArgumentParser()
parser.add_argument("--mode")
parser.add_argument("--date", required=True)
parser.add_argument("--table", default="orders")
args = parser.parse_args()

Your code can also read LAKESAIL_JOB_ID and LAKESAIL_JOB_RUN_ID from its environment, and LAKESAIL_LOGICAL_TIME (see Which time a run is for).

Notebook jobs

A notebook job runs one of your own notebooks on a schedule or from an orchestrator. Pick it under Notebook, or start from the notebook with Run as job.

Each run copies the notebook's saved contents when it starts, so later edits do not affect it. It runs on the job's compute profile and catalogs, not the notebook's.

Arguments and parameters reach the notebook on the command line. Read them with mo.cli_args().

Output from print() appears in the run's logs. Cell outputs are not saved.

Parameters

Parameters are the inputs that change per run, such as a date or a table name. The version declares each parameter by name, and each run passes a value for it. This is how an orchestrator such as Airflow passes its inputs. SQL, Python, and notebook jobs declare parameters the same way.

When you declare a parameter, you choose what happens to a run that leaves it out:

Kind	A run that leaves it out	Python and notebook jobs	SQL jobs	Scheduled versions
Required	Is refused	Yes	Yes	No
Optional with a default	Gets the default	Yes	Yes	Yes
Optional without a default	Passes nothing for it, so your code handles that case	Yes	No	Yes

Two rules shape the table:

• A scheduled run passes no values of its own, so it gets only defaults. A scheduled version cannot have required parameters, and a scheduled SQL job needs a default for every parameter.
• A SQL query cannot read a marker that has no value, so in a SQL job every parameter is either required or has a default.

A required parameter cannot have a default, since the default would never be used. A run that passes a name the version does not declare is refused. The run's detail shows the values the run used, defaults included.

Parameter values are strings, and anyone who can read the run can read them. Never pass a secret as a parameter; reference it from the job's Sail environment variables instead.

In a SQL query

A query reads a parameter as a named marker, :name. Values are strings, so cast where the query needs another type:

SELECT *
FROM orders
WHERE order_date = CAST(:day AS DATE)
  AND region = :region

The query is not checked against the declarations when you save it. The run fails if the query has a marker without a value. A value the query does not use is ignored. A Python job's parameter names hold letters, digits, underscores, periods and hyphens, and do not start with a hyphen. A SQL job's names hold only letters, digits and underscores, and start with a letter or underscore: the query reads :run-date as :run minus date. Use run_date instead.

For now, parameters work only in queries (SELECT). This is a current limit of Sail, which runs the SQL, and it will be lifted. A command such as INSERT or CREATE TABLE AS fails whenever a run passes a value, and a parameter with a default always passes one. So until then, a SQL job that writes data cannot declare parameters; the run fails with a message that says so.

Job files

Python files, wheel libraries, and requirements files can come from two places:

• Workspace file: select an existing file or upload one while editing the job. LakeSail stores it in the workspace S3 bucket for the compute profile's network. The job's selected team determines which files appear in the picker.
• S3 URI: enter the full URI of a file in the scripts/ location of the network workspace bucket. Using Upload places the file there automatically.

Uploaded files live in the workspace bucket under scripts/, in your AWS account. They are not removed when a job stops referencing them or is deleted; clean them up in S3 when you no longer need them.

Changing the job's team or moving it to a compute profile on another network changes which workspace files are available. Select or upload the file again after either change.

File	Required extension	Upload limit
Python job	.py	10 MiB
Wheel library	.whl	100 MiB
Requirements file	.txt	1 MiB

Compute

Jobs run on the cluster's compute nodes (not the management nodes), sized through a compute profile, which is a named, reusable bundle of engine and sizing settings. Every job references one; pick an existing profile or create one inline. The profile sets:

• Execution mode: standalone (single Sail pod) or cluster (distributed driver + workers).
• Driver: instance type and disk for the coordinating pod.
• Worker (cluster mode): instance type, disk, and max nodes per client session for the Sail workers.
• Libraries and environment: PyPI/wheel dependencies and Sail config to install before the run.

Size the profile for the job's actual working set. Because the compute lives in the profile, editing the profile changes the next run of every job that references it without redeploying the jobs. See Compute Profiles for sizing and Limits & Quotas for job limits.

Catalogs

Jobs inherit the catalogs and default catalog from their compute profile. In Catalogs, click Customize to add or remove catalogs or choose a different default for this job. Customizing replaces the profile's catalog set.

A catalog tells Sail where tables are stored. It does not copy the data or grant storage access. For S3-backed tables, the data must be in the network workspace bucket or in an external data target attached to the job's compute profile. See Connect a Catalog.

For SQL jobs that use a saved query, the query's catalogs also apply. The job's catalog settings take precedence when the same catalog is attached more than once.

A catalog must be shared with every team assigned to the job.

Sail environment variables

Add job-specific variables under Execution. Compute profile variables apply first; a job variable with the same name overrides the profile value. Values can be literals, managed secret references, or JSON templates with secret references.

Automatic restarts and long-running jobs

Automatic restarts (maxRetries in the API) set how often a failed runner restarts within the same run before the run fails. The run keeps its status, Running, while it restarts. Leave the field blank to use the compute profile's value, enter 0 to disable restarts, or -1 to restart without limit.

Not every failure restarts. These do:

• the runner's pod is lost or crashes, for example when its node is replaced;
• a platform error;
• your code marks the failure as transient: it raises SailRetryableError, or exits with code 75.

An error in your own code ends the run at once, because running the same code again would fail the same way. LakeSail also ends the run if it detects a memory failure in the Sail driver or a worker. A restart would run out of memory again. The run ends out_of_memory, and its message says which instance type to raise. See Worker memory failures for limits on this check.

For a job that should run until you stop it, set Automatic restarts to -1 and leave the timeout empty, so nothing ends the run on a schedule. A restart starts your code again from the beginning and keeps nothing from memory, so a long-running job should save its progress to storage, such as a Spark streaming checkpointLocation, and resume from it. When its node is drained, the job first gets SailStopRequested and a short time to save; see Stopping cleanly.

Automatic restarts differ from Retry Run, which starts a run that has already ended again, on fresh resources. See Retry a run.

Drafts and versioning

Every change to a live job flows through a draft, so you can iterate without disturbing scheduled runs.

1. Open the job and click Edit. Changes go into a draft. Drafts do not fire on schedule and are excluded from the main run history.
2. Test the draft by running it manually. Draft runs are clearly labeled.
3. When ready, publish the draft. It becomes the new live version, and the previous version moves to the version history.

The version history is searchable, and every run links to the job version it executed. Saved queries and compute profiles are resolved when work starts, so later edits to those resources can affect a retry of the same version.

Roll back

To revert to an earlier version:

1. Open the job's Versions tab.
2. Find the version you want and click Activate.

The activated version becomes the live one on the next run (scheduled or manual).

Teams and access

Every job is assigned to at least one team. The current job form selects one team.

• Members of an assigned team can act according to their team roles.
• Organization roles can also grant access.

Changing the Team field replaces the job's team assignment. It also changes which saved queries, catalogs, and workspace files are available in the form.

API reference

• Jobs: CreateJob, UpdateJob, drafts, versions, team assignment.
• Catalogs: required prerequisite for the job's data binding.