Defining jobs

A job is a reusable, versioned workload. This page covers the static definition: what a job is, the two execution shapes, and how drafts and versioning work.

For scheduling, see Scheduling. For runs and debugging, see Runs & debugging.

Prerequisites

Jobs run on a cluster with a compute profile. Before you can save a job:

• A cluster must have Readiness set to Ready. A Deployed status alone does not guarantee that work can run. See Set up a cluster.
• A compute profile must target that cluster. Each network provides a default Glue catalog automatically. To use other data, connect a catalog and add it to the profile or job.

Create a job

1. Open Jobs in the sidebar and click Create Job.
2. Give it a Name and assign it to a Team. Team and organization roles determine who can edit and trigger the job.
3. Pick the Job type (SQL or Python) and fill in the source (below).
4. Select a compute profile.
5. (Optional) Customize the profile's catalogs for this job.
6. (Optional) Add job-specific Sail environment variables. These override profile variables with the same name.
7. Click Save.

Saving creates the first published version of the job.

SQL jobs

Use SQL for analytics and ETL transformations. There are two source variants:

• Saved query: reference an existing query by ID. The query text is resolved at run time, so edits to the query flow through to the next run of the job.
• Snapshot: paste inline SQL text into the job. You can optionally track the origin (e.g. a Git path) for drift detection.

A file variant pointing at SQL in workspace storage is reserved in the API but not supported yet.

Pick saved query when the SQL is shared across jobs; pick snapshot when the SQL belongs to exactly one job and shouldn't drift.

Python jobs

Use Python when you need the Python ecosystem, such as ML or third-party libraries. Two source variants:

• Python wheel: run a wheel by package name and entry point. Best for reusable code that's versioned in a package registry.
• Python file: run a single .py workspace file or S3 URI. Best for scripts and ad-hoc work.

Job files

Python files, wheel libraries, and requirements files can come from two places:

• Workspace file: select an existing file or upload one while editing the job. LakeSail stores it in the workspace S3 bucket for the compute profile's network. The job's selected team determines which files appear in the picker.
• S3 URI: enter the full URI of a file in the scripts/ location of the network workspace bucket. Using Upload places the file there automatically.

Changing the job's team or moving it to a compute profile on another network changes which workspace files are available. Select or upload the file again after either change.

File	Required extension	Upload limit
Python job	.py	10 MiB
Wheel library	.whl	100 MiB
Requirements file	.txt	1 MiB

Compute

Jobs run on the cluster's compute nodes (not the management nodes), sized through a compute profile, which is a named, reusable bundle of engine and sizing settings. Every job references one; pick an existing profile or create one inline. The profile sets:

• Execution mode: standalone (single Sail pod) or cluster (distributed driver + workers).
• Driver: instance type and disk for the coordinating pod.
• Worker (cluster mode): instance type, disk, and max nodes for the Sail workers.
• Libraries and environment: PyPI/wheel dependencies and Sail config to install before the run.

Size the profile for the job's actual working set. Because the compute lives in the profile, editing the profile changes the next run of every job that references it without redeploying the jobs. See Compute profiles for sizing and Limits & quotas for job limits.

Catalogs

Jobs inherit the catalogs and default catalog from their compute profile. In Catalogs, click Customize to add or remove catalogs or choose a different default for this job. Customizing replaces the profile's catalog set.

A catalog tells Sail where tables are stored. It does not copy the data or grant storage access. For S3-backed tables, the underlying data must be in the network workspace bucket. See Connect a catalog.

For SQL jobs that use a saved query, the query's catalogs also apply. The job's catalog settings take precedence when the same catalog is attached more than once.

A catalog must be shared with every team assigned to the job.

Sail environment variables

Add job-specific variables under Execution. Compute profile variables apply first; a job variable with the same name overrides the profile value. Values can be literals, managed secret references, or JSON templates with secret references.

Drafts and versioning

Every change to a live job flows through a draft, so you can iterate without disturbing scheduled runs.

1. Open the job and click Edit. Changes go into a draft. Drafts do not fire on schedule and are excluded from the main run history.
2. Test the draft by running it manually. Draft runs are clearly labeled.
3. When ready, publish the draft. It becomes the new live version, and the previous version moves to the version history.

The version history is searchable, and every run links to the job version it executed. Saved queries and compute profiles are resolved when work starts, so later edits to those resources can affect a retry of the same version.

Roll back

To revert to an earlier version:

1. Open the job's Versions tab.
2. Find the version you want and click Activate.

The activated version becomes the live one on the next run (scheduled or manual).

Teams and access

Every job is assigned to at least one team. The current job form selects one team.

• Members of an assigned team can act according to their team roles.
• Organization roles can also grant access.

Changing the Team field replaces the job's team assignment. It also changes which saved queries, catalogs, and workspace files are available in the form.

API reference

• Jobs: CreateJob, UpdateJob, drafts, versions, team assignment.
• Catalogs: required prerequisite for the job's data binding.