Runs & Debugging

A job run tracks one execution of a job. Run Job, Save and Run Now, and each scheduled tick create a run. Retry Run resets the latest eligible run and executes it again with the same run ID and job version.

A run started by the schedule shows Logical Time: the scheduled time it covers, which can be earlier than when it started. See Which time a run is for.

For the job definition, see Defining Jobs. For scheduling, see Scheduling.

Prerequisites

• A saved job that has been triggered at least once (manually or on schedule) so there's a run to inspect.

Run lifecycle

The status shown for a run tells you exactly where it is stuck.

Status	Meaning
Pending	Run accepted but held until it is released
Ready	Eligible for execution
Starting	The job worker has claimed the run and is creating its Kubernetes resources
Creating Sail / Waiting for Sail	The worker deploys Sail (the query engine) and waits for it
Creating Runner	The worker creates the runner for your code
Running	Your code is executing
Succeeded / Failed / Cancelled / Timeout	Terminal; the run stays here unless it is retried. Failed, Cancelled and Timeout can be retried
Unknown	The platform could not determine the run's state

Inspect a run

Click into a run to see:

• Timeline and status message: where the run is and why it failed.
• Output: SQL output and its location in the network workspace bucket. Results with columns are stored as Parquet; statements that return no columns do not create a result file.
• Job and version: the definition used for the run.
• Parameters: the values the run used, defaults included. See Parameters.
• Timing: dispatch, runner, completion, and duration fields when available.

Logs

Each run has two logs, shown on the run's detail page under Logs:

• Runner: the process that runs your Python code and submits its work to Sail. Output from print() and logging appears here. Start here for setup problems, such as a file or library that could not be fetched.
• Engine: the Sail driver that executes it. Engine-side failures such as out-of-memory kills and panics appear here.

Logs stream live while the run is in progress and stay readable after it finishes, until the run's resources are cleaned up. After cleanup they are gone; the run's status, timing, and error message remain. If a log is not available yet, the pod has not started; wait and retry.

Through the API, IssueJobRunLogStream returns a short-lived URL and token; a streaming GET on it follows the log until the container exits.

Recovery actions

Retry a run

Retry Run resets and re-dispatches the existing run. Its run ID, job version, and parameters stay the same. LakeSail clears the previous execution state and output, then starts a fresh workload attempt. This is separate from automatic restarts, which restart a failed runner within a run that has not ended yet.

The referenced saved query and current compute profile settings are resolved again when the retry starts. If either changed after the original attempt, the retry can run with those changes even though the job version stays the same.

Retry is available only when all of these are true:

• The run is the job's latest run.
• Its status is Failed, Timeout, or Cancelled.
• The job is active and the run still uses its current version.
• The cluster and billing state allow another execution.

To preserve the failed attempt as a separate run, use Run Job instead.

Hold and release a run

This flow is available through the API for orchestration. Create or retry a run with hold: true to leave it Pending. Call ReleaseJobRun when its dependencies are ready; release changes the run from Pending to Ready. It does not detach or reclaim compute.

Create a run safely from an orchestrator

A request to CreateJobRun can time out after the run was created. Retrying it blindly then starts the job twice. To make the retry safe, send an Idempotency-Key header with a value you choose, such as the orchestrator's task instance ID and try number.

• The first request with a key creates the run and returns 201.
• A later request from the same member with the same key for the same job returns that run with 200. It creates nothing, even if the job was paused since.
• A request from another member with a used key is refused with 409, without details of the run.
• A request that differs from the first one, for example in its parameters, is refused with 409. The message names the run the key created and the fields that differ.

A key belongs to its run for as long as the run exists, and the run's detail shows it. Keys are per job: the same key on another job creates a separate run.

Cancel a run

Cancel Run, on a run's page and in the runs list, stops a run that has not finished. It is CancelJobRun in the API. The run moves to Cancelled, its runner is stopped whatever the resource cleanup mode says, and the rest of its resources follow that mode. The job_run.status.cancelled notification fires.

Cancel is idempotent: a run that already ended is returned unchanged, so an orchestrator can call it on every teardown path, for example from an Airflow on_kill, without checking the status first. A cancelled run can be retried.

The scheduler uses the same path for the replace concurrency policy.

Stopping a run is not instant: its code keeps executing for up to about half a minute while Kubernetes shuts the pod down. Cancelled therefore means LakeSail stopped the run and released its compute, not that nothing happened. Writes your code made before it was stopped stay in place. If the code finished inside that window, the run stays Cancelled, its runner timing and output location are still recorded, and Status Message says so. The same holds for Timeout.

Release a run's kept resources

With the resource cleanup mode retain, or on_success when the run did not succeed, a run keeps its Sail server and namespace after it ends, so you can inspect them, for example with kubectl. They are retained until the end of the run's retention, 24 hours after it ended by default, and then cleaned up. Its Sail server keeps running until then, and that compute is billed. Resource Cleanup on the run's page shows when.

Change retention, on the run's page, moves that time earlier or later, for example to Monday for a run that failed on Friday. It is SetJobRunRetention in the API. Retrying the run starts from the default again.

Clean up now, on the run's page, ends the retention: the Sail server and namespace are removed now, logs included. The run stays on record. It is EndJobRunRetention in the API. Runs cannot be deleted. Once the retention ends, by either route, the page shows the resources as being cleaned up, and the retention can no longer be changed.

Timeout

A job's timeout applies to each run from the moment it reaches Running. That is when its runner is created, so starting the runner pod and installing libraries count towards it, not only your code: leave room for that when you pick a value. It is the timeoutSeconds field on the job, set through the API; a value of 0, the default, means no timeout. A run past it is stopped and ends in Timeout, with the job_run.status.timeout notification. As with cancel, the runner is stopped in every cleanup mode.

Error handling & automatic retries (Python jobs)

Separately from Retry Run, Max Retries controls automatic pod attempts within the same job run. What your script does determines whether an attempt is retried.

By default, a failure in your code is not retried. If your script raises an exception or exits non-zero, the same file with the same arguments is likely to fail the same way, so the run fails immediately and reports the error. This avoids repeating the failure until the retry budget is gone and returning a generic "backoff limit exceeded" instead of the original error.

That default is a heuristic, not a guarantee. Code that reads from S3, calls an external service, or depends on anything over a network can fail transiently and would succeed on another attempt. Those cases are the exception, so you declare them explicitly.

What your script does	Outcome	Retried?
Returns normally, or sys.exit() / sys.exit(0)	Succeeded	Not applicable
Raises SailRunSucceeded	Succeeded	Not applicable
Raises any exception (ValueError, NameError, …)	Failed, with your traceback	No
Exits non-zero (sys.exit(1), missing argparse argument)	Failed, exit code reported	No
Raises SailRetryableError	Failed	Yes
Exits with code 75	Failed	Yes

Requesting a retry for a transient failure

Use SailRetryableError for a temporary failure (e.g., a throttled S3 read, a rate-limited API, a flaky upstream, etc.). LakeSail provides this builtin in file and wheel jobs. The following example uses a Python file:

from pyspark.sql import SparkSession

spark = SparkSession.builder.getOrCreate()
try:
    df = spark.table("production.sales.orders")
except Exception as e:
    # Transient: worth another attempt on a fresh pod.
    raise SailRetryableError(f"could not read input: {e}") from e

df.write.mode("overwrite").parquet(args.output)

If raising isn't convenient, such as deep inside a wheel or when shelling out to another process, exit with code 75 instead. That's EX_TEMPFAIL from sysexits.h, the conventional "temporary failure, please retry":

import sys

if not upstream_is_ready():
    sys.exit(75)   # retry this run

Reserve both for genuinely transient conditions. A bug in your code retried three times is just the same bug three times, and it delays the error reaching you.

A Python file that does not exist, or a wheel entry point the runner cannot find, fails once as well, with config_error and the file or entry point named in Status Message. A download that fails for a moment, such as a throttled S3 read, is retried.

Stopping cleanly

When LakeSail stops a runner, your code gets a moment to clean up. That happens when the run is cancelled or passes its timeout, and when the runner is evicted or its node is drained. LakeSail raises SailStopRequested wherever your code is running. It's a builtin like SailRetryableError, so there's nothing to import. Your finally blocks and with statements run as it unwinds:

try:
    write_partitions()
except SailStopRequested:
    save_checkpoint()   # keep it short: the runner is killed when the grace period ends
    raise

Or, when the checkpoint means the run needs no restart:

try:
    write_partitions()
except SailStopRequested:
    save_checkpoint()
    raise SailRunSucceeded("checkpoint saved")

• Your code has about 30 seconds from the stop until the runner is killed. A cleanup that takes longer is cut off.
• Let the stop propagate. SailStopRequested isn't an Exception, so a broad except Exception: doesn't catch it. The run counts as stopped however your code ends after the stop, even if it returns normally or exits with 0, because the stop may have cut its work short. Code that carries on is killed when the grace period ends.
• If your code has finished its work, or saved enough that the run needs no restart, raise SailRunSucceeded instead. It's a builtin too, and ends the run as Succeeded after an eviction or a drain. A cancel or a timeout still ends the run as Cancelled or Timeout: LakeSail records that before it stops the runner.
• After an eviction or a drain, the runner is restarted and your code runs again from the start. A cancel or a timeout ends the run.
• Running out of memory gives no warning: the kernel kills the runner at once, and no cleanup runs.

Reading the error

A failure from your own code reports the exception type, its message, and the traceback of your file in Status Message, so the failing line is visible on the run:

ValueError: bad input
  File "etl.py", line 12, in <module>

Output written to stdout, including df.show(), is not included in Status Message.

Python jobs manage their own output

SQL jobs store results in the network workspace bucket under clusters/ and report the location to LakeSail. Results stay there after the run; delete them in S3 when you no longer need them. A Python job writes wherever your script tells it to (commonly a --output argument), so the run won't show a results path. Check the location your script wrote to.

Worker memory failures

LakeSail checks worker pods to identify memory failures. In a Python job, spark.stop() can remove these pods before LakeSail receives the run's outcome. If the pods are gone, a worker memory failure can appear as user_error instead of out_of_memory. Check the run's logs for the original error.

Why a run ended

A run that ended carries three fields in the API and in webhook payloads, so an orchestrator can decide what to do without parsing Status Message:

• terminationReason: the kind of ending, from the table below.
• exitCode: the exit status of a Python job's code: 0 on success, the value passed to sys.exit, 1 for an uncaught exception, 75 for SailRetryableError, or 137 when the runner was killed for memory. When the Sail engine is the one killed, the code's own exit status stays. A SQL job has none of its own, so it has one only when its runner died without reporting, such as a crash in a native library or a kill, and then it is the runner's. A run that failed before its code started has none.
• attempt: 1, plus one for each Retry Run.

The API also returns restarts: how often the runner was restarted automatically in this attempt, and why it last was.

terminationReason	Meaning	What to do
succeeded	The job's code finished.	Nothing.
user_error	The job's code failed: an exception, a non-zero exit, or a SQL error.	Fix the code or its input. A retry fails the same way.
config_error	The runner could not start with the job's settings, for example libraries that will not install, or a script or wheel entry point that cannot be found.	Fix the job's configuration.
timeout	The run passed its timeout.	Raise the timeout or make the job faster.
cancelled	Someone cancelled the run.	Nothing, or retry it.
out_of_memory	The runner, the Sail driver or a Sail worker was killed or evicted for using too much memory. The run's message says which, and which instance type to raise.	Use a compute profile with a larger instance type, or process less at a time.
unschedulable	The cluster had no room for the runner.	Retry later, or free or add capacity.
infra_error	The platform or the cluster failed, not the job.	Retry the run.
unknown	The runner ended without saying why, for example after its restarts ran out.	Read the logs, then retry.

When a run misbehaves

Read the run's Status Message and its logs first; application errors such as SQL syntax problems, missing columns, and Python exceptions surface there. For runs that fail to start, hang, time out, or fire at the wrong time, see the job-run entries in Troubleshooting.

When to retry or create a run

• Retry Run: rerun the latest failed, timed-out, or cancelled run under the same run ID and version.
• Run Job: create a separate run from the job's current version.
• ReleaseJobRun: move an intentionally held API run from Pending to Ready.
• CancelJobRun: stop a run that has not finished.

Use Retry Run for a transient failure when preserving the same run identity matters. Use Run Job when you want the previous attempt to remain a separate record or the job has changed.

API reference

• Job runs: CreateJobRun, DescribeJobRun, RetryJobRun, ReleaseJobRun, CancelJobRun, SetJobRunRetention, EndJobRunRetention.