Runs & debugging

A job run tracks one execution of a job. Run Job, Save and Run Now, and each scheduled tick create a run. Retry Run resets the latest eligible run and executes it again with the same run ID and job version.

For the job definition, see Defining jobs. For scheduling, see Scheduling.

Prerequisites

• A saved job that has been triggered at least once (manually or on schedule) so there's a run to inspect.

Run lifecycle

The status shown for a run tells you exactly where it is stuck.

Status	Meaning
Pending	Run accepted but held until it is released
Ready	Eligible for execution
Starting	Kubernetes pod being created
Creating Sail / Waiting for Sail	Sail (the query engine) is being deployed
Creating Runner	Runner pod starting
Running	Your code is executing
Succeeded / Failed / Cancelled / Timeout	Terminal states

Inspect a run

Click into a run to see:

• Timeline and status message: where the run is and why it failed.
• Output: SQL output and its location in the network workspace bucket. Results with columns are stored as Parquet; statements that return no columns do not create a result file.
• Job and version: the definition used for the run.
• Timing: dispatch, runner, completion, and duration fields when available.

Recovery actions

Retry a run

Retry Run resets and re-dispatches the existing run. Its run ID, job version, and parameters stay the same. LakeSail clears the previous execution state and output, then starts a fresh workload attempt.

The referenced saved query and current compute profile settings are resolved again when the retry starts. If either changed after the original attempt, the retry can run with those changes even though the job version stays the same.

Retry is available only when all of these are true:

• The run is the job's latest run.
• Its status is Failed, Timeout, or Cancelled.
• The job is active and the run still uses its current version.
• The cluster and billing state allow another execution.

To preserve the failed attempt as a separate run, use Run Job instead.

Hold and release a run

This flow is available through the API for orchestration. Create or retry a run with hold: true to leave it Pending. Call ReleaseJobRun when its dependencies are ready; release changes the run from Pending to Ready. It does not detach or reclaim compute.

Error handling & automatic retries (Python jobs)

Separately from Retry Run, Max Retries controls automatic pod attempts within the same job run. What your script does determines whether an attempt is retried.

By default, a failure in your code is not retried. If your script raises an exception or exits non-zero, the same file with the same arguments is likely to fail the same way, so the run fails immediately and reports the error. This avoids repeating the failure until the retry budget is gone and returning a generic "backoff limit exceeded" instead of the original error.

That default is a heuristic, not a guarantee. Code that reads from S3, calls an external service, or depends on anything over a network can fail transiently and would succeed on another attempt. Those cases are the exception, so you declare them explicitly.

What your script does	Outcome	Retried?
Returns normally, or sys.exit() / sys.exit(0)	Succeeded	Not applicable
Raises any exception (ValueError, NameError, …)	Failed, with your traceback	No
Exits non-zero (sys.exit(1), missing argparse argument)	Failed, exit code reported	No
Raises SailRetryableError	Failed	Yes
Exits with code 75	Failed	Yes

Requesting a retry for a transient failure

When you know a failure is temporary, such as a throttled S3 read, a rate-limited API, or a flaky upstream, catch it and re-raise it as SailRetryableError. It's provided as a builtin, like sail_spark, so there's nothing to import:

try:
    df = sail_spark.table("production.sales.orders")
except Exception as e:
    # Transient: worth another attempt on a fresh pod.
    raise SailRetryableError(f"could not read input: {e}") from e

df.write.mode("overwrite").parquet(args.output)

If raising isn't convenient, such as deep inside a wheel or when shelling out to another process, exit with code 75 instead. That's EX_TEMPFAIL from sysexits.h, the conventional "temporary failure, please retry":

import sys

if not upstream_is_ready():
    sys.exit(75)   # retry this run

Reserve both for genuinely transient conditions. A bug in your code retried three times is just the same bug three times, and it delays the error reaching you.

Reading the error

A failure from your own code reports the exception type, its message, and the traceback of your file in Status Message, so the failing line is visible on the run:

ValueError: bad input
  File "etl.py", line 12, in <module>

Output written to stdout, including df.show(), is not included in Status Message.

Python jobs manage their own output

SQL jobs store results in the network workspace bucket and report the location to LakeSail. A Python job writes wherever your script tells it to (commonly a --output argument), so the run won't show a results path. Check the location your script wrote to.

Debugging checklist

When a run misbehaves, work from the layer closest to the failure outward:

1. Run status is Failed

Open the run and read Status Message. Application errors such as SQL syntax problems, missing columns, or Python exceptions surface there with stack traces.

Common causes:

• Table or column doesn't exist in the catalog.
• Permission denied on S3 / Glue (check the IAM role's workload boundary).
• Python wheel missing a dependency.

2. Run stuck in Waiting for Sail

The Sail pod couldn't reach Ready within ~10 minutes. See the troubleshooting entry for the full diagnosis. Usually: cluster out of compute capacity, image pull failure, or EC2 service quota hit.

3. Run status is Timeout

The job exceeded its configured timeout. Either raise the timeout (if the workload legitimately grew) or investigate why it's slower than expected. A common cause is a catalog that returned a much larger result set than before.

4. Scheduled tick didn't fire

• Is the job Active (not paused)?
• Does the cron expression match what you think it does? Test it with a cron parser.
• Was the cluster healthy at the expected time? A Failed or Destroying cluster can prevent a scheduled run from starting.
• Does the missed-schedule policy match your expectations? latest only fires once on recovery.

5. Schedule fires but runs overlap unexpectedly

Revisit the concurrency policy. If ticks are piling up, skip or replace prevents the pileup. Avoid allow without a sensible maxConcurrentRuns.

6. "failed to create kubernetes client" at start

Infrastructure-layer failure, not job logic. See troubleshooting.

When to retry or create a run

• Retry Run: rerun the latest failed, timed-out, or cancelled run under the same run ID and version.
• Run Job: create a separate run from the job's current version.
• ReleaseJobRun: move an intentionally held API run from Pending to Ready.

Use Retry Run for a transient failure when preserving the same run identity matters. Use Run Job when you want the previous attempt to remain a separate record or the job has changed.

API reference

• Job runs: CreateJobRun, DescribeJobRun, RetryJobRun, ReleaseJobRun, DeleteJobRun.