Continuous Jobs

A continuous job runs until you stop it. While the job is active, LakeSail keeps one run going and restarts your code whenever it exits. Use it for streaming and other code that should always be up.

For the job definition, see Defining Jobs. For runs in general, see Runs & Debugging.

Your code must checkpoint

LakeSail restarts a continuous job's code from the beginning whenever it exits, its node is drained, or a new version needs it. Nothing in memory survives a restart. Save progress to storage as you go, such as the last file or offset processed, and resume from it at start. Without this, a restart reprocesses or loses work.

Prerequisites

• A Python job. SQL jobs cannot be continuous.
• A compute profile on a cluster whose infrastructure is up to date. If it is not, the Continuous option says so and names the update to run.

Make a job continuous

1. On the job's create or edit form, under Trigger, select Continuous.
2. Save the job.

If the job is Active, its run starts when you save. Schedule, concurrency, timeout and Automatic restarts do not apply: a continuous job has no schedule, one run at a time, no timeout, and restarts without limit.

In the API, set the version's trigger to {"kind": "continuous"} and leave timeoutSeconds empty or 0.

How the run is kept going

• When your code exits, for any reason, the runner starts it again in the same run. The first few restarts are quick.
• When the code keeps exiting, the run shows Crash Looping and restarts it about once a minute. The run's status stays Running; in the API, crashLooping is true. Status Message says why: the code's last error or, for code that finishes without one, that a continuous job's code must keep running. The run also shows it when the runner itself keeps stopping, most often because the code ran out of memory; Status Message says so. It clears once the code stays up for a while. LakeSail sends the job_run.crash_loop.started notification when the loop starts, and job_run.crash_loop.recovered when it ends.
• When the code fails while the Sail engine does not answer, the runner waits for the engine before it restarts the code.
• When the run itself ends, for example because its cluster went away, LakeSail starts a new run. Runs that keep failing to start are retried less and less often.

A restart starts your code from the beginning and keeps nothing from memory. Save progress to storage, such as a Spark streaming checkpointLocation, and resume from it.

The run's page shows Restarts: how often the code restarted, and why it last did.

Stop and resume

• Pause the job, or use Pause Job on its run (CancelJobRun in the API). The run stops and no new run takes its place. Its compute is released.
• Resume the job to start a new run.
• Your code can stop the job itself: raise SailStopJob. The run ends as Failed with the reason stopped_by_job, and the job is paused. Fix what made the code stop, then resume the job.

if config_is_invalid():
    raise SailStopJob("the input topic does not exist")

SailStopJob is a builtin like SailRetryableError, so there is nothing to import. Any other exception restarts the code.

When LakeSail stops the runner, your code gets SailStopRequested and about a minute and a half to clean up. See Stopping cleanly.

You cannot delete a job while its continuous run is live. Pause it first.

Publish a new version

Save on a continuous job publishes your changes and applies them to the running job at once. Use Save as Draft to keep the running job unchanged, then publish the draft when you are ready, for example outside busy hours. Before an update reaches a running job, the console says what it will restart and asks you to confirm; if you decline, your changes stay a draft.

Publishing or activating a version applies it to the running job at once. The old run ends as Superseded, and a new run takes over. LakeSail restarts only what the change needs:

What changed	What restarts
The code or its arguments	Only your code, in the same pod, within about a minute
How the code is given: a Python file, a wheel or a notebook	The runner pod. The Sail engine keeps running
Sail settings, catalogs, the engine image, libraries, environment variables or secrets	The Sail engine as well as the runner, so its session state is lost
The compute profile, instance types or execution mode	Everything: the run starts again on new compute, which takes minutes
The trigger, to Manual or Scheduled	The running job stops for good; it runs again only when you run it or its schedule fires

A change of compute interrupts the job for several minutes, and any change that restarts the code loses what it holds in memory. The API applies a version without asking, so a script that calls it directly is responsible for not interrupting the job: send the request with dryRun=true first and read applyImpact in the response to see what applying the version would restart, or keep the change as a draft. Only a change of compute and a stop are certain; for the rest, applyImpact is the expected restart.

Releases

A run keeps the runner and Sail engine release it started on, even when you publish new code. A new LakeSail release never restarts your code by itself.

When a newer release is available, the run's page says so. Check the release works for your job, then use Restart on Current Release. Use Restart on a current run to start your code again without changing anything else. Both are RestartJobRun in the API, and releaseOutdated on the run says which one it does.

Retry and Run Job

A continuous job owns its run, so Retry Run and Run Job are not available for it. To start the code again, use Restart. To start a new run after the job was paused, resume it.

Restart is only for continuous jobs. Their runner keeps running between attempts of your code, so it can start the code again inside the same run. A scheduled or manual job's runner ends with your code, and how it ended decides the run's outcome, so there is nothing to restart in place: retry a run that ended, or cancel it and run the job again.

API reference

• Jobs: CreateJob, CreateJobVersion, PublishJobDraft, ActivateJobVersion (each with dryRun), UpdateJob.
• Job runs: CancelJobRun, RestartJobRun, DescribeJobRun.