How Compute Runs and Scales
LakeSail separates cluster infrastructure from workload compute. Cluster infrastructure remains in place while it is deployed. Sail compute starts when a job run, session, or notebook starts.
Two layers of compute
- Management nodes run the cluster's system services. Cluster settings control their minimum, desired, and maximum counts.
- Workload compute runs Sail for jobs, sessions, and notebooks. Its size comes from a compute profile, and capacity is added on demand.
A compute profile stores configuration. Creating or editing one does not start or reserve compute.
From start to ready
- A workload starts: a job run, a session, or a notebook.
- LakeSail schedules a Sail runtime on the cluster.
- AWS capacity is added if none is ready.
- The work runs.
- Compute is reclaimed when the runtime stops.
When suitable capacity is not ready, AWS must launch a new instance before the Sail runtime can start. This is why the first workload on a cold cluster takes longer.
Execution modes
- Standalone uses one Sail pod. Work still runs in parallel within that pod. This mode has less startup overhead and fits most interactive work and small-to-medium jobs.
- Cluster uses a driver and separate workers. Sail adds workers for distributed execution, up to the maximum set in the compute profile.
Worker settings apply only in cluster mode.
How scaling works
Management and workload compute scale independently:
- Cluster settings size management nodes.
- Sail scales workers up to the profile's Max nodes. This is a ceiling, not reserved capacity.
- AWS capacity is added as needed and reclaimed automatically.
Runtime lifetime
| Workload | Runtime starts | Runtime stops |
|---|---|---|
| Job | For each run | When the run finishes |
| Session | When the session starts | On close, after its idle timeout and close delay, or at maximum duration |
| Notebook | When the notebook starts | On stop, after its idle timeout and close delay, or at maximum duration |
Jobs receive fresh compute for each run. Sessions stay warm for interactive use. Stopping a notebook releases its runtime but preserves its contents.
What affects startup and usage
- Instance startup: launching new AWS capacity takes longer than using capacity that is ready.
- Execution mode: cluster mode may start worker capacity in addition to the driver.
- Libraries: installing packages adds startup work.
- Instance size and runtime: LakeSail measures workload vCPU and memory over time.
- Maximum workers: raising the maximum increases the available ceiling but does not create workers by itself.
Stop sessions and notebooks when they are no longer needed to release compute sooner. See Trial & Usage for metering and rates.