How Compute Runs and Scales
LakeSail separates cluster infrastructure from workload compute. Cluster infrastructure remains in place while it is deployed. Sail compute starts when a job run, session, or notebook starts.
Two layers of compute
- Management nodes run the cluster's system services. Cluster settings control their minimum, desired, and maximum counts.
- Workload compute runs Sail for jobs, sessions, and notebooks. Its size comes from a compute profile, and capacity is added on demand.
A compute profile stores configuration. Creating or editing one does not start or reserve compute.
From start to ready
- A workload starts: a job run, a session, or a notebook.
- LakeSail schedules a Sail runtime on the cluster.
- AWS capacity is added if none is ready.
- The work runs.
- Compute is reclaimed when the runtime stops.
When suitable capacity is not ready, AWS must launch a new instance before the Sail runtime can start. This is why the first workload on a cold cluster takes longer.
In cluster mode, LakeSail requests the driver node and all worker nodes at workload start. The initial worker node count is the profile's Max nodes. LakeSail makes this request once per workload. Sail starts worker pods when a client session starts. Karpenter can remove unused nodes after the workload's prewarm hold and the configured consolidation delay. Prewarming also has a deadline that includes scheduling and node launch, so late-starting capacity may be held for less time. This deadline does not limit Spark execution. A later client session can need new nodes.
Execution modes
- Standalone uses one Sail pod. Work still runs in parallel within that pod. This mode has less startup overhead and fits most interactive work and small-to-medium jobs.
- Cluster uses a driver and separate workers. Sail starts a worker pool for each Spark session.
Worker settings apply only in cluster mode.
How scaling works
Management and workload compute scale independently:
- Cluster settings size management nodes.
- Each Spark session requests the profile's Max nodes workers at startup.
- AWS capacity is added as needed and reclaimed automatically.
Spark sessions in the same workload share one driver. Each session has its own worker pool. Each worker runs on a separate node. Max nodes limits workers per client session. It does not limit the total worker count for the workload. With Max nodes set to 4, three active Spark sessions can use twelve worker nodes and one shared driver node.
SparkSession.builder.create() creates a separate Spark session and requests another worker pool. Workers use available nodes for the workload. LakeSail adds nodes when more capacity is needed. Available capacity and AWS quotas still apply.
SparkSession.builder.getOrCreate() reuses the active or default session in the same Python process. It creates a session if neither exists. Reuse the same spark variable when you want to use the same worker pool.
Sail can stop idle workers. It adds workers as needed, up to each session's limit.
Runtime lifetime
| Workload | Runtime starts | Runtime stops |
|---|---|---|
| Job | For each run | When the run finishes |
| Session | When the session starts | On close, after its idle timeout and close delay, or at maximum duration |
| Notebook | When the notebook starts | On stop, after its idle timeout and close delay, or at maximum duration |
Jobs receive fresh compute for each run. Sessions stay warm for interactive use. Stopping a notebook releases its runtime but preserves its contents.
What affects startup and usage
- Instance startup: launching new AWS capacity takes longer than using capacity that is ready.
- Execution mode: cluster mode requests worker nodes when the driver starts.
- Libraries: installing packages adds startup work.
- Instance size and runtime: LakeSail measures workload vCPU and memory over time.
- Max nodes: a larger count requests more worker nodes at workload start and raises each client session's worker limit.
Stop sessions and notebooks when they are no longer needed to release compute sooner. See Trial & Usage for metering and rates.