Compute Profiles

A compute profile is a saved answer to a question every workload asks: what machine should this run on, and how should it be set up? You pick the engine size, the cluster, and any libraries once, give it a name, and then point jobs, sessions, and notebooks at it instead of configuring each one by hand.

In the UI, profiles live under Compute → Profiles. In the API they are called workload configs.

Why profiles exist

Jobs, sessions, and notebooks all need the same thing: a cluster to run on and a description of how big the Sail engine should be. Rather than repeat that on every workload, a profile captures it once so you can:

• Standardize sizing. "small", "interactive", and "etl-large" become shared, named choices instead of per-workload guesses.
• Change compute without touching the workload. Edit the profile; the next run of every job that references it picks up the new sizing.
• Reuse settings. Jobs, sessions, and notebooks that can use the selected cluster can share the same profile.

Execution mode

Execution mode is the most important choice in a profile, because it changes the shape of the compute:

Mode	Shape	Use for
standalone (default)	A single Sail pod. Execution is still parallelized across threads inside that pod.	Most interactive work and small-to-medium jobs. Lower overhead, faster to start.
cluster	A driver pod plus separate worker pods that communicate over RPC.	Large datasets that need distributed execution across multiple nodes.

Worker settings only apply in cluster mode. In standalone mode there are no workers, so only the driver sizing matters.

Sizing compute

On a LakeSail-managed cluster you configure:

• Driver: the EC2 instance type (e.g. c8g.2xlarge) and EBS volume size, IOPS, and throughput for the coordinating pod.
• Worker (cluster mode only): instance type, EBS settings, and max nodes.

In cluster mode, max nodes limits workers per client session. It does not limit the total worker count for a workload. LakeSail requests that many worker nodes once when the workload starts. Sail starts one worker pod per node when a Spark session starts. Karpenter can remove unused nodes after the workload's prewarm hold and the configured consolidation delay. Prewarming also has a deadline that includes scheduling and node launch, so late-starting capacity may be held for less time. This deadline does not limit Spark execution.

Spark sessions in one workload share the driver. Each session has its own worker pool. With max nodes set to 4, three active Spark sessions can use twelve worker nodes. The driver node is separate.

Workers use available nodes for the workload. LakeSail adds nodes when more capacity is needed. Available capacity and AWS quotas still apply.

Workers use on-demand AWS capacity. Spot is not supported. Karpenter selects each node's availability zone. Worker pods prefer the driver's zone, but workers can run in other zones. Prewarmed nodes prefer the driver's first observed zone, with fallback to any zone if it lacks capacity or is unknown. This preference is set once; replacing the driver in another zone does not move existing prewarmed nodes.

See How Compute Runs and Scales for the full lifecycle.

What a profile holds

The two sections above cover what most people set. A profile can hold:

• Name: how it shows up in the picker when you attach a profile to a workload.
• Cluster: the cluster the profile targets. Immutable after creation; a profile is tied to one cluster.
• Execution mode: standalone or cluster. See Execution mode.
• Automatic restarts: how often a failed runner restarts within the same run, for jobs that do not set their own. 0 disables restarts; -1 restarts without limit. See Automatic restarts.
• Compute: driver and worker sizing. See Sizing compute.
• Libraries: PyPI packages, inline requirements, or requirements and wheel files from an S3 URI. Jobs can also upload these files to workspace storage when customizing their libraries. Jobs and notebooks run Python 3.14 with the PySpark 4.1 Spark Connect client, pysail and boto3 already installed. In a job, your libraries must work with those versions, and a library that brings its own copy of one, such as pyspark, fails the run at setup.
• Environment variables: values passed to the Sail engine and, for notebooks, to the notebook kernel as well. Values can be literals, managed secret references, or JSON templates containing secret references.
• Catalogs: catalogs available to workloads using the profile. Select Default beside a catalog so SQL can omit the catalog name. On a LakeSail-managed cluster, workloads use the network's default Glue catalog when you do not attach one.
• External data: S3 buckets outside the workspace bucket that the profile's workloads can use. See Connect External S3 Data.

The API also accepts settings that are not in the current profile form:

• EBS volume type and retention: choose gp3 or io2 and whether to delete the volume with its node. The profile form uses gp3 and deletes the volume.
• Endpoint limits: the number of active Spark Connect sessions per endpoint that can share one provisioned runtime. The default is 1. This setting does not limit Spark sessions that your code creates within that runtime.
• Image URI: an override for the Sail container image. Leave it empty to use the platform default.

Profile access follows the teams granted access to its cluster. A profile does not have a separate team assignment. Creating one needs a cluster that has at least one team, reached either through an organization role that grants profile creation or through membership of one of those teams.

Reading the profile list

Each profile shows Readiness and Used by. Readiness reflects whether its cluster can run workloads. Used by summarizes job and notebook references and active runtimes.

Creating a profile

Create a profile two ways:

• From Compute → Profiles → Add Compute Profile.
• Inline with New profile while creating a job, session, or notebook, without leaving the form.

Either way, the profile is saved and can be reused by workloads with access to its cluster.

How workloads use a profile

• Jobs require a compute profile. Every job version references one, and the run inherits its cluster, sizing, and engine settings. Changing the profile changes the next run without a job redeploy. A job can override max retries, libraries, environment variables, and catalogs. See Defining Jobs.
• Sessions take a profile when created. Through the API, the endpoint limit controls how many Spark Connect sessions can share one runtime. See Sessions.
• Notebooks reference a profile for the Sail pod behind the notebook. It's editable only while the notebook is Stopped. See Notebooks.

Editing and deleting

• Editing: you can change the name, compute, libraries, environment variables, automatic restarts, and catalogs when no runtime is active. Stop notebooks, close sessions, let job runs finish, and wait for cleanup to complete. Changes apply to future workloads.
• Immutable: cluster and workload type. To use a different cluster, create a new profile.
• Deleting: jobs and notebooks that reference a profile prevent deletion. Move stopped notebooks to another profile and delete referring jobs before deleting the profile. An active runtime prevents deletion only while it depends on the profile's catalogs, default catalog, or secrets.

API reference

• API Reference: ListWorkloadConfigs, CreateWorkloadConfig, DescribeWorkloadConfig, UpdateWorkloadConfig, DeleteWorkloadConfig, and GetWorkloadConfigReferences. Workload configs are the API name for compute profiles.
• Jobs: job versions reference a workload config for their compute.
• Clusters: the cluster a profile targets.
• Data targets: external data targets and the grants that give compute profiles access to them.