Compute profiles

A compute profile is a saved answer to a question every workload asks: what machine should this run on, and how should it be set up? You pick the engine size, the cluster, and any libraries once, give it a name, and then point jobs, sessions, and notebooks at it instead of configuring each one by hand.

In the UI, profiles live under Compute → Profiles. In the API they are called workload configs.

Why profiles exist

Jobs, sessions, and notebooks all need the same thing: a cluster to run on and a description of how big the Sail engine should be. Rather than repeat that on every workload, a profile captures it once so you can:

• Standardize sizing. "small", "interactive", and "etl-large" become shared, named choices instead of per-workload guesses.
• Change compute without touching the workload. Edit the profile; the next run of every job that references it picks up the new sizing.
• Reuse settings. Jobs, sessions, and notebooks that can use the selected cluster can share the same profile.

Execution mode

Execution mode is the most important choice in a profile, because it changes the shape of the compute:

Mode	Shape	Use for
standalone (default)	A single Sail pod. Execution is still parallelized across threads inside that pod.	Most interactive work and small-to-medium jobs. Lower overhead, faster to start.
cluster	A driver pod plus separate worker pods that communicate over RPC.	Large datasets that need distributed execution across multiple nodes.

Worker settings only apply in cluster mode. In standalone mode there are no workers, so only the driver sizing matters.

Sizing compute

On a LakeSail-managed cluster you configure:

• Driver: the EC2 instance type (e.g. c8g.2xlarge) and EBS volume size, IOPS, and throughput for the coordinating pod.
• Worker (cluster mode only): instance type, EBS settings, and max nodes.

The cluster adds and removes worker nodes on demand, up to max nodes. The maximum is a ceiling, not reserved capacity. Workers currently use on-demand AWS capacity; Spot is not supported.

See How compute runs and scales for the full lifecycle.

External clusters

If the profile targets a cluster that LakeSail doesn't manage, execution mode and instance types aren't available. The workload runs on the cluster's existing resources, and the profile carries engine-level settings such as libraries and environment variables.

What a profile holds

The two sections above cover what most people set. A profile can hold:

• Name: how it shows up in the picker when you attach a profile to a workload.
• Cluster: the cluster the profile targets. Immutable after creation; a profile is tied to one cluster.
• Execution mode: standalone or cluster. See Execution mode.
• Max retries: the default retry count for jobs. 0 means no retries; -1 means retry indefinitely. A job can override this value.
• Compute: driver and worker sizing. See Sizing compute.
• Libraries: PyPI packages, inline requirements, or requirements and wheel files from an S3 URI. Jobs can also upload these files to workspace storage when customizing their libraries.
• Environment variables: values passed to the Sail engine. Values can be literals, managed secret references, or JSON templates containing secret references.
• Catalogs: catalogs available to workloads using the profile. Select Default beside a catalog so SQL can omit the catalog name. On a LakeSail-managed cluster, workloads use the network's default Glue catalog when you do not attach one.

The API also accepts settings that are not in the current profile form:

• EBS volume type and retention: choose gp3 or io2 and whether to delete the volume with its node. The profile form uses gp3 and deletes the volume.
• Endpoint limits: the number of active Spark Connect sessions that can share one provisioned runtime. The default is 1.
• Image URI: an override for the Sail container image. Leave it empty to use the platform default.

Profile access follows the teams linked to its cluster. A profile does not have a separate team assignment.

Reading the profile list

Each profile shows Readiness and Used by. Readiness reflects whether its cluster can run workloads. Used by summarizes job and notebook references and active runtimes.

Creating a profile

Create a profile two ways:

• From Compute → Profiles → Add Compute Profile.
• Inline with New profile while creating a job, session, or notebook, without leaving the form.

Either way, the profile is saved and can be reused by workloads with access to its cluster.

How workloads use a profile

• Jobs require a compute profile. Every job version references one, and the run inherits its cluster, sizing, and engine settings. Changing the profile changes the next run without a job redeploy. A job can override max retries, libraries, environment variables, and catalogs. See Defining jobs.
• Sessions take a profile when created. Through the API, the endpoint limit controls how many Spark Connect sessions can share one runtime. See Sessions.
• Notebooks reference a profile for the Sail pod behind the notebook. It's editable only while the notebook is Stopped. See Notebooks.

Editing and deleting

• Editing: you can change the name, compute, libraries, environment variables, retries, and catalogs when no runtime is active. Stop notebooks, close sessions, let job runs finish, and wait for cleanup to complete. Changes apply to future workloads.
• Immutable: cluster and workload type. To use a different cluster, create a new profile.
• Deleting: jobs and notebooks that reference a profile prevent deletion. Move stopped notebooks to another profile and delete referring jobs before deleting the profile. An active runtime prevents deletion only while it depends on the profile's catalogs, default catalog, or secrets.

API reference

• API Reference: ListWorkloadConfigs, CreateWorkloadConfig, DescribeWorkloadConfig, UpdateWorkloadConfig, DeleteWorkloadConfig, and GetWorkloadConfigReferences. Workload configs are the API name for compute profiles.
• Jobs: job versions reference a workload config for their compute.
• Clusters: the cluster a profile targets.