Concepts

LakeSail runs batch and interactive data workloads inside your own cloud account, against data you already have. To do this safely and predictably, the system is split into several layers. This page walks through those layers in the order you'll encounter them. By the end you should be able to point at any object in the UI and know where it fits.

How the pieces fit together

Organization ─── Members, Teams, Roles        ← who can do what
      │
      ├── Cloud Account    ← AWS connection (assumed role)
      │        └── Network     ← a VPC inside that account
      │                └── Cluster   ← Kubernetes compute inside that VPC
      │
      ├── Catalog          ← connection to table metadata
      │
      ├── Compute Profile  ← cluster, sizing, libraries, and catalogs
      │
      ├── Query            ← saved SQL used by Jobs
      │
      └── Workloads
              ├── Jobs ───────── Job Runs
              ├── Sessions ───── interactive Spark Connect runtime
              └── Notebooks ──── hosted editor and runtime

Read top-to-bottom: an organization scopes membership and resources; infrastructure nests account, then network, then cluster; catalogs describe tables; compute profiles connect workloads to infrastructure and data; and jobs, sessions, and notebooks are the ways you run work. Saved queries can be referenced by SQL jobs.

Many objects have a stored definition and a separate lifecycle, such as Pending, Active, or Destroyed. For infrastructure, status records the latest lifecycle operation, while readiness shows whether workloads can run.

You and your organization

When you create a workspace, you create an organization that contains its members, settings, and resources.

Inside the organization, identity has four moving parts:

• Users are the people. A user can belong to multiple organizations (the same email works across them), but they act in one at a time.
• Members are the organization-scoped link between a user and an org.
• Teams group members together, typically by function (data-eng, analysts) or by project. A member can belong to any number of teams.
• Roles grant permissions. Organization roles apply across the organization; team roles apply within a team.

You'll mostly interact with this layer through two questions: "who's on my team?" and "what can they do?" For a first-time solo user it's invisible: you're the sole member with full access. It becomes load-bearing the moment a second person shows up, because that's when "who can spin up a cluster in production?" stops being rhetorical.

Each part has a dedicated page: Members, Teams, and Roles & permissions. To add the first teammate, see Invite teammates.

Where your compute lives

LakeSail does not run your compute on its own hardware. It runs inside your AWS account, which is why the infrastructure layer has three nested objects instead of one. Each level has a different lifecycle and a different blast radius. Separating them means rotating credentials, resizing compute, and tearing down environments can happen independently.

Cloud account

A cloud account is a trust relationship between LakeSail and an AWS account you own. You deploy a CloudFormation stack that creates an IAM role with a scoped trust policy. LakeSail records the role ARN and verifies the connection automatically. The manual setup path lets you paste the ARN instead. LakeSail assumes the role on demand and uses temporary credentials issued by AWS. For the full trust model, the permissions it grants, and what LakeSail cannot do, see Security & IAM.

Cloud accounts move through a short lifecycle: Pending while you set up the account, Verifying while LakeSail checks the connection, then Active. If something changes, such as deleting the role or revoking its trust, the account moves to Failed or Disconnected, and any dependent networks and clusters are effectively orphaned until you repair it. One cloud account can host many networks.

For exactly what the IAM role grants and the full trust model, see Security & IAM; for the connection walkthrough, see Connect an AWS account.

Network

A network is a VPC that LakeSail provisions inside a cloud account. You pick the region and the IPv4 CIDR range (e.g. 10.0.0.0/16). LakeSail creates the VPC, subnets across availability zones, security groups, and the rest of the plumbing. Clusters in the network share its VPC boundary and workspace bucket.

The lifecycle mirrors infrastructure reality: Pending → Provisioning → Deployed on the way up, Destroying → Destroyed on the way down. Provisioning is visible because it takes minutes, not seconds. Putting networks in their own layer (rather than tying a VPC directly to a cluster) means you can run multiple clusters (for example, dev and prod, or different sizes for different teams) without re-creating networking, and you can delete a cluster without tearing down the VPC.

One cloud account can host many networks. A network belongs to exactly one cloud account and one region. The Quickstart covers creating one.

Cluster

A cluster is a Kubernetes cluster that LakeSail provisions inside a network. It is where job runs, sessions, and notebooks get scheduled. A cluster has two kinds of nodes, and the distinction matters:

• Management (system) nodes run LakeSail's cluster services and workload autoscaler. Choose a cluster tier to set their sizing and resilience, or use advanced configuration to size them manually. Cluster tiers do not limit workload capacity.
• Compute nodes run your workloads. They are chosen per-workload through a compute profile, so you don't oversize a cluster to handle a rare big workload. The cluster scales compute on demand.

Cluster lifecycle: Pending → Provisioning → Deployed, then Updating while you resize, and Destroying → Destroyed on teardown. A Failed state means CloudFormation or the Kubernetes bootstrap did not succeed; the provisioning progress bar shows the stage it stopped at. Deployed records the last infrastructure operation; check Readiness to tell whether work can run now.

A cluster has an access policy: either private-only (reachable only from inside the VPC) or public with a CIDR allowlist. You can change this later without rebuilding the cluster. To create, size, or destroy one, see Set up a cluster.

Where your data lives

A catalog tells Sail how to find tables. It can connect to AWS Glue, Iceberg REST, Unity Catalog, OneLake, Hive Metastore, or an in-memory catalog. LakeSail does not copy data when you connect a catalog. For S3-backed tables on LakeSail-managed compute, the underlying data must be in the network workspace bucket.

Each network includes a default AWS Glue catalog. It starts empty and is available to workloads automatically unless you choose different catalog settings.

Each cloud network also has a workspace bucket in your AWS account. LakeSail uses it for managed catalog data, job files, notebook contents, and SQL job results. Uploading a job file places it in the workspace bucket for the job's network and team. See Workspace bucket for its layout and lifecycle.

If existing S3 table data is outside the workspace bucket, see Make S3 table data available to LakeSail.

This is why catalogs are separate from clusters: the same data can be read by multiple clusters in a network, and the same cluster can read from multiple catalogs. The pairing is picked per-workload.

Like networks and clusters, a provisioned catalog has a lifecycle (Pending → Provisioning → Deployed → Destroying → Destroyed) because real AWS resources are being created. A catalog that points at something you already manage skips most of that. It is a configuration in LakeSail. To connect one, see Connect a catalog.

What actually runs

Every workload runs on a cluster with compute described by a compute profile: a named, reusable bundle of engine and sizing settings (execution mode, driver/worker instance types, libraries) that you point a workload at instead of sizing it inline. The same profile can back many workloads, and editing it changes their next run.

See How compute runs and scales for the runtime lifecycle.

Workloads themselves come in three shapes, matched to how they're used:

Jobs and job runs

A job is a reusable, versioned SQL or Python workload. A job has:

• Versions: edits go into a draft. Publishing the draft creates the new live version and preserves the previous job definition.
• A status of Active (runs on its schedule) or Paused (defined but won't fire). Pausing is cheap and reversible, and it is the first thing to reach for when something misbehaves.
• A team assignment, which grants access through the team's member roles.

Running a job creates a job run with its own status, version, timing, and result. Retrying resets the latest eligible run under the same ID and version. Job runs move through Pending → Ready → Starting → Creating Sail → Waiting for Sail → Creating Runner → Running → Succeeded (or Failed, Cancelled, Timeout), so the status shows where a run stopped.

Jobs fit batch and scheduled work where reproducibility and history matter more than latency. See Defining jobs, Scheduling, and Runs & debugging.

Queries

A query is a reusable LakeSail SQL definition: saved, named, created by a member, and assigned to a team. It does not execute on its own. A SQL job can reference the query and resolve its text when the run is dispatched. See Queries.

Sessions

A session is a live Spark Connect runtime. A compatible client connects to its endpoint and submits SQL or DataFrame operations directly. Sessions do not run LakeSail query resources.

Sessions move through Pending and Active, may become Idle, and end in Closed. Failed means the session could not start or continue. Idle sessions close after a short delay and release their compute. The session owner can issue short-lived tokens for external Spark Connect clients.

Sessions fit interactive work that needs low latency and shared state between requests. See Sessions.

Notebooks

A notebook is a hosted, Marimo-based Python notebook backed by its own Sail session. Where a session is a connection you attach an external client to, a notebook is both the editor and the runtime, hosted in the browser. It's the interactive surface for writing PySpark against your catalogs without wiring up a local client. See Notebooks.

Jobs, sessions, and notebooks in one sentence

Jobs are for pipelines: write once, run many times on a schedule, with saved queries available as SQL sources. Sessions give external Spark Connect clients an interactive runtime. Notebooks provide that interactive loop in a hosted Python editor. All three ultimately land on the same clusters, reading from the same catalogs, governed by the same roles.

Putting it together

A complete mental model of a LakeSail workspace:

An organization of members and teams connects one or more cloud accounts, provisions networks and clusters inside them, registers catalogs, defines compute profiles and queries, and runs jobs, sessions, and notebooks on that compute.

A useful diagnostic, when something in the UI is not doing what you expect: ask which layer you are looking at. A failing job run is a workload problem. A cluster that won't provision is an infrastructure problem, usually one layer down, in the network or cloud account. A permission denial is an identity problem. The layers are the seams the rest of the product is built along.