Clusters

A cluster is a Kubernetes cluster LakeSail provisions inside your network. It is where jobs, sessions, and notebooks run. This page covers how to create, size, reconfigure, and destroy a cluster.

For a full first-time walkthrough (cloud account → network → cluster), start with the Quickstart. This page is the standalone reference.

Prerequisites

• A connected cloud account with Active status.
• A network with Deployed status, in the region you want the cluster.

Create a cluster

1. Open Compute → Clusters and click Create Cluster.
2. Fill in:
   • Network: the network the cluster lives in. The cluster inherits its cloud account and region from the network.
   • Cluster Name: a human-readable identifier. Cannot be changed after creation.
   • Cluster tier: choose a tier based on the guidance in the picker. Its details show the management instance, node range, disk, and estimated baseline AWS cost.
   • Allowed IPs List (optional): see Change what can reach the cluster endpoint below. Leave empty for private-only access.
   • Collect container metrics (optional): adds per-pod and per-container metrics in AWS. It is off by default and adds AWS charges that rise with load. Container logs are always collected.
3. Click Create Cluster.

The cluster moves through Pending → Provisioning → Deployed. Provisioning typically takes 15 to 25 minutes while Karpenter, networking, the Sail control plane, and add-ons bootstrap, and the status may remain on Provisioning for an extended period before completing.

Cluster tiers

A tier sizes the cluster's management nodes: the always-on machines that start and stop workloads, carry notebook and Spark Connect traffic, and connect the cluster to LakeSail. Workload compute is sized separately, per compute profile, so a tier does not limit how many workloads run or how fast.

• Budget: one burstable machine. Dev and proof-of-concept use.
• Cost Effective: two burstable machines; survives losing one.
• Performance: two dedicated machines. The default for production.
• High Availability: three dedicated machines across availability zones; survives a zone outage.

The picker shows each tier's sizing and an estimated baseline AWS cost for the region. Burstable machines can add AWS surplus-credit charges under sustained load.

Use advanced configuration

The tiers are already sized for their stated use. Turn on Advanced configuration only when you need to set these management-node values yourself:

• Management Node Size
• Min / Desired / Max Management Nodes
• Disk Size (GB)

The form warns you when the selected instance cannot fit the system pods, capacity is too low, or the node counts are out of order. See Troubleshooting for the full list.

Resize an existing cluster

Most properties can be changed in place. The cluster moves to Updating and stays available for existing workloads where possible.

You can edit:

• Management node size (instance type)
• Min / Desired / Max management nodes
• Disk size (GB)
• Allowed IPs list
• Container metrics collection

You cannot change (these require creating a new cluster):

• The network the cluster lives in
• The cluster name
• The region (set by the network)

To resize:

1. Open Compute → Clusters and click into the one you want to change.
2. Click Edit.
3. Adjust any of the editable fields. The same validation warnings as create apply.
4. Click Update Cluster.

The cluster moves Deployed → Updating → Deployed. Changing the management instance family replaces the management node group; the form warns you before you save.

A larger management node also gives the session proxy and the node autoscaler more memory to work with. Only their limits grow; the memory they reserve stays the same.

Change what can reach the cluster endpoint

The Allowed IPs list controls what can reach the cluster endpoint from outside the VPC. It is a network control, not a permission: to choose who may run work on the cluster, see Control which teams can use a cluster.

• Empty (default): private-only. Only traffic from inside the VPC can reach the endpoint.
• Comma-separated CIDRs, e.g. 203.0.113.0/24, 10.0.0.0/8: the public endpoint is restricted to listed IPs. VPC-internal traffic always uses the private endpoint.

This setting can be toggled on existing clusters without recreating them. It is a property of the access policy rather than the underlying compute.

Control which teams can use a cluster

A cluster grants access to teams. A job or notebook can only be assigned to a team the cluster includes, and saving one with a team the cluster does not include fails with a message that the team is not linked to the selected cluster.

Compute Profiles work differently. A profile has no team of its own, so it carries no team to link. What it needs is a cluster that has at least one team; a cluster with none is reported as not accessible to you. Who may then create the profile depends on where their permission comes from:

• An organization role that grants profile creation is enough on its own. The member does not have to be on any of the cluster's teams.
• A team role that grants it requires membership of one of the cluster's teams.

Creating a cluster grants access to the creator's teams, so a single-team organization never has to think about this. It starts to matter when a second team needs the same cluster, or when the person creating the cluster is not on the team that will use it.

The cluster page lists the teams that have access, under Compute → Clusters. Changing that list is not yet available in the console; it is done through the API, with AddTeamsToCluster and RemoveTeamFromCluster.

Removing a team does not check whether its jobs or notebooks still use the cluster. Those workloads stay assigned and fail when they next run, so move or reassign them before removing the team.

Team access is separate from what the team may then do. A member acts according to their team roles, so adding a team does not by itself let every member create or destroy anything. See Roles & Permissions.

When to create a new cluster instead

If the editor will not accept the change (region, network, or name) or the new configuration differs significantly from the current one, create a new cluster alongside the old one. Move workloads over by re-creating sessions and re-targeting jobs, then destroy the original.

Destroy a cluster

Close sessions, stop notebooks, and let job runs finish before destroying a cluster. LakeSail blocks a user-requested destroy while compute is active.

1. Open the cluster detail page.
2. Click Destroy (or Delete infrastructure).
3. Confirm. The cluster moves Destroying → Destroyed, releasing the underlying AWS resources.

Destroying is irreversible

Job run history, catalog registrations, and team assignments stay in the organization, but the compute is released. Budget a few minutes for teardown to complete.

Reading a cluster's state

The console reports three things about a cluster, in the list and again on its detail page. They answer different questions:

Signal	Question it answers
Status	What did the last infrastructure operation do?
Readiness	Can work run here right now?
Pods	What did the last health check count?

Status shows whether the latest provision or update was successful.

Readiness is usually the one you want. It reads Ready when the cloud account, the network, and the cluster are all usable and there is an endpoint to schedule onto, and Not ready otherwise, naming what blocks it. Unknown means the check has not run or could not finish.

Pods is the evidence behind readiness: how many pods the last health check found running, and how many it could not schedule. A dash means no count was taken. Hover either for the check's own message.

An Update action appears on a cluster whose infrastructure template is out of date, and runs the same update as the row menu.

Health, metrics, and direct access

The cluster's detail page shows its Readiness and a Pods count. Through the API you can go further:

• Health (GetExternalClusterHealth…): overall status, each EKS node group with its desired and ready node counts, and the health of core system pods.
• Metrics (GetExternalClusterMetrics…): CPU and memory utilization over a period you choose.
• Kubeconfig (GetExternalClusterKubeconfig…): a kubeconfig for the EKS cluster, for kubectl access to the cluster in your own account.

Lifecycle reference

Status	Meaning
Pending	Creation accepted, queued
Provisioning	AWS resources being created
Deployed	The last provision or update completed successfully; check Readiness before scheduling work
Updating	A configuration change is rolling out
Failed	Provisioning or update failed. See the progress panel for the failed stage
Destroying	Teardown in progress
Destroyed	Cluster resources released; record retained for audit
Deleted	Record removed from the organization

API reference

• Clusters: CreateExternalCluster, UpdateExternalCluster, DeleteExternalCluster, and the full lifecycle.
• Networks: required prerequisite.
• Cloud accounts: required prerequisite.