Troubleshooting

Issues on this page correspond to error states you will see in the UI, not hypothetical edge cases. Each entry lists the message exactly as it appears so search lands you here.

Before anything else, check which layer is failing: cloud account, network, cluster, or workload. A failure at a lower layer (e.g. cloud account) makes everything above it look broken. If the layering isn't familiar, see Concepts first.

Cloud account: "We couldn't verify the connection"

Message: "We couldn't verify the connection. Check the stack status in CloudFormation, then try Verify Manually in the Advanced section."

What happened: LakeSail tried to assume the IAM role you pasted and the call failed (failed_to_assume_role). The CloudFormation stack may not have finished, the role ARN may be wrong, or the trust policy may have been edited.

Fix:

1. Open the CloudFormation stack in your AWS console. Confirm that CloudFormation finished creating it successfully.
2. Open the Outputs tab. Copy the Role ARN exactly. Stray spaces and extra characters are a common cause.
3. In LakeSail, open the connection setup modal, expand Advanced, paste the ARN into IAM Role ARN, and click Verify Manually.
4. If it still fails, open the role in IAM and confirm the trust policy includes the LakeSail principal and external ID shown in the modal. If you modified the template, redeploy the stack unchanged.

Cloud account: "Failed to create connection"

Message: "Failed to create connection. Please try again."

What happened: The ARN verified, but the backend couldn't persist the connection record.

Fix: Retry. If it repeats, capture the timestamp and contact support. This is not a config issue on your side.

Network: "Failed to load regions"

Message: "Failed to load regions. Please try again."

What happened: LakeSail tried to enumerate AWS regions through your connected cloud account and the assume-role call failed. The trust policy may have changed, the role may have been deleted, or the external ID may be out of sync.

Fix:

1. Go to Settings → Cloud Accounts and check the account's status.
2. If it is Disconnected or Failed, redeploy the CloudFormation stack using the same connection ID (do not create a new one) and re-verify.
3. If it is Active, wait a minute and retry. Transient IAM propagation delays resolve on their own.

Network: CIDR overlap

What happened: The CIDR you entered overlaps the LakeSail platform VPC (or a VPC you're peering with), so provisioning refuses to continue.

Fix: Pick a private CIDR that does not overlap the LakeSail platform VPC or any VPC you plan to peer.

Cluster: "Min/Desired nodes cannot exceed max nodes"

Messages:

• "Min nodes cannot exceed max nodes."
• "Desired nodes cannot exceed max nodes."

What happened: Node group sizes are out of order.

Fix: Adjust the numbers so min ≤ desired ≤ max. The autoscaler needs a valid range before it will accept the config.

Cluster: instance too small for system pods

Message pattern: "{instance type} supports only N pods per node, but each system node needs at least M (K DaemonSet pods + L rolling-restart margin). Choose a larger instance type."

What happened: The instance type you picked has a per-node pod limit below what LakeSail's system pods need. This is not about your workload. It covers the VPC CNI, kube-proxy, EBS CSI, and a margin for rolling restarts.

Fix: Pick a larger management-node instance. m8g.large (the default) fits comfortably; anything smaller usually does not.

Cluster: not enough CPU for system controllers

Message pattern: "{instance type} with N minimum node(s) provides ~Xm allocatable CPU, but at least Ym are needed for system controllers. Choose a larger instance type or increase the minimum node count."

What happened: Your min node count × the instance's allocatable CPU is not enough for Karpenter and other system controllers. Pods will fail to schedule.

Fix: Either raise Min Management Nodes or pick a larger instance type. The defaults (1 min, m8g.large) are fine for most evaluation setups. For all the sizing fields and their defaults, see Set up a cluster.

Secret: unavailable to a resource or workload

Message pattern: "secret reference is missing, revoked, or not available..."

What happened: The secret was deleted, its current value was revoked, or it is not shared with every team that can use the resource.

Fix:

1. Open Settings → Secrets and confirm that the secret exists and is Active.
2. If it is Revoked, edit the secret and add a replacement value.
3. Check its teams. A job or catalog secret must include every team assigned to that resource. A compute profile secret must include every team linked to its cluster.
4. If the reference is inside a JSON environment variable, confirm that its secretId matches the secret ID returned by ListSecrets.
5. Start a new run or restart the session or notebook after correcting the secret.

See Manage secrets for the full workflow.

Secret: team cannot be removed

Message: "cannot remove secret teams while referenced resources would lose access"

What happened: A job, catalog, or compute profile still references the secret and is available to the team you tried to remove.

Fix: Remove the reference or update the resource's teams first. Then remove the team from the secret.

Secret: cannot be deleted

Message: "cannot delete a secret while it is referenced"

What happened: A job, catalog, compute profile, or JSON template still references the secret.

Fix: Remove every reference, then delete the secret from Settings → Secrets.

Job run stuck in Waiting for Sail

For the full run lifecycle and a step-by-step debugging checklist, see Runs & debugging.

Underlying error: "timeout waiting for sail server to be ready"

What happened: The Sail engine pod for this run could not reach Ready within ~10 minutes. Usual causes: the cluster is scheduling new compute nodes (cold start), an image pull failed, or the node group cannot grow because of AWS capacity or quota.

Fix:

1. Check the compute profile's Max nodes. If all available worker capacity is busy, raise the maximum or wait for other work to finish.
2. If no new nodes appear in AWS after 5 minutes, check your EC2 service quota in the region. Karpenter will surface quota errors to the cluster event log.
3. If the issue persists on a healthy cluster, cancel the run and retry. Repeated failures with no resource explanation are worth sending to support with the run ID.

Job run fails at start: "failed to create kubernetes client"

What happened: LakeSail could not open the tunnel to your cluster. Either the cluster is unreachable from the control plane (VPC changed, security groups modified, NAT gateway down) or the cluster's credentials rotated out from under it.

Fix:

1. Check the cluster status. If it is Failed or Updating, wait for it to return to Deployed before retrying.
2. If you recently modified the network's security groups or route tables outside LakeSail, revert. The VPC is managed, and changes made directly in AWS can break connectivity.
3. If the cluster looks healthy but every run fails with this error, contact support with the run ID.

Session: token rejected

What happened: The session token reached the validity period selected when it was issued.

Fix: Generate a new token from the session detail page. Plan to refresh or rotate tokens before they expire. See Sessions for the full list of connection errors.

Still stuck?

Email support@lakesail.com. Include the organization ID, the failing resource ID (cloud account, network, cluster, or run), the timestamp, and the error message or relevant logs.