Troubleshooting

Issues on this page correspond to error states you will see in the UI, not hypothetical edge cases. Each entry lists the message exactly as it appears so search lands you here.

Before anything else, check which layer is failing: cloud account, network, cluster, or workload. A failure at a lower layer (e.g. cloud account) makes everything above it look broken. If the layering isn't familiar, see Concepts first.

Cloud account: "We couldn't verify the connection"

Message: "We couldn't verify the connection. Check the stack status in CloudFormation, then try Verify Manually in the Advanced section."

What happened: LakeSail tried to assume the IAM role you pasted and the call failed (failed_to_assume_role). The CloudFormation stack may not have finished, the role ARN may be wrong, or the trust policy may have been edited.

Fix:

1. Open the CloudFormation stack in your AWS console. Confirm that CloudFormation finished creating it successfully.
2. Open the Outputs tab. Copy the Role ARN exactly. Stray spaces and extra characters are a common cause.
3. In LakeSail, open the connection setup modal, expand Advanced, paste the ARN into IAM Role ARN, and click Verify Manually.
4. If it still fails, open the role in IAM and confirm the trust policy includes the LakeSail principal and external ID shown in the modal. If you modified the template, redeploy the stack unchanged.

Cloud account: "Failed to create connection"

Message: "Failed to create connection. Please try again."

What happened: The ARN verified, but the backend couldn't persist the connection record.

Fix: Retry. If it repeats, capture the timestamp and contact support. This is not a config issue on your side.

Network: "Failed to load regions"

Message: "Failed to load regions. Please try again."

What happened: LakeSail tried to enumerate AWS regions through your connected cloud account and the assume-role call failed. The trust policy may have changed, the role may have been deleted, or the external ID may be out of sync.

Fix:

1. Go to Settings → Cloud Accounts and check the account's status.
2. If it is Disconnected or Failed, redeploy the CloudFormation stack using the same connection ID (do not create a new one) and re-verify.
3. If it is Active, wait a minute and retry. Transient IAM propagation delays resolve on their own.

Network: CIDR overlap

What happened: The CIDR you entered overlaps the LakeSail platform VPC (or a VPC you're peering with), so provisioning refuses to continue.

Fix: Pick a private CIDR that does not overlap the LakeSail platform VPC or any VPC you plan to peer.

Cluster: "Min/Desired nodes cannot exceed max nodes"

Messages:

• "Min nodes cannot exceed max nodes."
• "Desired nodes cannot exceed max nodes."

What happened: Node group sizes are out of order.

Fix: Adjust the numbers so min ≤ desired ≤ max. The autoscaler needs a valid range before it will accept the config.

Cluster: instance too small for system pods

Message pattern: "{instance type} supports only N pods per node, but each system node needs at least M (K DaemonSet pods + L rolling-restart margin). Choose a larger instance type."

What happened: The instance type you picked has a per-node pod limit below what LakeSail's system pods need. This is not about your workload. It covers the VPC CNI, kube-proxy, EBS CSI, and a margin for rolling restarts.

Fix: Pick a larger management-node instance. m8g.large (the default) fits comfortably; anything smaller usually does not.

Cluster: not enough CPU for system controllers

Message pattern: "{instance type} with N minimum node(s) provides ~Xm allocatable CPU, but at least Ym are needed for system controllers. Choose a larger instance type or increase the minimum node count."

What happened: Your min node count × the instance's allocatable CPU is not enough for Karpenter and other system controllers. Pods will fail to schedule.

Fix: Either raise Min Management Nodes or pick a larger instance type. The defaults (1 min, m8g.large) are fine for most evaluation setups. For all the sizing fields and their defaults, see Clusters.

Cluster: not enough memory for system controllers

Message pattern: "{instance type} provides ~N MiB of allocatable memory per node, but the system controllers need at least M MiB on a single node. Choose an instance type with more memory."

What happened: One management node of this instance type cannot hold LakeSail's system pods at their memory limits. Adding nodes does not help, because one node may have to run them all. Instances with 2 GiB of memory are too small.

Fix: Pick a management-node instance type with at least 4 GiB of memory. The tier presets already meet this.

Secret: unavailable to a resource or workload

Message pattern: "secret reference is missing, revoked, or not available..."

What happened: The secret was deleted, its current value was revoked, or it is not shared with every team that can use the resource.

Fix:

1. Open Settings → Secrets and confirm that the secret exists and is Active.
2. If it is Revoked, edit the secret and add a replacement value.
3. Check its teams. A job or catalog secret must include every team assigned to that resource. A compute profile secret must include every team granted access to its cluster.
4. If the reference is inside a JSON environment variable, confirm that its secretId matches the secret ID returned by ListSecrets.
5. Start a new run or restart the session or notebook after correcting the secret.

See Manage Secrets for the full workflow.

Secret: team cannot be removed

Message: "cannot remove secret teams while referenced resources would lose access"

What happened: A job, catalog, or compute profile still references the secret and is available to the team you tried to remove.

Fix: Remove the reference or update the resource's teams first. Then remove the team from the secret.

Secret: cannot be deleted

Message: "cannot delete a secret while it is referenced"

What happened: A job, catalog, compute profile, or JSON template still references the secret.

Fix: Remove every reference, then delete the secret from Settings → Secrets.

External data: "Access denied"

What happened: A workload or access test could not reach an external data target. The grant, the bucket policy, or the path does not allow the request.

Fix: Check each of these in turn:

Check	What to do
Grant	Open the profile's Grants tab. Confirm the grant is active and has the access the workload needs. A directory read needs both List objects and Read objects.
Bucket policy	Ask the bucket owner to confirm the profile's statements are in the bucket policy, and that no explicit Deny overrides them.
Path	End a prefix path with /, and keep it inside the registered prefix. See Read the data.
Target details	Confirm the target's AWS account ID and region match the bucket.
Encryption	SSE-KMS encrypted objects are not supported. Check how the objects you read are encrypted.
Notebook code	Read through Sail or PySpark. boto3, pandas, and other direct Python clients do not get the profile's access.
Job runner role	Once a profile has a grant, its workloads stop using the cluster's job runner role. Permissions you added to that role no longer apply; grant them through a data target grant instead.

External data: "incorrectly configured region"

Message: "Received redirect without LOCATION, this normally indicates an incorrectly configured region"

What happened: The bucket is in a different region from the compute profile's cluster, and the workload used an s3:// path.

Fix: Use an https:// path that names the bucket's region. See Bucket in another region.

External data: grant cannot change

Message: "Stop active workloads using this compute profile before changing grants."

What happened: A notebook, session, or job run is using the compute profile. LakeSail changes the profile's IAM role only when nothing is running on it.

Fix: Stop the profile's notebooks, close its sessions, and let its job runs finish. Then grant or remove the target again.

External data: policy size limit

Message: "grants exceed the IAM managed policy size limit; remove a target or request fewer access levels"

What happened: All grants on a compute profile share one AWS managed policy, and this change would make it larger than AWS allows. See Policy size limit.

Fix: Remove a grant the profile no longer needs, or grant the target with less access. Then try again.

External data: target cannot be removed

Message: "Compute profiles have access grants for this target. Remove all grants before deleting it. Ask an organization administrator for help with profiles you cannot access."

What happened: At least one compute profile still holds a grant for the target.

Fix: Remove the grant from each profile's Grants tab, then remove the target under Settings → External Data. If a profile is hidden from you, ask an organization administrator to remove its grant.

Notebook job refused

What happened: A notebook job could not be saved or started. The message names the reason.

Fix:

Message starts with	What to do
"a job can only run a notebook you own"	Ask the notebook's owner to create the job.
"this notebook has no saved source for a job to copy"	Open the notebook and save it, or import a file into it, then create the job.
"this notebook runs on platform-managed compute"	Move the notebook to a compute profile in your own cloud account.
"this job's compute profile is not backed by your own cloud account"	Pick a compute profile on a cluster in your cloud account.
"this job's compute profile is in a different network than the notebook"	Pick a compute profile in the notebook's network.
"this job's compute profile runs on a cluster that predates notebook jobs"	Click Update on the cluster, or pick a profile on an up-to-date one. See Clusters.
"this job runs a notebook that no longer exists"	Point the job at an existing notebook, or delete the job.

Job run status is Failed

What happened: Usually the job's own code failed. SQL syntax problems, missing columns, and Python exceptions surface in the run's Status Message and logs. The run's terminationReason in the API says whether the code failed (user_error) or something else did, such as a memory limit or the cluster. See Why a run ended.

Fix: Read the message, then check the usual causes: a table or column that does not exist in the catalog, permission denied on S3 or Glue (see the workload boundary), or a Python wheel missing a dependency. Correct the job and retry the run.

Job run stuck in Waiting for Sail

Underlying error: "timeout waiting for sail server to be ready"

What happened: The Sail engine pod for this run could not reach Ready within ~10 minutes. Usual causes: the cluster is scheduling new compute nodes (cold start), an image pull failed, or the node group cannot grow because of AWS capacity or quota.

Fix:

1. Check the compute profile's Max nodes. If all available worker capacity is busy, raise the maximum or wait for other work to finish.
2. If no new nodes appear in AWS after 5 minutes, check your EC2 service quota in the region. Karpenter will surface quota errors to the cluster event log.
3. If the issue persists on a healthy cluster, cancel the run and retry. Repeated failures with no resource explanation are worth sending to support with the run ID.

Job run status is Timeout

What happened: The run stayed in Running longer than the job's configured timeout and was stopped. That time includes starting the runner pod and installing libraries.

Fix: Raise the timeout if the workload legitimately grew, or find out why it is slower than before. A common cause is a catalog that returned a much larger result set. See Timeout.

Scheduled tick did not fire

What happened: A scheduled run did not start when expected.

Fix:

1. Confirm the job is Active. A paused job does not run on schedule.
2. Check the cron expression against a cron parser; it may not mean what you think.
3. Check the cluster was healthy at that time. A Failed or Destroying cluster prevents a run from starting.
4. Check the missed-schedule policy. latest fires once on recovery, not once per missed tick.

Scheduled runs overlap unexpectedly

What happened: A tick fired while an earlier run was still going, and both ran or the first was cancelled.

Fix: Revisit the concurrency policy. skip drops a tick while an earlier run is unfinished; replace cancels that run and starts a new one; allow starts runs up to Max concurrent runs, and past that a tick waits as overdue.

Job run fails at start: "failed to create kubernetes client"

What happened: LakeSail could not open the tunnel to your cluster. Either the cluster is unreachable from the control plane (VPC changed, security groups modified, NAT gateway down) or the cluster's credentials rotated out from under it.

Fix:

1. Check the cluster status. If it is Failed or Updating, wait for it to return to Deployed before retrying.
2. If you recently modified the network's security groups or route tables outside LakeSail, revert. The VPC is managed, and changes made directly in AWS can break connectivity.
3. If the cluster looks healthy but every run fails with this error, contact support with the run ID.

Session: token rejected

What happened: The session token reached the validity period selected when it was issued.

Fix: Generate a new token from the session detail page. Plan to refresh or rotate tokens before they expire. See Sessions for the full list of connection errors.

Still stuck?

Email support@lakesail.com. Include the organization ID, the failing resource ID (cloud account, network, cluster, or run), the timestamp, and the error message or relevant logs.