Sessions

A session is a live Spark Connect runtime on a cluster. Clients submit SQL and DataFrame operations directly over gRPC. Compatible clients include PySpark, Scala, and community Go and Rust clients. This page covers when to use a session, how to connect, and how to keep one running.

Prerequisites

• A cluster whose Readiness is Ready. A Deployed status records a successful infrastructure operation, but does not by itself mean the cluster can accept work. See Set up a cluster.
• A compute profile for that cluster. The profile is where the cluster and the catalogs live: a session inherits whichever catalogs are attached to its profile, and the profile's default catalog becomes the Sail engine default, so SQL doesn't have to qualify table paths. For S3-backed tables, the underlying data must be in the profile's network workspace bucket. See Connect a catalog.
• A Spark Connect client locally (PySpark 3.5+, the Scala client jar, or spark-connect-go / spark-connect-rs).

When to use a session

Sessions are the right fit when you want an interactive, warm connection to a cluster:

• Ad-hoc exploration. Open a local Python script or notebook, connect via PySpark, and work with your catalog tables. Fast iteration without committing anything to a job definition.
• Debugging a failed job. A batch run landed in Failed or got stuck in Waiting for Sail. Attach an interactive session and re-run the same logic to see what the data actually looks like, which is much faster than editing and redeploying the job.
• Authoring pipelines. Write and test transforms interactively in PySpark, then copy the working code into a job definition once it's stable. The session is your scratch space; the job is the reproducible output.
• Programmatic consumers. A Go or Python service (e.g. a dashboard backend, a data-quality check, an on-demand report generator) connects via spark-connect-go or PySpark and runs Spark operations on request.

When not to use a session:

• Scheduled analytics or recurring dashboards. Use a job. Sessions are connection-driven and don't carry the reproducibility, versioning, or run history a schedule needs.
• Enterprise BI tools. The mainstream options don't natively speak Spark Connect today. Materialize output into a warehouse with a scheduled job and point the BI tool at that instead.
• A hosted notebook, no local setup. If you want an interactive Python environment without configuring a client or managing tokens, use a notebook instead. It is a session with the editor and runtime hosted for you.

The connection shape

Every session uses the Spark Connect URL shown when you create a connection token. The URL includes the session endpoint and a short-lived token scoped to that session.

Tokens expire. Plan to regenerate, not to hardcode.

Session timing

These clocks are independent:

Setting	Value	What happens
Token validity	8 hours by default; 1 hour to 90 days	The connection URL stops authenticating. The session stays open, so its owner can issue another token.
Idle timeout	30 minutes by default; 1 minute to 8 hours	No Spark Connect traffic moves the session to Idle.
Idle close delay	5 minutes	An Idle session becomes Closed and releases its compute. Traffic after the session becomes Idle does not reset this delay.
Maximum duration	24 hours	The session closes regardless of activity or token validity. Organization settings can change this value. The session detail page shows the effective Max Duration.

1. Create the session

1. Open Sessions in the sidebar and click Create Session.
2. Give the session a Session name, and pick the Team that can access it according to team roles (the field is hidden when you belong to only one team).
3. Pick a Compute profile. This is what selects the cluster and the catalogs. You can create a new profile inline if none fits.
4. Set Token valid for (hours) and Idle timeout (minutes), or accept the defaults.
5. Click Create session. The session moves Pending → Active.

2. Issue a token

From the session's detail page, click New connection token. The platform issues the JWT and shows the full sc:// URL, with the session's grpcEndpoint already filled in. Tokens are shown once; if you lose one, issue a new token (the old one keeps working until it expires).

Only the session owner can issue tokens for a session. Owner access does not override this ownership check.

3. Connect from a Spark Connect client

PySpark

from pyspark.sql import SparkSession

spark = SparkSession.builder.remote(
    "sc://<grpc-endpoint>/;token=<jwt>"
).getOrCreate()

spark.sql("SHOW TABLES").show()

Requires PySpark 3.5+ (pip install pyspark). The token= URI parameter is threaded through as Authorization: Bearer gRPC metadata automatically.

Spark Connect Scala

import org.apache.spark.sql.SparkSession

val spark = SparkSession.builder()
  .remote("sc://<grpc-endpoint>/;token=<jwt>")
  .getOrCreate()

Requires the spark-connect-client-jvm jar.

Go

The Apache spark-connect-go client takes the same URI shape. Pass the endpoint and token in the remote URL exactly as above.

Rust

Community Spark Connect clients for Rust are available (e.g. spark-connect-rs). They follow the same URI convention.

Connect an external agent

An external agent can use a session as its Spark runtime. Run a Spark Connect client in the agent's tool process and connect with the URL shown on the session detail page. The agent inherits the session's compute profile and attached catalogs.

LakeSail runs the Spark environment, not the agent or its model. The session owner must issue the connection token. Keep that token in the agent runtime, not in prompts or model context. Token expiration, the idle timeout, and the maximum duration still apply.

Keeping a session warm

A session stays warm while it receives traffic before the idle timeout. Once it becomes Idle, it closes after the idle close delay even if it receives another request.

To close a session manually, click Close session on the session detail page. A closed session can't be reopened, but Relaunch session on the same page creates a fresh session from the same compute profile, so you don't have to reconfigure it. The relaunched session gets a new session ID.

Troubleshooting

• UNAUTHENTICATED / token rejected. The token expired. Generate a new one.
• 403 Forbidden when issuing a token. Only the session owner can issue one. Open the session as its owner; the Owner role alone is not sufficient.
• The first request hangs. The session is warming. Subsequent requests should be fast.
• UNAVAILABLE / connection refused. Use the connection URL exactly as shown. If the session is still starting, wait for it to become Active, then create a new connection token.

API reference

• Sessions: create, describe, and close sessions; IssueSessionToken for the JWT used by Spark Connect clients.