Sessions

A session is a live Spark Connect runtime on a cluster. Clients submit SQL and DataFrame operations directly over gRPC. Compatible clients include PySpark, Scala, and community Go and Rust clients. This page covers when to use a session, how to connect, and how to keep one running.

Prerequisites

• A cluster whose Readiness is Ready. A Deployed status records a successful infrastructure operation, but does not by itself mean the cluster can accept work. See Clusters.
• A compute profile for that cluster. The profile is where the cluster and the catalogs live: a session inherits whichever catalogs are attached to its profile, and the profile's default catalog becomes the Sail engine default, so SQL doesn't have to qualify table paths. For S3-backed tables, the underlying data must be in the profile's network workspace bucket. See Connect a Catalog.
• A Spark Connect client locally (PySpark 3.5+, the Scala client jar, or spark-connect-go / spark-connect-rs).

When to use a session

Use a session when your own client needs a live connection: a local PySpark script, a service that runs Spark operations on request, or a debugging pass against the data a failed job saw. For scheduled work use a job; for an editor with the runtime hosted for you, use a notebook. BI tools do not speak Spark Connect; materialize results with a job and point the tool at the output.

The connection shape

Every session uses the Spark Connect URL shown when you create a connection token. The URL includes the session endpoint and a short-lived token scoped to that session.

Tokens expire. Plan to regenerate, not to hardcode.

Session timing

These clocks are independent:

Setting	Value	What happens
Token validity	The session's remaining maximum duration by default; 1 hour to 90 days	The connection URL stops authenticating. The session stays open, so its owner can issue another token. Tokens are not renewed automatically.
Idle timeout	30 minutes by default; 2 minutes to 8 hours	No Spark Connect traffic moves the session to Idle. A running query counts as traffic. The clock starts when the session becomes usable, not when it is created.
Idle close delay	5 minutes by default; at least 2 minutes	An Idle session becomes Closed and releases its compute. Traffic during the delay returns the session to Active.
Maximum duration	7 days	The session closes regardless of activity or token validity. Organization settings can change this value. The session detail page shows the effective Max Duration. A platform node upgrade can end a session that has run longer than 48 hours.

1. Create the session

1. Open Sessions in the sidebar and click Create Session.
2. Give the session a Session name, and pick the Team that can access it according to team roles (the field is hidden when you belong to only one team).
3. Pick a Compute profile. This is what selects the cluster and the catalogs. You can create a new profile inline if none fits.
4. Set Token valid for (hours) and Idle timeout (minutes), or accept the defaults.
5. Click Create session. The session moves Pending → Active.

2. Issue a token

From the session's detail page, click New connection token. The platform issues the JWT and shows the full sc:// URL, with the session's grpcEndpoint already filled in. Tokens are shown once; if you lose one, issue a new token (the old one keeps working until it expires).

The Logs section streams the Sail driver's log while the session is active. In cluster mode, worker logs are not exposed. Once the session's resources are cleaned up the log is no longer available.

Only the session owner can issue tokens for a session. Owner access does not override this ownership check.

3. Connect from a Spark Connect client

PySpark

from pyspark.sql import SparkSession

spark = SparkSession.builder.remote(
    "sc://<grpc-endpoint>/;token=<jwt>"
).getOrCreate()

spark.sql("SHOW TABLES").show()

Requires PySpark 3.5+ (pip install pyspark). The token= URI parameter is threaded through as Authorization: Bearer gRPC metadata automatically.

Spark Connect Scala

import org.apache.spark.sql.SparkSession

val spark = SparkSession.builder()
  .remote("sc://<grpc-endpoint>/;token=<jwt>")
  .getOrCreate()

Requires the spark-connect-client-jvm jar.

Go

The Apache spark-connect-go client takes the same URI shape. Pass the endpoint and token in the remote URL exactly as above.

Rust

Community Spark Connect clients for Rust are available (e.g. spark-connect-rs). They follow the same URI convention.

Connect an external agent

An external agent can use a session as its Spark runtime. Run a Spark Connect client in the agent's tool process and connect with the URL shown on the session detail page. The agent inherits the session's compute profile and attached catalogs.

LakeSail runs the Spark environment, not the agent or its model. The session owner must issue the connection token. Keep that token in the agent runtime, not in prompts or model context. Token expiration, the idle timeout, and the maximum duration still apply.

Keeping a session warm

A session stays warm while it receives traffic before the idle timeout. Once it becomes Idle, it closes after the idle close delay even if it receives another request.

To close a session manually, click Close session on the session detail page. A closed session can't be reopened, but Relaunch session on the same page creates a fresh session from the same compute profile, so you don't have to reconfigure it. The relaunched session gets a new session ID.

Troubleshooting

• UNAUTHENTICATED / token rejected. The token expired. Generate a new one.
• 403 Forbidden when issuing a token. Only the session owner can issue one. Open the session as its owner; the Owner role alone is not sufficient.
• The first request hangs. The session is warming. Subsequent requests should be fast.
• UNAVAILABLE / connection refused. Use the connection URL exactly as shown. If the session is still starting, wait for it to become Active, then create a new connection token.

API reference

• Sessions: create, describe, and close sessions; IssueSessionToken for the JWT used by Spark Connect clients.