Sessions

A session is a live Spark Connect runtime on a cluster. Clients submit SQL and DataFrame operations directly over gRPC. Compatible clients include PySpark, Scala, and community Go and Rust clients. This page covers when to use a session, how to connect, and how to keep one running.

Prerequisites

• A cluster whose Readiness is Ready. A Deployed status records a successful infrastructure operation, but does not by itself mean the cluster can accept work. See Clusters.
• A compute profile for that cluster. The profile is where the cluster and the catalogs live: a session inherits whichever catalogs are attached to its profile, and the profile's default catalog becomes the Sail engine default, so SQL doesn't have to qualify table paths. For S3-backed tables, the data must be in the profile's network workspace bucket or in an external data target attached to the profile. See Connect a Catalog.
• A Spark Connect client locally (PySpark 3.5+, the Scala client jar, or spark-connect-go / spark-connect-rs).

When to use a session

Use a session when your own client needs a live connection: a local PySpark script, a service that runs Spark operations on request, or a debugging pass against the data a failed run saw. For scheduled work use a job; for an editor with the runtime hosted for you, use a notebook. BI tools do not speak Spark Connect; materialize results with a job and point the tool at the output.

The connection shape

Every session uses the Spark Connect URL shown when you create a connection token. The URL includes the session endpoint and a short-lived token scoped to that session.

Tokens expire. Plan to regenerate, not to hardcode.

Session timing

These clocks are independent:

Setting	Value	What happens
Token validity	The session's remaining maximum duration by default; 1 minute (whole hours in the console) to 90 days, and never past the session's maximum duration	The connection URL stops authenticating. The session stays open, so its owner can issue another token. The platform does not issue the next token by itself; a client can, as below.
Idle timeout	30 minutes by default; 2 minutes to 8 hours	No Spark Connect traffic moves the session to Idle. A running query counts as traffic. The clock starts when the session becomes usable, not when it is created.
Idle close delay	5 minutes by default; at least 2 minutes	An Idle session becomes Closed and releases its compute. Traffic during the delay returns the session to Active.
Maximum duration	7 days	The session closes regardless of activity or token validity. Organization settings can change this value. The session detail page shows the effective Max Duration. A platform node upgrade can end a session that has run longer than 48 hours.

1. Create the session

1. Open Sessions in the sidebar and click Create Session.
2. Give the session a Session name, and pick the Team that can access it according to team roles (the field is hidden when you belong to only one team).
3. Pick a Compute profile. This is what selects the cluster and the catalogs. You can create a new profile inline if none fits.
4. Set Token valid for (hours) and Idle timeout (minutes), or accept the defaults.
5. Click Create session. The session moves Pending → Active.

2. Issue a token

From the session's detail page, click New connection token. The platform issues the JWT and shows the full sc:// URL, with the session's grpcEndpoint already filled in. Tokens are shown once; if you lose one, issue a new token (the old one keeps working until it expires).

The Logs section streams the Sail driver's log while the session is active. In cluster mode, worker logs are not exposed. Once the session's resources are cleaned up the log is no longer available.

Only the session owner can issue tokens for a session. Owner access does not override this ownership check.

3. Connect from a Spark Connect client

PySpark

from pyspark.sql import SparkSession

spark = SparkSession.builder.remote(
    "sc://<grpc-endpoint>/;token=<jwt>"
).getOrCreate()

spark.sql("SHOW TABLES").show()

Requires PySpark 3.5+ (pip install pyspark). The token= URI parameter is threaded through as Authorization: Bearer gRPC metadata automatically.

A LakeSail session can host more than one Spark session. In cluster mode, each Spark session has its own worker pool. Max nodes applies to each Spark session. SparkSession.builder.create() requests another pool. SparkSession.builder.getOrCreate() reuses the active or default session in the same Python process. It creates a session if neither exists. Reuse the same spark variable when you want to use the same workers. See How Compute Runs and Scales for node allocation and reuse.

Spark Connect Scala

import org.apache.spark.sql.SparkSession

val spark = SparkSession.builder()
  .remote("sc://<grpc-endpoint>/;token=<jwt>")
  .getOrCreate()

Requires the spark-connect-client-jvm jar.

Go

The Apache spark-connect-go client takes the same URI shape. Pass the endpoint and token in the remote URL exactly as above.

Rust

Community Spark Connect clients for Rust are available (e.g. spark-connect-rs). They follow the same URI convention.

Keep a long-running PySpark client connected

A connection token expires, and the URL then stops authenticating. A client that runs longer than one token, such as a scheduled pipeline or an agent, can issue its own next token instead of holding a long one. Copy this file next to your code:

"""A Spark Connect session on LakeSail whose connection token renews itself.

Copy this file next to your code. It needs PySpark or pyspark-client 4.0 or later, and nothing else:
gRPC and the HTTP call come with Python and PySpark.
"""

import json
import re
import threading
import time
import urllib.error
import urllib.request
from datetime import datetime

import grpc
from pyspark.sql.connect.client import DefaultChannelBuilder
from pyspark.sql.connect.session import SparkSession

LAKESAIL_API = "https://platform.lakesail.com/api/v1"


class LakeSailTokenError(Exception):
    """The platform refused a connection token, with its reason and what clears it."""


class _NoRedirect(urllib.request.HTTPRedirectHandler):
    # A redirect would carry the personal access token to another address, so none is followed.
    def redirect_request(self, req, fp, code, msg, headers, newurl):
        return None


_opener = urllib.request.build_opener(_NoRedirect)


class SessionTokenSource(grpc.AuthMetadataPlugin):
    """Gives every Spark Connect call a valid connection token, and issues the next one before the
    current one expires."""

    def __init__(
        self,
        session_id,
        organization_id,
        access_token,
        lifetime_seconds=3600,
        api_url=LAKESAIL_API,
    ):
        self._url = f"{api_url}/workloads/sessions/{session_id}/token"
        self._headers = {
            "Authorization": f"Bearer {access_token}",
            "LakeSail-Organization-Id": organization_id,
            "Content-Type": "application/json",
        }
        self._body = json.dumps({"durationSeconds": lifetime_seconds}).encode()
        self._lock = threading.Lock()
        self._token = None
        self._renew_at = 0.0

    def token(self):
        with self._lock:
            if time.time() >= self._renew_at:
                self._issue()
            return self._token

    def _issue(self):
        request = urllib.request.Request(
            self._url, self._body, self._headers, method="POST"
        )
        try:
            with _opener.open(request, timeout=30) as response:
                issued = json.load(response)
        except urllib.error.HTTPError as error:
            raise LakeSailTokenError(_platform_message(error)) from None
        # Whole seconds are enough, and every Python version parses them.
        expires_at = re.sub(r"\.\d+", "", issued["expiresAt"]).replace("Z", "+00:00")
        expires_at = datetime.fromisoformat(expires_at).timestamp()
        # Issue the next token with a quarter of this one's life left, at most five minutes early.
        now = time.time()
        self._token = issued["token"]
        self._renew_at = expires_at - min(300, (expires_at - now) / 4)

    def __call__(self, context, callback):
        # gRPC calls this before every call. An error fails that call as unavailable, which PySpark
        # retries.
        try:
            callback((("authorization", f"Bearer {self.token()}"),), None)
        except Exception as error:
            callback((), error)


def _platform_message(error):
    try:
        message = json.load(error)["message"]
    except Exception:
        message = error.reason
    return f"LakeSail refused a connection token (HTTP {error.code}): {message}"


class RenewingChannelBuilder(DefaultChannelBuilder):
    """A Spark Connect channel that asks token_source for the token of every call."""

    def __init__(self, grpc_endpoint, token_source, secure=True):
        super().__init__(f"sc://{grpc_endpoint}")
        self._token_source = token_source
        self._secure = secure

    def toChannel(self):
        token_credentials = grpc.metadata_call_credentials(self._token_source)
        if self._secure:
            transport_credentials = grpc.ssl_channel_credentials()
        else:
            # A local proxy without TLS, as in LakeSail's own tests.
            transport_credentials = grpc.local_channel_credentials()
        credentials = grpc.composite_channel_credentials(
            transport_credentials, token_credentials
        )
        return self._secure_channel(self.endpoint, credentials)


def renewing_spark_session(
    grpc_endpoint, session_id, organization_id, access_token, lifetime_seconds=3600
):
    """A new SparkSession on a LakeSail session, whose short-lived token renews itself.

    The first token is issued here, so a refusal (a wrong personal access token, a session that has
    ended) raises LakeSailTokenError with the platform's reason at once.
    """
    token_source = SessionTokenSource(
        session_id, organization_id, access_token, lifetime_seconds
    )
    token_source.token()
    channel_builder = RenewingChannelBuilder(grpc_endpoint, token_source)
    # create(), not getOrCreate(): an existing session in the process would not use this channel.
    return SparkSession.builder.channelBuilder(channel_builder).create()

Then open the session with a personal access token instead of a connection URL:

import os

from renewing_spark_session import renewing_spark_session

spark = renewing_spark_session(
    grpc_endpoint="<grpc-endpoint>",
    session_id="<session-id>",
    organization_id="<organization-id>",
    access_token=os.environ["LAKESAIL_TOKEN"],
)
spark.sql("SHOW TABLES").show()

The helper issues a connection token that lives one hour, and issues the next one before the current one expires, so a leaked connection token works for an hour at most. The personal access token is the long-lived secret: keep it out of code, and delete it if it leaks.

The helper issues its first token when it starts, so a refusal, such as a wrong personal access token or a session that has ended, raises LakeSailTokenError at once with the platform's reason. Later, if a token cannot be issued in time, the Spark Connect call fails as unavailable and PySpark retries it for some minutes, which rides out a network failure or a platform restart.

The helper needs PySpark or pyspark-client 4.0 or later, and nothing else. It does not work in an organization that requires DPoP.

Connect an external agent

An external agent can use a session as its Spark runtime. Run a Spark Connect client in the agent's tool process and connect with the URL shown on the session detail page. The agent inherits the session's compute profile and attached catalogs.

LakeSail runs the Spark environment, not the agent or its model. The session owner must issue the connection token. Keep that token in the agent runtime, not in prompts or model context. Token expiration, the idle timeout, and the maximum duration still apply.

Keeping a session warm

A session stays warm while it receives traffic before the idle timeout. Once it becomes Idle, it closes after the idle close delay even if it receives another request.

To close a session manually, click Close session on the session detail page. A closed session can't be reopened, but Relaunch session on the same page creates a fresh session from the same compute profile, so you don't have to reconfigure it. The relaunched session gets a new session ID.

Troubleshooting

• UNAUTHENTICATED / token rejected. The token expired. Generate a new one.
• 403 Forbidden when issuing a token. Only the session owner can issue one. Open the session as its owner; the Owner role alone is not sufficient.
• The first request hangs. The session is warming. Subsequent requests should be fast.
• UNAVAILABLE / connection refused. Use the connection URL exactly as shown. If the session is still starting, wait for it to become Active, then create a new connection token.

API reference

• Sessions: create, describe, and close sessions; IssueSessionToken for the JWT used by Spark Connect clients.