Notebooks
A notebook is a hosted, Marimo-based interactive Python notebook backed by a Sail session. Where a session is a runtime you connect your own client to, a notebook provides both the editor and the runtime.
When to use a notebook
- Exploratory Python + Spark in one place. You want to write PySpark and see results without wiring up a local client or managing tokens.
- Team-managed exploration. Team Admins and Editors can manage the notebook runtime while the creator retains access to the editor.
- Authoring before you productionize. Prototype a transform in a notebook, then run it as a job once it's stable. The job adds a schedule, versions, and run history.
When not to use one:
- Connecting an external client (local PySpark or a service). Use a session; it exposes Spark Connect over gRPC for exactly that.
Prerequisites
- A compute profile (workload config) on a cluster whose Readiness is Ready. The profile defines the Sail pod the notebook runs on: instance type, libraries, and environment. The notebook kernel runs on its own node of the profile's driver instance type, so results you collect into Python have the same memory as the driver. The node launches in the driver's availability zone when that zone has capacity. Its environment variables also reach the notebook kernel; see Environment variables and secrets.
- A catalog if you want to query your tables (recommended). A notebook inherits the catalogs attached to its compute profile, and can override that set with its own. For S3-backed tables, the data must be in the compute profile's network workspace bucket or in an external data target attached to the profile.
Notebook timing
A notebook uses a backing session with these clocks:
| Setting | Value | What happens |
|---|---|---|
| Idle timeout | 30 minutes by default; 2 minutes to 8 hours | An inactive notebook's backing session moves to Idle. Activity is the notebook being open in a browser, a request through the session proxy, or a running cell. |
| Idle close delay | 5 minutes by default | The idle session closes and the notebook stops. Opening the notebook during the delay keeps it. Notebook contents persist. |
| Maximum duration | 7 days per start | The notebook stops regardless of activity. Starting it again creates a new backing session and resets this clock. Organization settings can change this value. A platform node upgrade can end a notebook that has run longer than 48 hours. |
Create a notebook
- Open Notebooks in the sidebar and click Create Notebook.
- Fill in:
- Notebook name: e.g.
daily-sales-analysis. - Team: who can access the notebook according to their team roles. Only the creating member can open the notebook editor.
- Compute profile: the profile whose Sail pod backs the notebook. Pick an existing one or create a new one inline.
- Idle timeout (minutes): how long the notebook may remain inactive before its backing session becomes idle. See Notebook timing, or accept the default.
- Start from a marimo file (optional): the notebook opens with the file's cells instead of an empty notebook. See Import and download the notebook file.
- Notebook name: e.g.
- Click Create notebook. The notebook starts in Stopped.
Start, open, and stop
A notebook has its own lifecycle, independent of the workload it runs:
| Status | Meaning |
|---|---|
| Stopped | No Sail or Marimo pods are running |
| Starting | The platform is provisioning the Sail and Marimo pods |
| Running | Fully active and ready to use |
| Stopping | Tearing the pods down |
| Error | Provisioning or teardown failed; check the status message |
- Start provisions the pods and moves the notebook to Running.
- Open launches the Marimo editor against the running notebook.
- Stop tears the pods down to reclaim compute. Your notebook contents persist; only the runtime goes away.
Notebook contents are saved to the network's workspace bucket under notebooks/, in your AWS account. That is what survives a stop, and what remains if the notebook is deleted.
The platform also stops a notebook on its own. The idle timeout, set at creation and editable afterwards, controls when the backing session becomes idle. When it elapses, the session enters its idle close delay and then stops the notebook.
A cell that is still running counts as activity, whether or not the notebook is open in a browser. A long-running cell keeps the notebook alive until it finishes, and the idle timeout then runs from that point, so you have the full timeout to come back for the result.
Stopping early still pays
A Running notebook holds compute until the idle timeout fires, so stopping it yourself when you step away reclaims compute sooner. Starting it again is quick on a warm cluster.
Use Spark in a notebook
A new notebook's starter cells create the Spark session:
from pyspark.sql import SparkSession
spark = SparkSession.builder.getOrCreate()The runtime sets SPARK_REMOTE to the backing session's Spark Connect address, so the call needs no URL. As in a job, SparkContext and RDDs are not available; see Check Your Spark Code.
SQL cells run on the same session when their engine is spark, as in the starter SQL cell.
In cluster mode, each Spark session has its own worker pool. Max nodes applies to each Spark session. SparkSession.builder.create() requests another pool. SparkSession.builder.getOrCreate() reuses the active or default session in the same Python process. It creates a session if neither exists. Reuse the same spark variable across cells when you want to use the same workers. See How Compute Runs and Scales for node allocation and reuse.
Logs
Logs on the notebook page opens two live streams: notebook, the Marimo pod that runs the editor and your cells, and engine, the Sail driver of the backing session, where engine-side failures such as out-of-memory kills appear. They are available from the moment the notebook starts, and no longer available once its resources are cleaned up after a stop.
Environment variables and secrets
The environment variables on the notebook's compute profile are available inside cells through os.environ, alongside the Sail engine. Use a managed secret for any credential your code needs, such as a database password or an API token, and reference it from the profile.
Never paste a credential into a cell. Notebook contents are saved as plain text, so a pasted value is stored with the notebook and shared with everyone who can read it. The same applies to printed values: a secret written to cell output is saved with the notebook.
Values are resolved when the notebook starts. After changing a variable or replacing a secret, stop and start the notebook to pick up the new value.
A few variable names are set by the notebook runtime itself, such as the Spark Connect address and the Python import path. A profile variable with one of those names is not delivered; the notebook refuses to start and its status message names the variable. Rename it in the compute profile.
External data access
Spark reads and writes in a notebook go through Sail, so they can reach any external data target attached to the compute profile. The notebook's own Python process does not get the profile's AWS access. If you call S3 directly with boto3, pandas, or another Python client, give that client its own credentials through a managed secret reference instead of pasting them into a cell.
Import and download the notebook file
A notebook's cells live in one marimo file in your network's workspace bucket. The notebook's page shows it in the Notebook file row, with when it was last saved, and lets you replace it or download it without starting the notebook. Only the notebook's owner sees the row.
The file must be a marimo notebook: a .py file of up to 1 MiB that imports marimo and declares its app. Jupyter (.ipynb), Databricks, and marimo markdown notebooks are refused. The file's code runs on your cluster with the notebook's access to your data, so you confirm that you trust it before it is uploaded.
Import replaces the notebook's cells with a file, and the notebook starts from it next time. Stop the notebook first: while it runs, the editor saves over the file. Download the current cells first if you want to keep them.
Download saves the notebook's file. While the notebook runs, you get its last save, which can be a few seconds behind the editor.
A notebook has no file until it first runs or you import one.
Run a notebook as a job
A notebook job runs this notebook on a schedule or from an orchestrator. Each run copies the notebook's saved contents when it starts, so you can keep editing while it runs.
- Open the notebook and click Run as job.
- The job form opens with Notebook as the job type and this notebook selected. It starts on the first of the notebook's teams you belong to, and on the notebook's compute profile if that team can use it. Check both, add a schedule or parameters if you need them, then click Save.
The job runs with its own catalogs, which start from the compute profile's. Catalogs you customized on the notebook are not copied.
Run as job is available once the notebook has saved source: after it first runs, or after you import a file into it. Only the notebook's owner can run it as a job.
Change the compute profile
The compute profile is editable only while the notebook is stopped. To move a notebook to bigger compute (or a different cluster), stop it, change the profile, and start it again.
Team access
The notebook's team controls access according to each member's team role:
- Team Admin and Editor can edit, start, and stop the notebook.
- Viewer can view its details.
- Only the creating member can open the notebook editor.
Organization roles can also grant management access, but they do not override the creator check for opening the editor.
API reference
- Notebooks:
CreateNotebook,StartNotebook,StopNotebook,OpenNotebook,UpdateNotebook,DeleteNotebook, plus team sharing. The notebook file:CreateNotebookSourceUpload,ImportNotebookSource,CreateNotebookSourceDownload,DescribeNotebookSource. - API Reference: the workload config (compute profile) a notebook references.