Configuration
This page lists all the available configuration options for Sail.
WARNING
The Sail configuration system is not stable yet.
For options with the ⚠ note, breaking changes can happen across versions without notice.
For all other options, breaking changes can happen across minor versions for the 0.x releases. Such changes will be documented in the changelog.
INFO
The default value of each option is shown as its string representation that can be used directly in environment variables.
Core Options
modeSAIL_MODElocalThe running mode for Sail. Valid values are local, local-cluster, and kubernetes-cluster.
- In
localmode, Sail runs in a single process, while query execution is still parallelized via threads. - In
local-clustermode, Sail starts a cluster within a single process. The driver and workers run on different threads in the same process and communicate with each other via RPC. - In
kubernetes-clustermode, Sail manages a cluster in Kubernetes. The driver and workers run in separate pods and communicate with each other via RPC.
Cluster Options
cluster.driver_external_hostSAIL_CLUSTER__DRIVER_EXTERNAL_HOST127.0.0.1The external host for the worker to connect to the driver.
cluster.driver_external_portSAIL_CLUSTER__DRIVER_EXTERNAL_PORT0The external port for the worker to connect to the driver.
If the value is 0, the port is assumed to be the same as the port
on which the driver listens.
cluster.driver_listen_hostSAIL_CLUSTER__DRIVER_LISTEN_HOST127.0.0.1The host on which the driver listens.
cluster.driver_listen_portSAIL_CLUSTER__DRIVER_LISTEN_PORT0The port on which the driver listens.
If the value is 0, a random port is assigned by the operating system.
cluster.enable_tlsSAIL_CLUSTER__ENABLE_TLSfalseWhether to enable TLS for cluster communication.
cluster.job_output_bufferSAIL_CLUSTER__JOB_OUTPUT_BUFFER16The number of batches to buffer in the job output stream.
cluster.task_launch_timeout_secsSAIL_CLUSTER__TASK_LAUNCH_TIMEOUT_SECS300The timeout in seconds for launching a task.
cluster.worker_external_hostSAIL_CLUSTER__WORKER_EXTERNAL_HOST127.0.0.1The external host for other workers to connect to the worker.
cluster.worker_external_portSAIL_CLUSTER__WORKER_EXTERNAL_PORT0The external port for other workers to connect to the worker.
If the value is 0, the port is assumed to be the same as the port
on which the worker listens.
cluster.worker_heartbeat_interval_secsSAIL_CLUSTER__WORKER_HEARTBEAT_INTERVAL_SECS30The interval in seconds for worker heartbeats.
cluster.worker_heartbeat_timeout_secsSAIL_CLUSTER__WORKER_HEARTBEAT_TIMEOUT_SECS120The timeout in seconds for worker heartbeats.
cluster.worker_initial_countSAIL_CLUSTER__WORKER_INITIAL_COUNT4The initial number of workers to launch.
cluster.worker_launch_timeout_secsSAIL_CLUSTER__WORKER_LAUNCH_TIMEOUT_SECS300The timeout in seconds for launching a worker.
cluster.worker_listen_hostSAIL_CLUSTER__WORKER_LISTEN_HOST127.0.0.1The host on which the worker listens.
cluster.worker_listen_portSAIL_CLUSTER__WORKER_LISTEN_PORT0The port on which the worker listens.
If the value is 0, a random port is assigned by the operating system.
cluster.worker_max_countSAIL_CLUSTER__WORKER_MAX_COUNT0The maximum number of workers that can be launched.
cluster.worker_max_idle_time_secsSAIL_CLUSTER__WORKER_MAX_IDLE_TIME_SECS120The maximum idle time in seconds before a worker is removed.
cluster.worker_stream_bufferSAIL_CLUSTER__WORKER_STREAM_BUFFER16The number of batches to buffer in the worker shuffle stream.
cluster.worker_task_slotsSAIL_CLUSTER__WORKER_TASK_SLOTS8The maximum number of tasks that can be launched on a worker.
Kubernetes Options
kubernetes.driver_pod_nameSAIL_KUBERNETES__DRIVER_POD_NAMEThe name of the pod that runs the driver, or empty if the driver pod name is not known. This is used to set owner references for worker pods.
kubernetes.imageSAIL_KUBERNETES__IMAGEsail:latestThe container image to use for the driver and worker pods.
kubernetes.image_pull_policySAIL_KUBERNETES__IMAGE_PULL_POLICYIfNotPresentThe image pull policy for the driver and worker pods.
kubernetes.namespaceSAIL_KUBERNETES__NAMESPACEdefaultThe Kubernetes namespace in which the driver and worker pods will be created.
kubernetes.worker_pod_name_prefixSAIL_KUBERNETES__WORKER_POD_NAME_PREFIXsail-worker-The prefix of the name of worker pods.
This should usually end with a hyphen (-).
kubernetes.worker_service_account_nameSAIL_KUBERNETES__WORKER_SERVICE_ACCOUNT_NAMEdefaultThe name of the service account to use for the worker pods.
Runtime Options
runtime.enable_secondarySAIL_RUNTIME__ENABLE_SECONDARYfalseWhether to enable a secondary Tokio runtime for separating I/O and compute tasks.
runtime.stack_sizeSAIL_RUNTIME__STACK_SIZE8388608The stack size in bytes for each thread.
Spark Options
spark.session_timeout_secsSAIL_SPARK__SESSION_TIMEOUT_SECS300The Spark session timeout in seconds.
Execution Options
execution.batch_sizeSAIL_EXECUTION__BATCH_SIZE16384The batch size for physical plan execution.
Parquet Options
parquet.allow_single_file_parallelismSAIL_PARQUET__ALLOW_SINGLE_FILE_PARALLELISMtrue(Writing) Whether to parallelize writing for each single Parquet file.
If the value is true, each column in each row group in each file are serialized in parallel.
parquet.binary_as_stringSAIL_PARQUET__BINARY_AS_STRINGfalse(Reading) Whether to read binary columns as string columns when reading Parquet files.
If the value is true, the Parquet reader will read columns of
the Binary or LargeBinary as the Utf8 type, and the BinaryView type as the Utf8View type.
This is helpful when reading Parquet files generated by some legacy writers, which do not correctly set
the UTF-8 flag for strings, causing string columns to be loaded as binary columns by default.
parquet.bloom_filter_fppSAIL_PARQUET__BLOOM_FILTER_FPP0.05(Writing) The false positive probability for bloom filters when writing Parquet files.
parquet.bloom_filter_ndvSAIL_PARQUET__BLOOM_FILTER_NDV1000000(Writing) The number of distinct values for bloom filters when writing Parquet files.
parquet.bloom_filter_on_readSAIL_PARQUET__BLOOM_FILTER_ON_READtrue(Writing) Whether to use available bloom filters when reading Parquet files.
parquet.bloom_filter_on_writeSAIL_PARQUET__BLOOM_FILTER_ON_WRITEfalse(Writing) Whether to write bloom filters for all columns when writing Parquet files.
parquet.compressionSAIL_PARQUET__COMPRESSIONzstd(3)(Writing) The default Parquet compression codec.
Valid values are uncompressed, snappy, gzip(level),
lzo, brotli(level), lz4, zstd(level), and lz4_raw,
where level is an integer defining the compression level.
These values are not case-sensitive.
parquet.data_page_row_count_limitSAIL_PARQUET__DATA_PAGE_ROW_COUNT_LIMIT20000(Writing) The best-effort maximum number of rows in data page for the Parquet writer.
parquet.dictionary_enabledSAIL_PARQUET__DICTIONARY_ENABLEDtrue(Writing) Whether to enable dictionary encoding for the Parquet writer.
parquet.dictionary_page_size_limitSAIL_PARQUET__DICTIONARY_PAGE_SIZE_LIMIT1048576(Writing) The best-effort maximum dictionary page size in bytes for the Parquet writer.
parquet.enable_page_indexSAIL_PARQUET__ENABLE_PAGE_INDEXtrue(Reading) Whether to enable page index when reading Parquet files.
If the value is true, the Parquet reader reads the page index if present.
This can reduce I/O and the number of rows decoded.
parquet.max_row_group_sizeSAIL_PARQUET__MAX_ROW_GROUP_SIZE1048576(Writing) The target maximum number of rows in each row group for the Parquet writer.
Larger row groups require more memory to write, but can get better compression and be faster to read.
parquet.maximum_buffered_record_batches_per_streamSAIL_PARQUET__MAXIMUM_BUFFERED_RECORD_BATCHES_PER_STREAM32(Writing) The maximum number of buffered record batches per stream for the Parquet writer.
This may improve performance when writing large Parquet files, at the expense of higher memory usage.
parquet.maximum_parallel_row_group_writersSAIL_PARQUET__MAXIMUM_PARALLEL_ROW_GROUP_WRITERS2(Writing) The maximum number of row group writers to use for the Parquet writer.
This may improve performance when writing large Parquet files, at the expense of higher memory usage.
parquet.metadata_size_hintSAIL_PARQUET__METADATA_SIZE_HINT0(Reading) The metadata size hint in bytes when reading Parquet files.
If the value n is greater than 8, the Parquet reader will try and fetch the last n
bytes of the Parquet file optimistically. Otherwise, two reads are performed to fetch the
metadata. The first read fetches the 8-byte Parquet footer and the second read fetches
the metadata length encoded in the footer.
parquet.pruningSAIL_PARQUET__PRUNINGtrue(Reading) Whether to prune row groups when reading Parquet files.
If the value is true, the Parquet reader attempts to skip entire row groups based
on the predicate in the query and the metadata (minimum and maximum values) stored in
the Parquet file.
parquet.pushdown_filtersSAIL_PARQUET__PUSHDOWN_FILTERSfalse(Reading) Whether to push down filter expressions when reading Parquet files.
If the value is true, the Parquet reader applies filter expressions in decoding operations to
reduce the number of rows decoded. This optimization is sometimes called "late materialization".
parquet.reorder_filtersSAIL_PARQUET__REORDER_FILTERSfalse(Reading) Whether to reorder filter expressions when reading Parquet files.
If the value is true, the Parquet reader reorders filter expressions heuristically in decoding operations to
minimize the cost of evaluation. If the value is false, the filters are applied in the same order as written in the query.
parquet.schema_force_view_typesSAIL_PARQUET__SCHEMA_FORCE_VIEW_TYPEStrue(Reading) Whether to force view types for binary and string columns when reading Parquet files.
If the value is true, the Parquet reader will read columns of the Utf8 or Utf8Large types as the Utf8View type,
and the Binary or BinaryLarge types as the BinaryView type.
parquet.skip_arrow_metadataSAIL_PARQUET__SKIP_ARROW_METADATAfalse(Writing) Whether to skip encoding the embedded arrow metadata when writing Parquet files.
parquet.skip_metadataSAIL_PARQUET__SKIP_METADATAtrue(Reading) Whether to skip the metadata when reading Parquet files.
If the value is true, the Parquet reader skip the optional embedded metadata that may be in
the file schema. This can help avoid schema conflicts when querying
multiple Parquet files with schemas containing compatible types but different metadata.
parquet.statistics_enabledSAIL_PARQUET__STATISTICS_ENABLEDpage(Writing) Whether statistics are enabled for any column for the Parquet writer.
Valid values are none, chunk, and page.
These values are not case-sensitive.
parquet.write_batch_sizeSAIL_PARQUET__WRITE_BATCH_SIZE1024(Writing) The Parquet writer batch size in bytes.