Supported Features
This page presents a high-level overview of supported Spark SQL features.
Literals
Sail supports all the Spark SQL literal syntax.
Data Types
Sail supports all Spark SQL data types except the VARIANT type introduced in Spark 4.0. Support for the VARIANT type is tracked in the GitHub issue.
Expressions
Sail supports most Spark SQL expression syntax, including unary and binary operators, predicates, CASE clause etc.
Sail supports most common Spark SQL functions. The effort to reach full function parity with Spark is tracked in the GitHub issue.
Statements
Data Retrieval (Query)
The following table lists the supported clauses in the SELECT statement.
| Clause | Supported |
|---|---|
FROM <relation> | ✅ |
FROM <format>.<path> (files) | 🚧 |
WHERE | ✅ |
GROUP BY | ✅ |
HAVING | ✅ |
ORDER BY | ✅ |
OFFSET | ✅ |
LIMIT | ✅ |
JOIN | ✅ |
UNION | ✅ |
INTERSECT | ✅ |
EXCEPT / MINUS | ✅ |
WITH (common table expressions) | ✅ |
VALUES (inline tables) | ✅ |
OVER <window> | ✅ |
/*+ ... */ (hints) | 🚧 |
CLUSTER BY | 🚧 |
DISTRIBUTE BY | 🚧 |
SORT BY | 🚧 |
PIVOT | 🚧 |
UNPIVOT | 🚧 |
LATERAL VIEW | ✅ |
LATERAL <subquery> | 🚧 |
TABLESAMPLE | 🚧 |
TRANSFORM | 🚧 |
The EXPLAIN statement is also supported, but the output shows the Sail logical and physical plan.
The DESCRIBE QUERY statement is not supported yet.
Data Manipulation
| Statement | Supported |
|---|---|
INSERT INTO <table> | ✅ |
INSERT OVERWRITE <table> | 🚧 |
INSERT OVERWRITE DIRECTORY <path> | 🚧 |
LOAD DATA | 🚧 |
COPY INTO | 🚧 |
MERGE INTO | 🚧 |
UPDATE | 🚧 |
DELETE FROM | 🚧 |
The COPY INTO, MERGE INTO, UPDATE, and DELETE FROM statements are not core Spark features. But some extensions support these statements for lakehouse tables (e.g., Delta Lake).
Catalog Management
| Statement | Supported |
|---|---|
ALTER DATABASE | 🚧 |
ALTER TABLE | 🚧 |
ALTER VIEW | 🚧 |
ANALYZE TABLE | 🚧 |
CREATE DATABASE | ✅ |
CREATE FUNCTION | 🚧 |
CREATE TABLE | ✅ (partial) |
CREATE TEMPORARY VIEW | ✅ |
CREATE VIEW | 🚧 |
DESCRIBE DATABASE | 🚧 |
DESCRIBE FUNCTION | 🚧 |
DESCRIBE TABLE | 🚧 |
DROP DATABASE | ✅ |
DROP FUNCTION | 🚧 |
DROP TABLE | ✅ |
DROP VIEW | ✅ |
REFRESH <path> | 🚧 |
REFRESH FUNCTION | 🚧 |
REFRESH TABLE | 🚧 |
REPAIR TABLE | 🚧 |
SHOW COLUMNS | ✅ |
SHOW CREATE TABLE | 🚧 |
SHOW DATABASES | ✅ |
SHOW FUNCTIONS | 🚧 |
SHOW PARTITIONS | 🚧 |
SHOW TABLE | 🚧 |
SHOW TABLES | ✅ |
SHOW TBLPROPERTIES | 🚧 |
SHOW VIEWS | ✅ |
TRUNCATE TABLE | 🚧 |
USE DATABASE | ✅ |
Currently, Sail only supports in-memory catalog, which means the databases and tables are available only within the session. Remote catalog support is in our roadmap.
For the CREATE TABLE statement, the CREATE TABLE ... AS ... syntax is not supported yet.
Configuration Management
| Statement | Supported |
|---|---|
RESET (reset options) | 🚧 |
SET (list or set options) | 🚧 |
Artifact Management
| Statement | Supported |
|---|---|
ADD FILE | ❌ |
ADD JAR | ❌ |
LIST FILE | ❌ |
LIST JAR | ❌ |
Cache Management
| Statement | Supported |
|---|---|
CACHE TABLE | 🚧 |
CLEAR CACHE | 🚧 |
UNCACHE TABLE | 🚧 |