AWS S3

You can use the s3:// URI to read and write data stored in AWS Simple Storage Service (S3). The URI can also refer to data stored in S3-compatible object storage services, such as MinIO or services provided by other vendors.

For example, s3://bucket/path/to/data refers to the path/to/data path in the bucket bucket. Sail determines whether the path refers to a single object or a key prefix. If the path turns out to be a key prefix, we assume the key prefix is followed by / and represents a directory.

AWS Credentials

All AWS credential providers work out-of-box in Sail. You can authenticate with AWS S3 using any of the supported methods, including AWS config and credentials files, EC2 instance profiles, environment variables, and container credentials.

Credential rotation happens automatically if you use temporary credentials.

INFO

You can refer to the AWS documentation for more details about the credential providers.

AWS Region

If the AWS region is configured, all S3 buckets must be in the same region. Otherwise, an error will be returned when accessing the data.

To allow inferring regions for S3 buckets and accessing S3 data in all regions, you can set the AWS_REGION environment variable to an empty string.

Accessing Public Data on AWS S3

Some datasets on S3 allow public access without an AWS account. You can skip retrieving AWS credentials by setting the environment variable AWS_SKIP_SIGNATURE=true.

INFO

AWS_SKIP_SIGNATURE is not a standard environment variable used by AWS SDKs. It is an environment variable recognized by Sail.