Connect External S3 Data

An external data target is an S3 bucket or prefix you register with LakeSail. It can be in any AWS account, not only the one connected to LakeSail. You then grant it to each compute profile that needs it, and choose what that profile may do. Nothing is copied, and the data stays under the bucket owner's control.

To move the data into the network's workspace bucket instead, see Copy Data to the Workspace Bucket.

Prerequisites

• A LakeSail role with permission to manage cloud accounts, to register or remove a target. The standard organization Owner and Member roles include this permission.
• Permission to edit the compute profile, to grant it the target.
• The bucket owner: someone who can edit the bucket policy in the AWS account that owns the bucket.
• Objects that are not encrypted with SSE-KMS, which is not supported yet.

Register a target

1. Open Settings → External Data and click Add External Data.
2. Fill in:
   • AWS connection: the cloud account whose clusters will read the data. Only compute profiles on this cloud account's clusters can be granted the target.
   • Name: how the target shows up when you grant it.
   • AWS account ID: the 12-digit ID of the account that owns the bucket.
   • S3 location: the bucket, for example s3://analytics-data, or a prefix inside it, for example s3://analytics-data/warehouse/production/.
   • AWS region: the bucket's region. A bucket outside the cluster's region needs an https:// path that names its region.
   • Maximum access: the most any compute profile may receive. You choose a subset of these when you grant the target.
3. Click Add target.

LakeSail then offers optional connection-wide bucket policy statements. Read Choose bucket policy statements before you add them, then click Continue to compute profiles.

Grant the target to a compute profile

Stop the profile's notebooks and sessions, and let its job runs finish, before you start. LakeSail does not change grants while a workload using the profile is running.

1. Open Compute → Profiles, select the profile, and open its Grants tab.
2. Click Grant access and choose the target.
3. Under Access, select what this profile needs. To read a prefix as a directory, select both List objects and Read objects.
4. Click Grant access.

The grant moves Configuring → No test result while LakeSail sets up the profile's IAM role. See How a grant changes the profile's IAM role.

Authorize the profile in the bucket policy

The bucket owner must allow the profile's IAM role in the bucket policy. Skip this step if the target's connection-wide statements are already there.

1. On the grant, click View profile policy and copy the statements.
2. Send them to the bucket owner.
3. The bucket owner adds them to the bucket policy, keeping the statements already in it.

Test access

Click Test access on the grant to check each access it has, using the profile's IAM role. A pass covers only the objects the test touched, so also run a real workload against your tables.

The test uses the bucket's own region. It can pass for a bucket in another region while s3:// paths in workloads fail. See Bucket in another region.

Testing Write objects without Delete objects leaves a test object behind. The result shows it with Copy object URI, so you can delete it from S3.

Read the data

Read external data through Sail or PySpark. In a notebook, boto3, pandas, and other direct Python clients do not get the profile's access.

End a prefix path with /, or the read can fail with Access denied:

df = spark.read.parquet("s3://analytics-data/warehouse/production/")

Catalog table and partition locations must also sit inside the registered prefix.

Bucket in another region

Workloads run in the region of the compute profile's cluster. If the bucket is in a different region, s3:// paths to it fail with "Received redirect without LOCATION, this normally indicates an incorrectly configured region". Use an https:// path that names the bucket's region instead. Reading across regions also adds AWS transfer costs and latency.

	Path
Instead of	s3://analytics-data/warehouse/production/
Use	https://analytics-data.s3.ap-south-1.amazonaws.com/warehouse/production/

The pattern is https://<bucket>.s3.<bucket-region>.amazonaws.com/<prefix>/. Keep the trailing /.

Use the same path to read and write:

path = "https://analytics-data.s3.ap-south-1.amazonaws.com/warehouse/production/"
df = spark.read.parquet(path)
df.write.mode("append").parquet(path)

Two cases cannot be read from another region yet: catalog tables, because the catalog stores s3:// locations, and buckets whose names contain dots. To use them, copy the data to the workspace bucket.

Grant states

Status	Meaning
Configuring	LakeSail is updating the profile's IAM role
No test result	The grant is active and has not been tested yet
Testing access	An access test is running
Last test passed	Each access worked on the objects tested
Last test incomplete	Some access could not be checked; read the result for details
Last test failed	At least one access check failed; see Troubleshooting
Update failed	The IAM change did not apply; read the message, then click Retry configuration
Removing access	LakeSail is removing the grant
Removal incomplete	The removal did not finish; read the message, then click Remove grant again

Choose bucket policy statements

LakeSail generates two kinds of bucket policy statement. The bucket owner adds one kind or the other, depending on how much they want to trust.

• Profile-specific statements name one compute profile's IAM role. Use them when the bucket owner wants to approve each profile separately. You add statements for every profile you grant, and remove them when you remove the grant.
• Connection-wide statements allow any IAM role in the compute account that follows LakeSail's naming pattern for profile roles under this cloud account. Use them when the bucket owner is comfortable trusting every profile on the cloud account, so later profiles need no bucket policy change. Each profile still needs its own grant in LakeSail.

How a grant changes the profile's IAM role

The first grant gives the profile its own IAM role, which replaces the cluster's job runner role for its workloads. It keeps workspace bucket and Glue access, but permissions you added to the job runner role do not carry over. Removing the last grant deletes the role.

Profiles on one cluster can reach different data this way, but they share the cluster's nodes and network. Use a separate cluster to separate those too.

Policy size limit

All grants on a profile share one IAM managed policy, which AWS caps at 6,144 characters. That fits about ten targets, fewer with long bucket names and prefixes. LakeSail refuses a change that would go over; see Troubleshooting.

Remove access

• Remove one profile's access: click Remove grant on the profile's Grants tab. LakeSail shows the profile-specific statements to take out of the bucket policy, if you added them. Removing a profile's last grant also deletes its IAM role.
• Remove a target: remove all of its grants first, then click Remove on the target's row under Settings → External Data. LakeSail does not touch the bucket or its bucket policy, so the bucket owner removes any statements they added.

API reference

• Data targets: register, list, and remove external data targets, and grant, test, and revoke them on compute profiles.