Notifications

Notifications are how LakeSail tells you something happened — most commonly that a job run failed. The system has two parts: channels (where a notification goes) and rules (what triggers one). You configure them independently and connect them.

Prerequisites

• Permission to manage organization notifications (typically an admin role). Members can usually be the target of a notification rule but can't create or modify org-wide rules and channels.
• For rules: at least one channel must exist before you can save a rule that delivers to it. Create the channel first.
• For Slack channels: an incoming webhook URL on your Slack workspace.
• For PagerDuty / Rootly channels: an integration key / token from the respective service.

Channels

A channel is a delivery destination. LakeSail supports five types.

Type	Config	Best for
email	Target member or team (all members of that team receive it)	Individual alerts, digest delivery
slack	Incoming webhook URL	Team channels
webhook	URL + optional signing secret	Custom integrations, automation
pager_duty		Paging on-call during business-critical failures
rootly		Incident management workflows

Create a channel

1. Open Settings → Notifications → Channels .
2. Click Create channel.
3. Pick the Type and fill in the config:
   • Email — pick either a member or a team (not both). Team channels deliver to every current member.
   • Slack — paste the incoming webhook URL (e.g. https://hooks.slack.com/services/...).
   • Webhook — paste the target URL. Optionally provide a signing secret; LakeSail signs each request so your endpoint can verify authenticity.
   • PagerDuty / Rootly — provide the integration key / token as prompted.
4. Click Test to send a test payload. Confirm it lands where you expect.
5. Save.

Webhook signing

If you set a signing secret on a webhook channel, every delivery includes a LakeSail-Signature header of the form sha256=<hex digest>. Compute HMAC-SHA256 of the raw request body with your secret and compare — reject anything that doesn't match.

Rotate the secret periodically by editing the channel; the new secret takes effect on the next delivery.

Rules

A rule connects events to channels. It answers three questions:

1. What fired? — the event type. Example: job_run.status.failed.
2. Where did it fire? — the scope.
3. Who hears about it? — the channels.

Event types

Events are keyed <resource>.<property>.<value>, e.g.:

• job_run.status.failed
• job_run.status.succeeded
• job_run.status.timeout
• job_run.status.cancelled

Browse the full catalog from the rule creation screen — LakeSail lists every event type it knows how to emit.

Scope

Scope determines which resources a rule listens to:

• resource — one specific resource (e.g. "notify me when this one job fails").
• team — every resource owned by a team.
• organization — every resource in the org.
• own_resources — every resource owned by the current member.

Scope plus event type plus channel is the full rule: "when job_run.status.failed fires on anything owned by the data-eng team, deliver to the #data-alerts Slack channel."

Delivery mode

• immediate — send as events fire. Right for failures.
• daily_digest — batch into one email/message per day. Right for noisy events like successes.
• weekly_digest — same, weekly.

Digests are per-rule, so you can have immediate delivery for failures and a weekly digest for successes on the same channel.

Create a rule

1. Open Settings → Notifications → Rules .
2. Click Create rule.
3. Fill in:
   • Name — e.g. data-eng prod job failures.
   • Resource type — what kind of resource the rule applies to (e.g. job_run).
   • Event types — one or more keys from the catalog.
   • Scope — resource / team / organization / own_resources (plus the specific ID when narrower).
   • Channels — one or more channels to deliver to.
   • Delivery mode — immediate / daily_digest / weekly_digest.
   • Enabled — toggle to turn the rule on.
4. Save.

The rule fires starting with the next qualifying event. It does not backfill past events.

Recommended setups

Small team, one shared alerts channel:

• One Slack channel.
• One rule: job_run.status.failed + job_run.status.timeout, scope organization, delivery immediate.
• Optional: a second rule with job_run.status.succeeded, scope organization, delivery daily_digest, same channel.

Per-team ownership:

• One Slack channel per team (e.g. #data-eng-alerts, #analytics-alerts).
• One rule per team: failures, scope team, delivery immediate, scoped to the owning team.

On-call paging:

• A PagerDuty channel wired to your on-call rotation.
• A rule scoped to a small set of production jobs (scope resource, one rule per job), event job_run.status.failed, delivery immediate.
• Keep the scope narrow — paging everyone on every failure burns trust fast.

Troubleshooting

• Test delivery works but real events don't arrive — check that the rule is enabled and that the event type matches exactly. job_run.status.fail won't match job_run.status.failed.
• Slack webhooks silently drop — Slack returns 200 even when the channel is archived or the webhook is disabled. Post a test message from the Slack admin UI to confirm the webhook is live.
• Webhook signature mismatch — make sure you're hashing the raw body (before any JSON parsing or whitespace normalization) and comparing against the LakeSail-Signature header's sha256= value.
• Digest never arrives — digests send only if events accumulated in the window. Quiet periods produce no digest.

API reference

• Notifications — channel and rule CRUD, plus TestNotificationChannel and ListNotificationEventTypes for the event-key catalog.