WatermarkPlan
Defined in: plan/logical-plan.ts:652
Attach an event-time watermark to a streaming DataFrame. Bounds late data for stateful streaming operators (windowed aggregations, joins, dropDuplicates).
- Spark Connect: Relation.WithWatermark { input, event_time, delay_threshold }
- Catalyst: EventTimeWatermark(eventTime, delayThreshold, child)
Properties
Section titled “Properties”child: LogicalPlan;Defined in: plan/logical-plan.ts:654
delayThreshold
Section titled “delayThreshold”delayThreshold: string;Defined in: plan/logical-plan.ts:658
Interval string such as "10 minutes" or "1 hour".
eventTime
Section titled “eventTime”eventTime: string;Defined in: plan/logical-plan.ts:656
Column name carrying the event-time timestamp.
type: "watermark";Defined in: plan/logical-plan.ts:653