StreamingQueryProgress
Defined in: streaming/types.ts:43
One progress report from a streaming query. Spark Connect sends progress as a JSON string. This is the parsed shape.
The named fields below cover the values Spark sets on nearly every report.
The index signature lets newer Spark versions or third-party
StreamingQueryListener events add fields without breaking compilation.
Indexable
Section titled “Indexable”[key: string]: unknownForward-compat for fields Spark adds in newer versions.
Properties
Section titled “Properties”batchDuration?
Section titled “batchDuration?”optional batchDuration?: number;Defined in: streaming/types.ts:55
End-to-end duration of the batch in milliseconds.
batchId?
Section titled “batchId?”optional batchId?: number;Defined in: streaming/types.ts:53
Monotonically increasing batch number within this run.
durationMs?
Section titled “durationMs?”optional durationMs?: Record<string, number>;Defined in: streaming/types.ts:57
Per-phase durations (e.g. addBatch, getOffset, triggerExecution).
optional id?: string;Defined in: streaming/types.ts:45
Spark-internal query identifier (the StreamingQuery.id).
inputRowsPerSecond?
Section titled “inputRowsPerSecond?”optional inputRowsPerSecond?: number;Defined in: streaming/types.ts:61
Average input rate over this batch.
optional name?: string;Defined in: streaming/types.ts:49
Optional query name supplied via writeStream.queryName(...).
numInputRows?
Section titled “numInputRows?”optional numInputRows?: number | bigint;Defined in: streaming/types.ts:59
Total input rows processed in this batch. Long on the server.
processedRowsPerSecond?
Section titled “processedRowsPerSecond?”optional processedRowsPerSecond?: number;Defined in: streaming/types.ts:63
Average processed rate over this batch.
runId?
Section titled “runId?”optional runId?: string;Defined in: streaming/types.ts:47
Run identifier for the current execution attempt.
optional sink?: unknown;Defined in: streaming/types.ts:69
Sink description.
sources?
Section titled “sources?”optional sources?: unknown[];Defined in: streaming/types.ts:67
Per-source progress (offsets, rows, rates).
stateOperators?
Section titled “stateOperators?”optional stateOperators?: unknown[];Defined in: streaming/types.ts:65
Per-state-operator metrics.
timestamp?
Section titled “timestamp?”optional timestamp?: string;Defined in: streaming/types.ts:51
ISO-8601 timestamp the batch was emitted at.