DataStreamWriter

Defined in: streaming/data-stream-writer.ts:30

Fluent writer for a streaming DataFrame, obtained via df.writeStream. Configure the output format, options, output mode, and trigger; terminate with start(path?) or toTable(name) to launch the query on the server.

foreach / foreachBatch are not yet supported because they require JS UDF execution, which isn’t shipped.

Example

const query = await df.writeStream
  .format("memory")
  .queryName("inflight")
  .outputMode("append")
  .trigger(Trigger.processingTime("5 seconds"))
  .start();
await query.processAllAvailable();
await query.stop();

See

Spark source: DataStreamWriter.scala

Constructors

Constructor

new DataStreamWriter(df): DataStreamWriter;

Defined in: streaming/data-stream-writer.ts:39

Parameters

Parameter	Type
df	DataFrame

Returns

DataStreamWriter

Methods

format()

format(source): this;

Defined in: streaming/data-stream-writer.ts:44

Set the sink format (e.g. "memory", "console", "parquet", "kafka").

Parameters

Parameter	Type
source	string

Returns

this

---

option()

option(key, value): this;

Defined in: streaming/data-stream-writer.ts:50

Set a single sink option. Booleans and numbers are stringified.

Parameters

Parameter	Type
key	string
value	string | number | boolean

Returns

this

---

options()

options(opts): this;

Defined in: streaming/data-stream-writer.ts:56

Set multiple sink options at once. Non-string values are stringified.

Parameters

Parameter	Type
opts	Record<string, string | number | boolean>

Returns

this

---

outputMode()

outputMode(mode): this;

Defined in: streaming/data-stream-writer.ts:66

How the streaming query writes results into the sink.

Not every sink supports every mode; see Spark’s docs for the matrix.

Parameters

Parameter	Type
mode	StreamingOutputMode

Returns

this

---

partitionBy()

partitionBy(...columns): this;

Defined in: streaming/data-stream-writer.ts:87

Partition the sink output by the given column names.

Parameters

Parameter	Type
…columns	string[]

Returns

this

---

queryName()

queryName(name): this;

Defined in: streaming/data-stream-writer.ts:72

Name the query so it can be looked up via spark.streams.get(name).

Parameters

Parameter	Type
name	string

Returns

this

---

start()

start(path?): Promise<StreamingQuery>;

Defined in: streaming/data-stream-writer.ts:97

Start the streaming query against the given path (file sinks) or no path (for memory, console, kafka, etc.). Returns a handle for inspecting and controlling the running query.

Parameters

Parameter	Type
path?	string

Returns

Promise<StreamingQuery>

---

toTable()

toTable(tableName): Promise<StreamingQuery>;

Defined in: streaming/data-stream-writer.ts:110

Start the streaming query and write output to the named table. The table is created if it doesn’t exist (sink semantics depend on format).

Parameters

Parameter	Type
tableName	string

Returns

Promise<StreamingQuery>

---

trigger()

trigger(t): this;

Defined in: streaming/data-stream-writer.ts:81

Configure the trigger policy. See Trigger for the variants.

Parameters

Parameter	Type
t	Trigger

Returns

this