DataStreamReader

Defined in: streaming/data-stream-reader.ts:26

Fluent reader for a streaming source, obtained via spark.readStream. Configure the format, options, and schema, then terminate with load(path?), table(tableName), or a format shortcut (csv, json, parquet, orc, text). The returned DataFrame carries isStreaming: true on its Read plan.

Example

spark.readStream
  .format("rate")
  .option("rowsPerSecond", "5")
  .load();

spark.readStream
  .schema("id INT, name STRING")
  .json("/data/incoming/");

See

Spark source: DataStreamReader.scala

Constructors

Constructor

new DataStreamReader(session): DataStreamReader;

Defined in: streaming/data-stream-reader.ts:32

Parameters

Parameter	Type
session	SparkSession

Returns

DataStreamReader

Methods

csv()

csv(path): DataFrame;

Defined in: streaming/data-stream-reader.ts:117

Shortcut for .format("csv").load(path).

Parameters

Parameter	Type
path	string

Returns

DataFrame

---

format()

format(source): this;

Defined in: streaming/data-stream-reader.ts:61

Set the source format (e.g. "rate", "kafka", "parquet").

Parameters

Parameter	Type
source	string

Returns

this

---

json()

json(path): DataFrame;

Defined in: streaming/data-stream-reader.ts:112

Shortcut for .format("json").load(path).

Parameters

Parameter	Type
path	string

Returns

DataFrame

---

load()

load(path?): DataFrame;

Defined in: streaming/data-stream-reader.ts:82

Build the streaming Read plan. path is optional because sources like rate, kafka, and custom connectors don’t take one.

Parameters

Parameter	Type
path?	string

Returns

DataFrame

---

option()

option(key, value): this;

Defined in: streaming/data-stream-reader.ts:67

Set a single source option. Booleans and numbers are stringified.

Parameters

Parameter	Type
key	string
value	string | number | boolean

Returns

this

---

options()

options(opts): this;

Defined in: streaming/data-stream-reader.ts:73

Set multiple source options at once. Non-string values are stringified.

Parameters

Parameter	Type
opts	Record<string, string | number | boolean>

Returns

this

---

orc()

orc(path): DataFrame;

Defined in: streaming/data-stream-reader.ts:127

Shortcut for .format("orc").load(path).

Parameters

Parameter	Type
path	string

Returns

DataFrame

---

parquet()

parquet(path): DataFrame;

Defined in: streaming/data-stream-reader.ts:122

Shortcut for .format("parquet").load(path).

Parameters

Parameter	Type
path	string

Returns

DataFrame

---

schema()

schema(schema): this;

Defined in: streaming/data-stream-reader.ts:41

Set the schema for the data source. Accepts a DDL-formatted string ("id INT, name STRING") or a StructType-like object with a toDDL() method.

Parameters

Parameter	Type
schema	| string | { toDDL: string; }

Returns

this

---

table()

table(tableName): DataFrame;

Defined in: streaming/data-stream-reader.ts:99

Read a streaming named table (catalog table or temp view).

Parameters

Parameter	Type
tableName	string

Returns

DataFrame

---

text()

text(path): DataFrame;

Defined in: streaming/data-stream-reader.ts:132

Shortcut for .format("text").load(path).

Parameters

Parameter	Type
path	string

Returns

DataFrame