bytewax_redis.sources

Sources for Redis.

Classes

class RedisStreamSource(

stream_names: list[str],

host: str,

port: int,

db: int,

batch_size: int = 100,

)

Bases:

• FixedPartitionedSource

Read from a set of Redis Streams.

At-least-once possible if recovery enabled.

Initialization

Initialize the RedisStreamSource.

Parameters:

• host – Redis server hostname.

• port – Redis server port.

• db – Redis database index.

• stream_name – Name of the Redis stream to read from.

• batch_size – Number of messages to read in each batch.

list_parts() → list[str]

List all available partitions for the Redis stream source.

Returns:

A list of available partitions, in this case, each stream in self.stream_names makes a partition.

build_part(

step_id: str,

for_part: str,

resume_state: Optional[str],

) → bytewax_redis.sources._RedisStreamPartition

Build a partition for reading from the Redis stream.

Constructs and returns a partition for reading data from the Redis stream. It accepts a resume_state parameter, which is the message ID to resume reading from.

Parameters:

• step_id – The ID of the current dataflow step.

• for_part – The partition being built, always “singleton”.

• resume_state – The ID of the last processed message to resume from.

Returns:

A _RedisStreamPartition object for reading from the Redis stream.