Effect.ts: Absence as First-Class

Streams • Lesson 18 of 51

18. Stream Chunking - Batching Absent Values

HOW to group stream elements into batches

Code Example

// Process items in chunks of 100
const stream = Stream.fromIterable(allItems);

const chunked = stream.pipe(
  Stream.rechunk(100), // Group into chunks of 100
  Stream.mapEffect((chunk) =>
    // Process whole chunk at once
    batchInsert(chunk)
  )
);

// Why chunks?
// - Database batch inserts
// - Network request batching
// - Memory-efficient processing

Interactive Example

const { Stream, Chunk } = await import('effect');
const items = Stream.fromIterable([1,2,3,4,5,6,7,8,9,10]);
let chunkCount = 0;
const chunked = items.pipe(
      Stream.rechunk(3),
      Stream.tap(() => Effect.sync(() => { chunkCount++; }))
);
await Effect.runPromise(Stream.runDrain(chunked));
return `Stream chunking!

10 items
Chunk size: 3
Chunks processed: ${chunkCount}

Batched processing!`;

Explanation

Chunking: another HOW for resolving sequential absence.

Traditional approach: process items one by one. Stream types batched unavailability:

1000 items are unavailable
Don't assume we get them all at once
Describe: "Resolve absence in groups of 100"

Absence-first batching: control HOW sequential absence is resolved!

Part 18 of 51 in the Effect.ts Absence Modeling series

←→