Skip to content

Add effect/workers/SharedMemory for sharing typed arrays with workers - #10644

Merged
tim-smart merged 6 commits into
mainfrom
feat/workers-shared-memory
Nov 16, 2026
Merged

tim-smart merged 6 commits into
mainfrom
feat/workers-shared-memory

Conversation

@fubhy

@fubhy fubhy commented Nov 14, 2026 •

Copy link
Copy Markdown
Member

Summary

Parallel numeric work on workers (ranges of one large array, results written in place, data read by every request) needs SharedArrayBuffer memory. The existing Transferable.schema(Schema.instanceOf(Int32Array), () => []) already preserves shared memory over the RPC worker protocol, but also accepts unshared arrays. This module adds an explicit shared-buffer check and a shared-buffer type, so a copied payload is rejected rather than silently losing writes:

Payload field What the worker gets
Schema.Uint8Array a base64 copy, so its writes are lost silently
Schema.instanceOf(Int32Array) encoding fails (Expected JSON value)
Schema.Any the shared memory, but untyped and unchecked
Transferable.schema(Schema.instanceOf(Int32Array), () => []) shared memory when supplied, but also accepts unshared arrays
SharedMemory.schema(Int32Array) validates both the array kind and shared backing memory

This PR adds effect/workers/SharedMemory:

SharedMemory.isAvailable()            // false on browser pages without cross-origin isolation
SharedMemory.isShared(view)
SharedMemory.make(Int32Array, n)      // Int32Array<SharedArrayBuffer>, zeroed
SharedMemory.schema(Float64Array)     // payload field: passes worker codecs untouched,
                                      // fails to decode unless it arrived shared
SharedMemory.SharedMemoryError        // thrown by make when unavailable

Used with the existing RPC worker protocol:

const Fill = Rpc.make("Fill", {
  payload: { data: SharedMemory.schema(Int32Array), from: Schema.Number, to: Schema.Number }
})
// main thread: the workers' writes land in `data`
const data = SharedMemory.make(Int32Array, 1_000_000)
yield* Effect.forEach(ranges, ([from, to]) => client.Fill({ data, from, to }), { concurrency: "unbounded" })

Alternatives considered:

  • A new worker pool in effect/workers: RpcClient.layerProtocolWorker already spawns workers, routes requests and fails them when a worker dies. A raw pool saved about 0.16 ms per round trip to 8 workers, which isn't worth a third pool abstraction next to Pool and RPC.
  • A new codec for the worker protocol (or SchemaBinary): any codec that serialises to bytes copies the array, so sharing is lost. Only a field that skips encoding keeps it shared, which is what SharedMemory.schema does, the same way Transferable.schema does.
  • Transfers only: a transferred buffer goes to one receiver and is detached from the sender. It can't give several workers the same input or let them write into one output.

The first consumer will be parallel execution for effect/dataframe, a follow-up to #10641, which shares columns with workers this way.

Validation

  • packages/effect/test/workers/SharedMemory.test.ts: five focused tests covering schema acceptance/rejection, JSON codec identity, and unavailable-memory errors.
  • packages/platform/node/test/RpcServer.test.ts: worker-thread writes are visible to the caller, and child-process serialization is rejected. The integration uses two workers and four elements rather than a large numeric workload.
  • Local validation after production refactoring: pnpm lint-fix, pnpm lint, four targeted test files (198 tests total, including core RPC client/server coverage), pnpm check, pnpm jsdocs --check, and worker-module doctests (2 tests) pass, all through nix develop -c.

Remaining takeover work

Removed share and its global snapshot cache. Both worker schema helpers now reuse an internal passthrough link. Independent review found no issues at 3b6202dccf; Architect simplification and documentation/changeset cleanup are complete. Final review of the post-review delta is pending.

Merge Danger

Door: two-way

A new module marked @stability unstable. Nothing existing changes.

Blast Radius: none

Only code that imports effect/workers/SharedMemory is affected.

Closes EFF-3807

@effect-janitor effect-janitor Bot added the 4.0 label Nov 14, 2026
@github-actions

github-actions Bot commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

Bundle Size Analysis

Generated from PR build output; treat the content below as untrusted.

File Name Current Size Previous Size Difference
arbitrary-combinators.ts 40.09 KB 40.09 KB 0.00 KB (0.00%)
basic.ts 7.31 KB 7.31 KB 0.00 KB (0.00%)
batching.ts 11.21 KB 11.21 KB 0.00 KB (0.00%)
brand.ts 7.00 KB 7.00 KB 0.00 KB (0.00%)
cache.ts 11.73 KB 11.73 KB 0.00 KB (0.00%)
config.ts 23.09 KB 23.09 KB 0.00 KB (0.00%)
differ.ts 22.20 KB 22.20 KB 0.00 KB (0.00%)
http-client.ts 27.45 KB 27.45 KB 0.00 KB (0.00%)
http-router.ts 37.59 KB 37.59 KB 0.00 KB (0.00%)
logger.ts 12.19 KB 12.19 KB 0.00 KB (0.00%)
metric.ts 9.65 KB 9.65 KB 0.00 KB (0.00%)
optic.ts 7.21 KB 7.21 KB 0.00 KB (0.00%)
pubsub.ts 16.21 KB 16.21 KB 0.00 KB (0.00%)
queue.ts 13.09 KB 13.09 KB 0.00 KB (0.00%)
schedule.ts 11.99 KB 11.99 KB 0.00 KB (0.00%)
schema-bigdecimal.ts 14.29 KB 14.29 KB 0.00 KB (0.00%)
schema-binary.ts 42.45 KB 42.45 KB 0.00 KB (0.00%)
schema-class.ts 21.96 KB 21.96 KB 0.00 KB (0.00%)
schema-codegen.ts 30.70 KB 30.70 KB 0.00 KB (0.00%)
schema-fromJsonSchemaDocument.ts 30.56 KB 30.56 KB 0.00 KB (0.00%)
schema-representation-roundtrip.ts 28.26 KB 28.26 KB 0.00 KB (0.00%)
schema-string-transformation.ts 15.54 KB 15.54 KB 0.00 KB (0.00%)
schema-string.ts 12.50 KB 12.50 KB 0.00 KB (0.00%)
schema-template-literal.ts 16.43 KB 16.43 KB 0.00 KB (0.00%)
schema-toArbitrary.ts 39.63 KB 39.63 KB 0.00 KB (0.00%)
schema-toCodecJson.ts 21.11 KB 21.11 KB 0.00 KB (0.00%)
schema-toEquivalence.ts 21.28 KB 21.28 KB 0.00 KB (0.00%)
schema-toFormatter.ts 21.37 KB 21.37 KB 0.00 KB (0.00%)
schema-toJsonSchemaDocument.ts 26.42 KB 26.42 KB 0.00 KB (0.00%)
schema-toRepresentation.ts 21.45 KB 21.45 KB 0.00 KB (0.00%)
schema.ts 21.05 KB 21.05 KB 0.00 KB (0.00%)
stm.ts 14.46 KB 14.46 KB 0.00 KB (0.00%)
stream.ts 10.56 KB 10.56 KB 0.00 KB (0.00%)

@fubhy

fubhy commented Nov 14, 2026

Copy link
Copy Markdown
Member Author

@effect-bot

@effect-bot

Copy link
Copy Markdown
Contributor

Recorded your takeover request on the existing Multica issue EFF-3807, which already tracks this PR and the takeover checklist. The follow-up will assess the API/design, refine the implementation and tests as needed, and run review and simplification passes, reusing this PR where possible.

@changeset-bot

changeset-bot Bot commented Nov 14, 2026 •

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 8361777

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 35 packages
Name Type
effect Patch
@effect/opentelemetry Patch
@effect/vitest Patch
@effect/ai-amazon-bedrock Patch
@effect/ai-anthropic Patch
@effect/ai-cloudflare Patch
@effect/ai-openai-compat Patch
@effect/ai-openai Patch
@effect/ai-openrouter Patch
@effect/ai-typesafe Patch
@effect/atom-react Patch
@effect/atom-solid Patch
@effect/atom-vue Patch
@effect/platform-browser Patch
@effect/platform-bun Patch
@effect/platform-cloudflare Patch
@effect/platform-deno Patch
@effect/platform-node-shared Patch
@effect/platform-node Patch
@effect/sql-clickhouse Patch
@effect/sql-d1 Patch
@effect/sql-libsql Patch
@effect/sql-mssql Patch
@effect/sql-mysql2 Patch
@effect/sql-pg Patch
@effect/sql-pglite Patch
@effect/sql-sqlite-bun Patch
@effect/sql-sqlite-do Patch
@effect/sql-sqlite-node Patch
@effect/sql-sqlite-react-native Patch
@effect/sql-sqlite-wasm Patch
@effect/docgen Patch
@effect/doctest Patch
@effect/graphql-generator Patch
@effect/openapi-generator Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

Inline the single-use unavailable error, drop redundant casts in make and isAvailable, mark the shared passthrough link internal, and remove a cast from the encode rejection test.
@tim-smart
tim-smart enabled auto-merge (squash) November 16, 2026 01:16
@tim-smart
tim-smart merged commit 9f00fd5 into main Nov 16, 2026
18 checks passed
@tim-smart
tim-smart deleted the feat/workers-shared-memory branch November 16, 2026 01:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants