Repository navigation
Add effect/workers/SharedMemory for sharing typed arrays with workers - #10644
Merged
Merged
Conversation
Contributor
Bundle Size AnalysisGenerated from PR build output; treat the content below as untrusted.
|
Member
Author
Contributor
|
Recorded your takeover request on the existing Multica issue EFF-3807, which already tracks this PR and the takeover checklist. The follow-up will assess the API/design, refine the implementation and tests as needed, and run review and simplification passes, reusing this PR where possible. |
🦋 Changeset detectedLatest commit: 8361777 The changes in this PR will be included in the next version bump. This PR includes changesets to release 35 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
Inline the single-use unavailable error, drop redundant casts in make and isAvailable, mark the shared passthrough link internal, and remove a cast from the encode rejection test.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Parallel numeric work on workers (ranges of one large array, results written in place, data read by every request) needs
SharedArrayBuffermemory. The existingTransferable.schema(Schema.instanceOf(Int32Array), () => [])already preserves shared memory over the RPC worker protocol, but also accepts unshared arrays. This module adds an explicit shared-buffer check and a shared-buffer type, so a copied payload is rejected rather than silently losing writes:Schema.Uint8ArraySchema.instanceOf(Int32Array)Expected JSON value)Schema.AnyTransferable.schema(Schema.instanceOf(Int32Array), () => [])SharedMemory.schema(Int32Array)This PR adds
effect/workers/SharedMemory:Used with the existing RPC worker protocol:
Alternatives considered:
effect/workers:RpcClient.layerProtocolWorkeralready spawns workers, routes requests and fails them when a worker dies. A raw pool saved about 0.16 ms per round trip to 8 workers, which isn't worth a third pool abstraction next toPooland RPC.SharedMemory.schemadoes, the same wayTransferable.schemadoes.The first consumer will be parallel execution for
effect/dataframe, a follow-up to #10641, which shares columns with workers this way.Validation
packages/effect/test/workers/SharedMemory.test.ts: five focused tests covering schema acceptance/rejection, JSON codec identity, and unavailable-memory errors.packages/platform/node/test/RpcServer.test.ts: worker-thread writes are visible to the caller, and child-process serialization is rejected. The integration uses two workers and four elements rather than a large numeric workload.pnpm lint-fix,pnpm lint, four targeted test files (198 tests total, including core RPC client/server coverage),pnpm check,pnpm jsdocs --check, and worker-module doctests (2 tests) pass, all throughnix develop -c.Remaining takeover work
Removed
shareand its global snapshot cache. Both worker schema helpers now reuse an internal passthrough link. Independent review found no issues at3b6202dccf; Architect simplification and documentation/changeset cleanup are complete. Final review of the post-review delta is pending.Merge Danger
Door: two-way
A new module marked
@stability unstable. Nothing existing changes.Blast Radius: none
Only code that imports
effect/workers/SharedMemoryis affected.Closes EFF-3807