Skip to content

[fix][ml] Fix cached read handle leaks when closing read-only managed ledgers - #26816

Open
void-ptr974 wants to merge 9 commits into
apache:masterfrom
void-ptr974:fix/readonly-ledger-handle-cleanup
Open

void-ptr974 wants to merge 9 commits into
apache:masterfrom
void-ptr974:fix/readonly-ledger-handle-cleanup

Conversation

@void-ptr974

@void-ptr974 void-ptr974 commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

Main Issue: #26815

Motivation

Read-only managed ledgers have no current write handle, so closing them does not reach the cached read-handle cleanup in the parent's write-close callback. The factory wrapper also lacks a close operation, leaving snapshot recovery without a way to release its cached handles.

Modifications

  • Drain the read-only ledger's cached handles through the existing invalidation mechanism when close starts. Mark close before draining, reject subsequent handle acquisition, and clean up an open that races with close, including pending opens.
  • Add a backward-compatible default ReadOnlyManagedLedger.asyncClose() and delegate it through the factory wrapper.
  • Close the read-only ledger in snapshot recovery's finally block on success, failure, and early return. Report cleanup failure without replacing the recovery result.
  • Preserve master's expectedReadCount overload when resolving the merge conflict and cover its forwarding with a regression test.

The close future represents logical closure; cleanup of handles whose opens are still pending may finish asynchronously.

Verifying this change

  • Make sure that the change passes the CI checks.

Local validation with test retries disabled:

  • 9 read-only handle-close cases covering normal and fenced close, repeated close, pending opens, an open racing close, close failure, the factory wrapper, and expected read counts.
  • 10 existing read-only cursor and managed-ledger tests.
  • 3 replay-entry regression tests from master.
  • 1 snapshot recovery close test.
  • ./gradlew quickCheck.

Does this pull request potentially affect one of the following parts:

  • Dependencies (add or upgrade a dependency)
  • The public API
  • The schema
  • The default values of configurations
  • The threading model
  • The binary protocol
  • The REST endpoints
  • The admin CLI options
  • The metrics
  • Anything that affects deployment

Release the read-only handle cache on close, including fenced closes and handles whose opens are pending, without changing writable managed-ledger shutdown.

Main Issue: apache#26815

Assisted-by: OpenAI Codex
@void-ptr974 void-ptr974 changed the title [fix][ml] Close cached handles when a read-only managed ledger closes [fix][ml] Fix cached read handle leaks when closing read-only managed ledgers Oct 2, 2026
Guard cache insertion against concurrent close and cover reads after normal or fenced close, repeated close, and independent cleanup failures.

Assisted-by: OpenAI Codex
Comment on lines +164 to +168
public synchronized void asyncClose(AsyncCallbacks.CloseCallback callback, Object ctx) {
closingOrClosed = true;
// There is no current write handle, so the parent's write-close callback cannot release this cache.
ledgerCache.forEach((ledgerId, readHandle) -> invalidateReadHandle(ledgerId));
super.asyncClose(callback, ctx);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think there's still a lifecycle gap in the actual factory path.

ManagedLedgerFactoryImpl.asyncOpenReadOnlyManagedLedger() returns a ReadOnlyManagedLedgerImplWrapper, but neither the public ReadOnlyManagedLedger interface nor the wrapper exposes a close method.

The read-only cursor path looks similar: ReadOnlyCursorImpl.asyncClose() only closes the cursor, not the backing managed ledger.

So these callers never reach this new asyncClose() implementation, and the cached handles can still be left open. One example is SnapshotSegmentAbortedTxnProcessorImpl#readSegmentEntries().

Could we also wire up the ownership/close path for the factory-facing APIs? A regression test through the factory API would help catch this.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for pointing this out. Addressed in 401ff9c67fd: ReadOnlyManagedLedger now exposes a backward-compatible default asyncClose(), and ReadOnlyManagedLedgerImplWrapper delegates it to the underlying implementation. Once the ledger has been acquired, SnapshotSegmentAbortedTxnProcessorImpl initiates the close in a finally block on every exit path, while logging close failures without masking the recovery result. I also added regression coverage through both the factory API and the production caller.

I kept the factory cursor path separate because a ReadOnlyCursor can also be created from a caller-owned ReadOnlyManagedLedger, in which case the cursor does not necessarily own the backing ledger. Safely changing openReadOnlyCursor would require an ownership-aware wrapper and separate lifecycle coverage. There are currently no in-tree production callers of that factory cursor API; I can address it separately if maintainers would like.

}

@Test
public void testCloseReleasesHandleWhoseOpenIsPending() throws Exception {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice coverage for a pending BookKeeper open.

One thing I noticed: the future is already in ledgerCache before close() runs, so this tests cleanup of an existing pending open, but not the race that the second closingOrClosed check is meant to handle.

Could we add a small targeted test for a cache insertion racing with close? I'd mainly like to make sure a late insertion can't leave a handle behind.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. 401ff9c67fd adds testCloseRejectsHandleInsertedAfterCloseStarts, which pauses the BookKeeper open before cache insertion, starts close, waits until the ledger is marked closing, and then allows the open to proceed. It verifies that the returned future fails with ManagedLedgerAlreadyClosedException, the late handle is closed, and the cache is empty.

I used two coordinated threads instead of calling close() directly inside newOpenLedgerOp(), because that mapping function runs while the cache section's non-reentrant write lock is held. Closing inline would attempt to acquire the same lock while draining the cache. I also replaced the fixed handle-count assumption with a recorded baseline. As a mutation check, removing the post-open close-state recheck makes this test fail.

@lhotari lhotari left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for tracking down this leak and for the thorough tests of the close path.

The close logic in ReadOnlyManagedLedgerImpl looks right to me. The flag is set before the cache is drained. The recheck after insertion covers the late-open race. Invalidation is async, so nothing blocks. A second close finds an empty cache and does nothing.

My main concern is the one @Denovo1998 raised: nothing in production calls the new asyncClose. ManagedLedgerFactoryImpl.asyncOpenReadOnlyManagedLedger returns ReadOnlyManagedLedgerImplWrapper. Neither the wrapper nor the ReadOnlyManagedLedger interface has a close method. ReadOnlyCursorImpl.asyncClose only closes the cursor. Factory shutdown does not help either, because read-only ledgers are never added to its ledgers map. SnapshotSegmentAbortedTxnProcessorImpl.readSegmentEntries is the only production caller, and it drops the ledger without closing it. The new tests build ReadOnlyManagedLedgerImpl directly, so they cannot show this. As it stands, the leak in the description still happens. Please add a close path to the interface and wrapper, call it from that caller, and add a test through the factory API. If you want to keep this PR as groundwork only, please say so in the description.

I also agree with the second open thread. No test reaches the recheck in getLedgerHandle. Deleting it leaves every test green. A deterministic option is to call readOnly.close() from inside the stubbed newOpenLedgerOp() execution, before super.getLedgerHandle returns. Then assert the returned future fails with ManagedLedgerAlreadyClosedException and the handle gets closed. Separately, testCloseReleasesHandleWhoseOpenIsPending assumes initialize() opens exactly one handle (openedHandles.size() == 2, get(1)). Recording the size before the call would be less brittle, especially next to #26817, which also touches ReadOnlyManagedLedgerImpl.

Expose an asynchronous close on the factory-facing read-only ledger API, delegate it through the wrapper, and close snapshot recovery ledgers after use.

Add deterministic coverage for the close/open race, factory delegation, and the broker caller lifecycle.

Assisted-by: OpenAI Codex
@void-ptr974

Copy link
Copy Markdown
Contributor Author

Thanks @Denovo1998 and @lhotari for the detailed reviews. I've pushed 401ff9c67fd to address the outstanding feedback:

  • added a backward-compatible ReadOnlyManagedLedger.asyncClose() and wrapper delegation;
  • closed the factory-created ledger from snapshot-segment recovery on every exit path;
  • added factory-facing and production-caller lifecycle coverage;
  • added deterministic coverage for the concurrent close/open path;
  • removed the test's fixed initialization-handle-count assumption; and
  • avoided holding the managed-ledger monitor while initiating read-handle cleanup.

The focused managed-ledger and broker tests pass locally, as does quickCheck. I also mutation-checked the three new paths: removing the post-open recheck, wrapper delegation, or caller close makes the corresponding test fail.

Could you please take another look when convenient?

@codelipenghui codelipenghui left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed head 401ff9c. No actionable findings identified within the reviewed scope.

Focused validation passed: 18 managed-ledger tests and 1 snapshot-recovery lifecycle test. The scoped cleanup and factory/caller close wiring look sound.

Preserve the expectedReadCount read overload alongside read-only ledger handle cleanup, and add a regression for count propagation.

Assisted-by: Codex
Register handle promises before IO, invalidate failed handles by identity, and release cached and temporary resources across close, initialization, and executor failures. Track read-only opens through shutdown, give factory cursors ownership, and dispose isolated entry caches and unreturned entries.

Assisted-by: Codex
Release late read-only ledgers and entries after a snapshot recovery wait times out or is interrupted. Preserve the recovery result when cleanup fails, and cover delayed resources and close failures.

Assisted-by: Codex
Detach the source watcher immediately after the close gate is claimed, independently of pending BookKeeper or cursor close callbacks. Run the cleanup without holding the managed-ledger monitor and cover a delayed current-handle close.

Assisted-by: Codex
@void-ptr974 void-ptr974 changed the title [fix][ml] Fix cached read handle leaks when closing read-only managed ledgers [fix][ml] Fix read resource leaks across managed ledger lifecycles Oct 25, 2026
Revert the lifecycle expansion in 3813e5d, c3598b5, and 67e1023. Restore the six-file read-only handle cleanup and master conflict resolution from 45436ce.

Keep the broader lifecycle, factory, cache, Shadow, and abandoned-wait changes on the backup branch for separate work.

Assisted-by: Codex
@void-ptr974 void-ptr974 changed the title [fix][ml] Fix read resource leaks across managed ledger lifecycles [fix][ml] Fix cached read handle leaks when closing read-only managed ledgers Oct 25, 2026

@codelipenghui codelipenghui left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed head f31fa58. No actionable findings identified within the reviewed scope.

The merge preserves expectedReadCount forwarding and the previously reviewed cleanup behavior. All 23 focused tests passed. The new regression test also failed when forwarding was deliberately removed and passed after restoration.

@lhotari lhotari left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks for working through the earlier points; each one is addressed.

ReadOnlyManagedLedger.asyncClose() now exists with a no-op default, ReadOnlyManagedLedgerImplWrapper.asyncClose() delegates to the implementation, and SnapshotSegmentAbortedTxnProcessorImpl.readSegmentEntries closes the ledger in a finally, so the production path now releases the cached handles. testFactoryWrapperCloseReleasesCachedHandle goes through the factory API, and SnapshotSegmentAbortedTxnProcessorCloseTest verifies the close call. The race test is deterministic, and the handle-count baseline is now taken after initialize().

Two small, non-blocking notes.

First, testCloseRejectsHandleInsertedAfterCloseStarts pins the failed-future return of the recheck in getLedgerHandle, but not its invalidateReadHandle call. The test pauses inside newOpenLedgerOp(), which runs under the map's write lock, so close's scan waits for the insertion and releases the handle itself. Deleting only the invalidateReadHandle in the recheck keeps all tests green. The window it covers (close finishing its scan before the insertion) needs a pause between the first flag check and the parent lookup. I would not block on it.

Second, a closed read-only ledger can stay the owner of the entry cache that RangeEntryCacheManagerImpl.getEntryCache registers by ledger name. ManagedLedgerFactoryImpl.close(ManagedLedger) returns early for a ledger that is not in its map, so the cache mapping is not removed. A later writable ledger with the same name reuses that cache, and RangeEntryCacheImpl.readFromStorage retries OffloadReadHandleClosedException through ml.reopenReadHandle, which now fails with ManagedLedgerAlreadyClosedException. This needs a read-only ledger to be the first opener, followed by an offloaded-ledger retry, so I consider it low risk. Removing the cache mapping in ReadOnlyManagedLedgerImpl.asyncClose would close it, if you want to.

For #26817: it also changes ReadOnlyManagedLedgerImpl and fixes the handle initialize() opens to read the last add confirmed, which bypasses ledgerCache and is not released by this PR's close. The two touch different methods, so I expect a clean merge, but whichever lands second should rebase and re-run ReadOnlyManagedLedgerHandleCloseTest.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants