Repository navigation
[fix][ml] Fix cached read handle leaks when closing read-only managed ledgers - #26816
void-ptr974 wants to merge 9 commits into
Conversation
Release the read-only handle cache on close, including fenced closes and handles whose opens are pending, without changing writable managed-ledger shutdown. Main Issue: apache#26815 Assisted-by: OpenAI Codex
Guard cache insertion against concurrent close and cover reads after normal or fenced close, repeated close, and independent cleanup failures. Assisted-by: OpenAI Codex
| public synchronized void asyncClose(AsyncCallbacks.CloseCallback callback, Object ctx) { | ||
| closingOrClosed = true; | ||
| // There is no current write handle, so the parent's write-close callback cannot release this cache. | ||
| ledgerCache.forEach((ledgerId, readHandle) -> invalidateReadHandle(ledgerId)); | ||
| super.asyncClose(callback, ctx); |
There was a problem hiding this comment.
I think there's still a lifecycle gap in the actual factory path.
ManagedLedgerFactoryImpl.asyncOpenReadOnlyManagedLedger() returns a ReadOnlyManagedLedgerImplWrapper, but neither the public ReadOnlyManagedLedger interface nor the wrapper exposes a close method.
The read-only cursor path looks similar: ReadOnlyCursorImpl.asyncClose() only closes the cursor, not the backing managed ledger.
So these callers never reach this new asyncClose() implementation, and the cached handles can still be left open. One example is SnapshotSegmentAbortedTxnProcessorImpl#readSegmentEntries().
Could we also wire up the ownership/close path for the factory-facing APIs? A regression test through the factory API would help catch this.
There was a problem hiding this comment.
Thanks for pointing this out. Addressed in 401ff9c67fd: ReadOnlyManagedLedger now exposes a backward-compatible default asyncClose(), and ReadOnlyManagedLedgerImplWrapper delegates it to the underlying implementation. Once the ledger has been acquired, SnapshotSegmentAbortedTxnProcessorImpl initiates the close in a finally block on every exit path, while logging close failures without masking the recovery result. I also added regression coverage through both the factory API and the production caller.
I kept the factory cursor path separate because a ReadOnlyCursor can also be created from a caller-owned ReadOnlyManagedLedger, in which case the cursor does not necessarily own the backing ledger. Safely changing openReadOnlyCursor would require an ownership-aware wrapper and separate lifecycle coverage. There are currently no in-tree production callers of that factory cursor API; I can address it separately if maintainers would like.
| } | ||
|
|
||
| @Test | ||
| public void testCloseReleasesHandleWhoseOpenIsPending() throws Exception { |
There was a problem hiding this comment.
Nice coverage for a pending BookKeeper open.
One thing I noticed: the future is already in ledgerCache before close() runs, so this tests cleanup of an existing pending open, but not the race that the second closingOrClosed check is meant to handle.
Could we add a small targeted test for a cache insertion racing with close? I'd mainly like to make sure a late insertion can't leave a handle behind.
There was a problem hiding this comment.
Thanks. 401ff9c67fd adds testCloseRejectsHandleInsertedAfterCloseStarts, which pauses the BookKeeper open before cache insertion, starts close, waits until the ledger is marked closing, and then allows the open to proceed. It verifies that the returned future fails with ManagedLedgerAlreadyClosedException, the late handle is closed, and the cache is empty.
I used two coordinated threads instead of calling close() directly inside newOpenLedgerOp(), because that mapping function runs while the cache section's non-reentrant write lock is held. Closing inline would attempt to acquire the same lock while draining the cache. I also replaced the fixed handle-count assumption with a recorded baseline. As a mutation check, removing the post-open close-state recheck makes this test fail.
lhotari
left a comment
There was a problem hiding this comment.
Thanks for tracking down this leak and for the thorough tests of the close path.
The close logic in ReadOnlyManagedLedgerImpl looks right to me. The flag is set before the cache is drained. The recheck after insertion covers the late-open race. Invalidation is async, so nothing blocks. A second close finds an empty cache and does nothing.
My main concern is the one @Denovo1998 raised: nothing in production calls the new asyncClose. ManagedLedgerFactoryImpl.asyncOpenReadOnlyManagedLedger returns ReadOnlyManagedLedgerImplWrapper. Neither the wrapper nor the ReadOnlyManagedLedger interface has a close method. ReadOnlyCursorImpl.asyncClose only closes the cursor. Factory shutdown does not help either, because read-only ledgers are never added to its ledgers map. SnapshotSegmentAbortedTxnProcessorImpl.readSegmentEntries is the only production caller, and it drops the ledger without closing it. The new tests build ReadOnlyManagedLedgerImpl directly, so they cannot show this. As it stands, the leak in the description still happens. Please add a close path to the interface and wrapper, call it from that caller, and add a test through the factory API. If you want to keep this PR as groundwork only, please say so in the description.
I also agree with the second open thread. No test reaches the recheck in getLedgerHandle. Deleting it leaves every test green. A deterministic option is to call readOnly.close() from inside the stubbed newOpenLedgerOp() execution, before super.getLedgerHandle returns. Then assert the returned future fails with ManagedLedgerAlreadyClosedException and the handle gets closed. Separately, testCloseReleasesHandleWhoseOpenIsPending assumes initialize() opens exactly one handle (openedHandles.size() == 2, get(1)). Recording the size before the call would be less brittle, especially next to #26817, which also touches ReadOnlyManagedLedgerImpl.
Expose an asynchronous close on the factory-facing read-only ledger API, delegate it through the wrapper, and close snapshot recovery ledgers after use. Add deterministic coverage for the close/open race, factory delegation, and the broker caller lifecycle. Assisted-by: OpenAI Codex
|
Thanks @Denovo1998 and @lhotari for the detailed reviews. I've pushed
The focused managed-ledger and broker tests pass locally, as does Could you please take another look when convenient? |
codelipenghui
left a comment
There was a problem hiding this comment.
Reviewed head 401ff9c. No actionable findings identified within the reviewed scope.
Focused validation passed: 18 managed-ledger tests and 1 snapshot-recovery lifecycle test. The scoped cleanup and factory/caller close wiring look sound.
Preserve the expectedReadCount read overload alongside read-only ledger handle cleanup, and add a regression for count propagation. Assisted-by: Codex
Register handle promises before IO, invalidate failed handles by identity, and release cached and temporary resources across close, initialization, and executor failures. Track read-only opens through shutdown, give factory cursors ownership, and dispose isolated entry caches and unreturned entries. Assisted-by: Codex
Release late read-only ledgers and entries after a snapshot recovery wait times out or is interrupted. Preserve the recovery result when cleanup fails, and cover delayed resources and close failures. Assisted-by: Codex
Detach the source watcher immediately after the close gate is claimed, independently of pending BookKeeper or cursor close callbacks. Run the cleanup without holding the managed-ledger monitor and cover a delayed current-handle close. Assisted-by: Codex
codelipenghui
left a comment
There was a problem hiding this comment.
Reviewed head f31fa58. No actionable findings identified within the reviewed scope.
The merge preserves expectedReadCount forwarding and the previously reviewed cleanup behavior. All 23 focused tests passed. The new regression test also failed when forwarding was deliberately removed and passed after restoration.
lhotari
left a comment
There was a problem hiding this comment.
LGTM. Thanks for working through the earlier points; each one is addressed.
ReadOnlyManagedLedger.asyncClose() now exists with a no-op default, ReadOnlyManagedLedgerImplWrapper.asyncClose() delegates to the implementation, and SnapshotSegmentAbortedTxnProcessorImpl.readSegmentEntries closes the ledger in a finally, so the production path now releases the cached handles. testFactoryWrapperCloseReleasesCachedHandle goes through the factory API, and SnapshotSegmentAbortedTxnProcessorCloseTest verifies the close call. The race test is deterministic, and the handle-count baseline is now taken after initialize().
Two small, non-blocking notes.
First, testCloseRejectsHandleInsertedAfterCloseStarts pins the failed-future return of the recheck in getLedgerHandle, but not its invalidateReadHandle call. The test pauses inside newOpenLedgerOp(), which runs under the map's write lock, so close's scan waits for the insertion and releases the handle itself. Deleting only the invalidateReadHandle in the recheck keeps all tests green. The window it covers (close finishing its scan before the insertion) needs a pause between the first flag check and the parent lookup. I would not block on it.
Second, a closed read-only ledger can stay the owner of the entry cache that RangeEntryCacheManagerImpl.getEntryCache registers by ledger name. ManagedLedgerFactoryImpl.close(ManagedLedger) returns early for a ledger that is not in its map, so the cache mapping is not removed. A later writable ledger with the same name reuses that cache, and RangeEntryCacheImpl.readFromStorage retries OffloadReadHandleClosedException through ml.reopenReadHandle, which now fails with ManagedLedgerAlreadyClosedException. This needs a read-only ledger to be the first opener, followed by an offloaded-ledger retry, so I consider it low risk. Removing the cache mapping in ReadOnlyManagedLedgerImpl.asyncClose would close it, if you want to.
For #26817: it also changes ReadOnlyManagedLedgerImpl and fixes the handle initialize() opens to read the last add confirmed, which bypasses ledgerCache and is not released by this PR's close. The two touch different methods, so I expect a clean merge, but whichever lands second should rebase and re-run ReadOnlyManagedLedgerHandleCloseTest.
Main Issue: #26815
Motivation
Read-only managed ledgers have no current write handle, so closing them does not reach the cached read-handle cleanup in the parent's write-close callback. The factory wrapper also lacks a close operation, leaving snapshot recovery without a way to release its cached handles.
Modifications
ReadOnlyManagedLedger.asyncClose()and delegate it through the factory wrapper.finallyblock on success, failure, and early return. Report cleanup failure without replacing the recovery result.expectedReadCountoverload when resolving the merge conflict and cover its forwarding with a regression test.The close future represents logical closure; cleanup of handles whose opens are still pending may finish asynchronously.
Verifying this change
Local validation with test retries disabled:
./gradlew quickCheck.Does this pull request potentially affect one of the following parts: