Repository navigation
[fix][broker] Fix message dispatch failures caused by ArrayIndexOutOfBoundsException in InMemoryRedeliveryTracker - #27064
Open
void-ptr974 wants to merge 3 commits into
Conversation
…BoundsException in InMemoryRedeliveryTracker Assisted-by: Codex
codelipenghui
approved these changes
Oct 23, 2026
codelipenghui
left a comment
Contributor
There was a problem hiding this comment.
LGTM, thanks for the detailed analysis.
The root cause is right. getRedeliveryCount() ran fastutil lookups under a StampedLock optimistic read, so a concurrent rehash (from a redelivery insert, or an ack-triggered shrink) could throw before validate(), and PulsarCommandSenderImpl then closes the connection. Holding a read lock across both lookups fixes it.
Verified locally:
- Against the previous implementation,
InMemoryRedeliveryTrackerTestfails withIndex 64 out of bounds for length 33. - With this PR, it passes 4/4, and
ExposeMessageRedeliveryCountTest.testRedeliveryCountpasses.
One optional, non-blocking suggestion inline.
Since #23640 was cherry-picked to branch-4.0 (release/4.0.1), this fix should go to the same release lines.
…marks Use a StampedLock read lock across both tracker map lookups while retaining the existing write locking. Add shared-tracker JMH workloads and reproduction commands for read, mixed update, and batch acknowledgement paths. Assisted-by: Codex
Denovo1998
approved these changes
Oct 27, 2026
dao-jun
approved these changes
Oct 28, 2026
Member
|
btw. this particular details was discussed already as part of the review of the PR that introduced the issue: https://github.com/apache/pulsar/pull/23640/changes#r1858986420 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
InMemoryRedeliveryTrackercan throwArrayIndexOutOfBoundsExceptionwhile the broker sends messages to a consumer.The tracker stores counts in two nested, non-thread-safe fastutil maps.
getRedeliveryCount()uses aStampedLockoptimistic read and validates the stamp only after both map lookups. An optimistic read does not block a writer, so either map can resize while a lookup is in progress.Consumers of the same persistent Shared or Key_Shared subscription share this tracker, but their connections can run on different Netty event loops. For example, consumer A's event loop can read a count while sending an entry, while consumer B's event loop handles a redelivery request and inserts other positions into the same tracker.
The inner-map failure can be illustrated by A reading
(ledgerId=7, entryId=0)while B adds other entries to ledger 7. In fastutil, key0uses the special slotvalue[n], wherenis the table capacity. Growing the table from 32 to 64 replaces a value array of length 33 with one of length 65.One possible interleaving is:
tryOptimisticRead()and gets a stamp, without holding a read lock.value[n]and loads the oldvaluearray reference, whose length is 33.nto 64, replacesvaluewith the new array of length 65, and releases the write lock.n = 64, then accesses the previously loaded old array at index 64. This throwsArrayIndexOutOfBoundsException.validate(stamp), so the fallback lookup under the read lock cannot run.The queried position remains valid and its count is never changed by B. Even if B has finished resizing before A reads
n, A can still combine the old array reference with the new capacity. The reproduced failure isIndex 64 out of bounds for length 33. The outer ledger map has the same failure mechanism when new ledgers are added.The sender catches the exception, interrupts the batch, and closes the consumer connection. This disrupts message dispatch and causes reconnection, redelivery, and additional delivery latency.
Modifications
StampedLockand holdreadLock()across both map lookups ingetRedeliveryCount(), replacing the optimistic read. Existing write locking for increments, removals and clearing remains in place; release locks infinally.InMemoryRedeliveryTrackerBenchmarkand reproduction commands inmicrobench/README.md, covering shared-tracker reads, mixed increments/reads, and batch acknowledgements without new dependencies.Lock performance comparison
The local comparison used the earlier
ReentrantReadWriteLock(RRWL) implementation atffc22043and theStampedLock.readLock()implementation now committed inab6ff644, with the same benchmark and dependencies. Environment: Linux x86_64 / Xeon E5-2673 v4, OpenJDK 21.0.12.1, JMH 1.37, fastutil 8.5.19, generational ZGC, 512 MiB heap, CPUs 0-7. Each configuration used 2 forks, 3 x 1 s warmup and 5 x 1 s measurement iterations, with-prof gc.Throughput is million operations/s, mean +/- JMH reported 99.9% error:
Single-thread empty reads were comparable. Across the six 8-thread configurations, StampedLock achieved 1.42-4.04x the mean throughput of RRWL in these runs. For 8-thread read-only workloads, allocation decreased from approximately 79 B/op with RRWL to below 0.004 B/op with StampedLock.
RRWL supports reentrancy and maintains per-thread read-hold counts outside its first-reader fast path. A separate JFR run found
HoldCounterandThreadLocalMap.Entryallocations in that read-lock path. StampedLock is non-reentrant; its read-lock fast path updates shared reader state without this bookkeeping, which is sufficient for this non-nested lookup. The profile supports this explanation for the allocation difference; it does not establish how much of the throughput difference comes from GC. Both locks still update shared lock state and can contend; scheduling and contention also affect throughput.All threads share one tracker; populated states contain 4,096 positions across 16 ledgers. Mixed writes update existing keys without resizing. The ACK workload performs N lookups followed by one
removeBatchcall on an empty tracker; one operation is a lookup or a batch ACK, so compare equal batch sizes. CPUs were pinned but not reserved exclusively, frequencies were uncontrolled, and some intervals are wide. These are tracker microbenchmark results, not estimates of broker throughput, tail latency or retained memory per dispatcher.Verifying this change
The four existing
InMemoryRedeliveryTrackerTestcases use two real threads and normal tracker operations, without modifying map internals. Across three fresh JVMs per implementation, the original optimistic tracker failed 7/12 case executions withArrayIndexOutOfBoundsException; RRWL and StampedLock read-lock variants each passed 12/12. Passing these runs supports the fix; holding a read lock across both map lookups excludes structural changes protected by the write lock and supplies the correctness argument.For the final source, isolated Linux x86_64 / OpenJDK 21 validation compiled the tracker, interface, benchmark and existing test using cached dependencies and managed-ledger classes, with
javac --release 17and the JMH annotation processor. All 4 regression cases and all 8 JMH smoke parameter combinations passed. Smoke runs check execution, rather than performance.The full repository
./gradlew quickCheck --offline --console=plain --no-daemonpassed in an isolated checkout.Does this pull request potentially affect one of the following parts: