Skip to content

[fix][broker] Fix message dispatch failures caused by ArrayIndexOutOfBoundsException in InMemoryRedeliveryTracker - #27064

Open
void-ptr974 wants to merge 3 commits into
apache:masterfrom
void-ptr974:fix-redelivery-tracker-concurrent-read
Open

void-ptr974 wants to merge 3 commits into
apache:masterfrom
void-ptr974:fix-redelivery-tracker-concurrent-read

Conversation

@void-ptr974

@void-ptr974 void-ptr974 commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

Motivation

InMemoryRedeliveryTracker can throw ArrayIndexOutOfBoundsException while the broker sends messages to a consumer.

The tracker stores counts in two nested, non-thread-safe fastutil maps. getRedeliveryCount() uses a StampedLock optimistic read and validates the stamp only after both map lookups. An optimistic read does not block a writer, so either map can resize while a lookup is in progress.

Consumers of the same persistent Shared or Key_Shared subscription share this tracker, but their connections can run on different Netty event loops. For example, consumer A's event loop can read a count while sending an entry, while consumer B's event loop handles a redelivery request and inserts other positions into the same tracker.

The inner-map failure can be illustrated by A reading (ledgerId=7, entryId=0) while B adds other entries to ledger 7. In fastutil, key 0 uses the special slot value[n], where n is the table capacity. Growing the table from 32 to 64 replaces a value array of length 33 with one of length 65.

One possible interleaving is:

Step Consumer A: sending and reading the count Consumer B: handling redelivery and inserting records
1 Calls tryOptimisticRead() and gets a stamp, without holding a read lock.
2 Starts evaluating value[n] and loads the old value array reference, whose length is 33.
3 Acquires the write lock and inserts other entries, triggering a resize.
4 Updates n to 64, replaces value with the new array of length 65, and releases the write lock.
5 Reads the new n = 64, then accesses the previously loaded old array at index 64. This throws ArrayIndexOutOfBoundsException.
6 Never reaches validate(stamp), so the fallback lookup under the read lock cannot run.
Old array retained by A:  [0] [1] ... [32]   length = 33
Capacity now read by A:                       n = 64
Attempted lookup:        oldArray[64]        -> out of bounds

The queried position remains valid and its count is never changed by B. Even if B has finished resizing before A reads n, A can still combine the old array reference with the new capacity. The reproduced failure is Index 64 out of bounds for length 33. The outer ledger map has the same failure mechanism when new ledgers are added.

The sender catches the exception, interrupts the batch, and closes the consumer connection. This disrupts message dispatch and causes reconnection, redelivery, and additional delivery latency.

Modifications

  • Keep StampedLock and hold readLock() across both map lookups in getRedeliveryCount(), replacing the optimistic read. Existing write locking for increments, removals and clearing remains in place; release locks in finally.
  • Add concurrent regression tests for reads during expansion of the inner and outer maps, covering zero and nonzero keys while reading an unchanged position.
  • Add InMemoryRedeliveryTrackerBenchmark and reproduction commands in microbench/README.md, covering shared-tracker reads, mixed increments/reads, and batch acknowledgements without new dependencies.

Lock performance comparison

The local comparison used the earlier ReentrantReadWriteLock (RRWL) implementation at ffc22043 and the StampedLock.readLock() implementation now committed in ab6ff644, with the same benchmark and dependencies. Environment: Linux x86_64 / Xeon E5-2673 v4, OpenJDK 21.0.12.1, JMH 1.37, fastutil 8.5.19, generational ZGC, 512 MiB heap, CPUs 0-7. Each configuration used 2 forks, 3 x 1 s warmup and 5 x 1 s measurement iterations, with -prof gc.

Throughput is million operations/s, mean +/- JMH reported 99.9% error:

Workload Threads RRWL StampedLock read lock
Empty tracker, read only 1 50.72 +/- 0.63 49.91 +/- 1.78
Populated tracker, read hits 1 39.97 +/- 0.36 45.88 +/- 2.92
Empty tracker, read only 8 3.36 +/- 0.33 9.52 +/- 4.80
Populated tracker, read hits 8 4.04 +/- 0.98 5.75 +/- 0.14
Existing positions, 1% increments 8 4.35 +/- 0.53 11.79 +/- 0.45
Existing positions, 10% increments 8 2.63 +/- 0.11 5.39 +/- 0.46
Empty tracker, ACK batch size 1 8 4.25 +/- 0.09 9.07 +/- 1.21
Empty tracker, ACK batch size 100 8 4.39 +/- 2.02 17.75 +/- 1.16

Single-thread empty reads were comparable. Across the six 8-thread configurations, StampedLock achieved 1.42-4.04x the mean throughput of RRWL in these runs. For 8-thread read-only workloads, allocation decreased from approximately 79 B/op with RRWL to below 0.004 B/op with StampedLock.

RRWL supports reentrancy and maintains per-thread read-hold counts outside its first-reader fast path. A separate JFR run found HoldCounter and ThreadLocalMap.Entry allocations in that read-lock path. StampedLock is non-reentrant; its read-lock fast path updates shared reader state without this bookkeeping, which is sufficient for this non-nested lookup. The profile supports this explanation for the allocation difference; it does not establish how much of the throughput difference comes from GC. Both locks still update shared lock state and can contend; scheduling and contention also affect throughput.

All threads share one tracker; populated states contain 4,096 positions across 16 ledgers. Mixed writes update existing keys without resizing. The ACK workload performs N lookups followed by one removeBatch call on an empty tracker; one operation is a lookup or a batch ACK, so compare equal batch sizes. CPUs were pinned but not reserved exclusively, frequencies were uncontrolled, and some intervals are wide. These are tracker microbenchmark results, not estimates of broker throughput, tail latency or retained memory per dispatcher.

Verifying this change

  • Make sure that the change passes the CI checks.

The four existing InMemoryRedeliveryTrackerTest cases use two real threads and normal tracker operations, without modifying map internals. Across three fresh JVMs per implementation, the original optimistic tracker failed 7/12 case executions with ArrayIndexOutOfBoundsException; RRWL and StampedLock read-lock variants each passed 12/12. Passing these runs supports the fix; holding a read lock across both map lookups excludes structural changes protected by the write lock and supplies the correctness argument.

For the final source, isolated Linux x86_64 / OpenJDK 21 validation compiled the tracker, interface, benchmark and existing test using cached dependencies and managed-ledger classes, with javac --release 17 and the JMH annotation processor. All 4 regression cases and all 8 JMH smoke parameter combinations passed. Smoke runs check execution, rather than performance.

The full repository ./gradlew quickCheck --offline --console=plain --no-daemon passed in an isolated checkout.

Does this pull request potentially affect one of the following parts:

  • Dependencies (add or upgrade a dependency)
  • The public API
  • The schema
  • The default values of configurations
  • The threading model
  • The binary protocol
  • The REST endpoints
  • The admin CLI options
  • The metrics
  • Anything that affects deployment

…BoundsException in InMemoryRedeliveryTracker

Assisted-by: Codex

@codelipenghui codelipenghui left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks for the detailed analysis.

The root cause is right. getRedeliveryCount() ran fastutil lookups under a StampedLock optimistic read, so a concurrent rehash (from a redelivery insert, or an ack-triggered shrink) could throw before validate(), and PulsarCommandSenderImpl then closes the connection. Holding a read lock across both lookups fixes it.

Verified locally:

  • Against the previous implementation, InMemoryRedeliveryTrackerTest fails with Index 64 out of bounds for length 33.
  • With this PR, it passes 4/4, and ExposeMessageRedeliveryCountTest.testRedeliveryCount passes.

One optional, non-blocking suggestion inline.

Since #23640 was cherry-picked to branch-4.0 (release/4.0.1), this fix should go to the same release lines.

…marks

Use a StampedLock read lock across both tracker map lookups while retaining the existing write locking. Add shared-tracker JMH workloads and reproduction commands for read, mixed update, and batch acknowledgement paths.

Assisted-by: Codex
@lhotari

lhotari commented Nov 18, 2026

Copy link
Copy Markdown
Member

btw. this particular details was discussed already as part of the review of the PR that introduced the issue: https://github.com/apache/pulsar/pull/23640/changes#r1858986420

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants