Skip to content

fix(ptyfs): release the pts file on unregister - #12029

Merged
Rbb666 merged 2 commits into
RT-Thread:masterfrom
BernardXiong:fix/ptyfs-pts-file-leak
Nov 16, 2026
Merged

Rbb666 merged 2 commits into
RT-Thread:masterfrom
BernardXiong:fix/ptyfs-pts-file-leak

Conversation

@BernardXiong

Copy link
Copy Markdown
Member

Summary

ptyfs_unregister_pts() removes the pts file from the root list and recycles its unit number, but never frees the struct ptyfs_file allocated by ptyfs_register_pts(). Because the unit number is recycled, repeated open()/close() of /dev/pts/ptmx leaks one file per cycle without hitting the live-pair limit.

The failure path also had two problems:

  • pts_unit was zero after rt_calloc(), so a failed registration made the free path unregister unit 0.
  • lwp_tty_delete() unconditionally called rt_device_unregister() on a tty that had never been registered when ptyfs_register_pts() failed, corrupting the object list in the OOM path.
  • On lwp_tty_create() failure, the softc was freed before its condvars were detached.

Fix

  • Free the pts file after unlinking it from the root list.
  • Initialize pts_unit to -1 so the free path only unregisters a real unit.
  • Detach the condvars before freeing the softc.
  • In lwp_tty_delete(), only unregister the device if the object was actually registered as a device.

Verification

Built bsp/qemu-virt64-aarch64 with RT_USING_SMART and RT_USING_DFS_PTYFS (aarch64 musl GCC 12.2.0), booted the ext4 rootfs under QEMU, mounted ptyfs and ran 20,000 : < /dev/pts/ptmx open/close cycles:

  • Before the fix: available 245164 KB -> 237932 KB (-7232 KB, about 369 bytes/cycle).
  • After the fix: available 245156 KB -> 245116 KB (-40 KB).

The test ran from the boot shell (root), not from an unprivileged process, and the OOM crash path itself was not forced.

Fixes #12023.

ptyfs_unregister_pts() removed the pts file from the root list and
recycled its unit number but never freed the allocation. Repeated
open/close of /dev/pts/ptmx therefore leaked one struct ptyfs_file per
cycle and eventually exhausted the kernel heap.

Free the file after unlinking it. Also initialize pts_unit to -1 so a
failed registration does not unregister unit 0, detach the condvars
before freeing the softc on the lwp_tty_create() failure path, and skip
rt_device_unregister() for a tty that was never registered.
@github-actions

Copy link
Copy Markdown

👋 感谢您对 RT-Thread 的贡献!Thank you for your contribution to RT-Thread!

为确保代码符合 RT-Thread 的编码规范,请在你的仓库中执行以下步骤运行代码格式化工作流(如果格式化CI运行失败)。
To ensure your code complies with RT-Thread's coding style, please run the code formatting workflow by following the steps below (If the formatting of CI fails to run).


🛠 操作步骤 | Steps

  1. 前往 Actions 页面 | Go to the Actions page
    点击进入工作流 → | Click to open workflow →

  2. 点击 Run workflow | Click Run workflow

  • Use workflow from 保持默认分支(通常为 master)
    Keep the default branch (usually master) in Use workflow from
  • 在 branch 输入框填写 PR 分支 fix/ptyfs-pts-file-leak
    Enter PR branch fix/ptyfs-pts-file-leak in the branch field
  • 设置需排除的文件/目录(目录请以"/"结尾)
    Set files/directories to exclude (directories should end with "/")
  1. 等待工作流完成 | Wait for the workflow to complete
    格式化后的代码将作为独立提交推送至你的分支。
    The formatting changes will be pushed to your branch as a separate commit.

完成后,提交将自动更新至 fix/ptyfs-pts-file-leak 分支,关联的 Pull Request 也会同步更新。
Once completed, commits will be pushed to the fix/ptyfs-pts-file-leak branch automatically, and the related Pull Request will be updated.

如有问题欢迎联系我们,再次感谢您的贡献!💐
If you have any questions, feel free to reach out. Thanks again for your contribution!

@github-actions github-actions Bot added RT-Smart RT-Thread Smart related PR or issues component: lwp Component labels Nov 10, 2026
@github-actions

github-actions Bot commented Nov 10, 2026 •

Copy link
Copy Markdown

📌 Code Review Assignment

🏷️ Tag: components

Reviewers: @Maihuanyi

Changed Files (Click to expand)
  • components/dfs/dfs_v2/filesystems/ptyfs/ptyfs.c
  • components/lwp/terminal/freebsd/tty_pts.c
  • components/lwp/terminal/tty_device.c

🏷️ Tag: components_lwp

Reviewers: @xu18838022837

Changed Files (Click to expand)
  • components/lwp/terminal/freebsd/tty_pts.c
  • components/lwp/terminal/tty_device.c

📊 Current Review Status (Last Updated: 2026-11-14 00:16 CST)


📝 Review Instructions

  1. 维护者可以通过单击此处来刷新审查状态: 🔄 刷新状态
    Maintainers can refresh the review status by clicking here: 🔄 Refresh Status

  2. 确认审核通过后评论 LGTM/lgtm
    Comment LGTM/lgtm after confirming approval

  3. PR合并前需至少一位维护者确认
    PR must be confirmed by at least one maintainer before merging

ℹ️ 刷新CI状态操作需要具备仓库写入权限。
ℹ️ Refresh CI status operation requires repository Write permission.

@github-actions

github-actions Bot commented Nov 10, 2026 •

Copy link
Copy Markdown

MemBrowse Memory Report · Explore what changed →

at32f415-start

  • ROM: .text +80 B (+0.1%, 65,524 B / 262,144 B, total: 25% used)

esp32-c3

  • dram0_0_seg: .dram0.data +36 B, .stack +4 B (+0.0%, 216,832 B / 325,120 B, total: 67% used)
  • iram0_2_seg: .flash.text +16 B (+0.0%, 421,880 B / 8,388,576 B, total: 5% used)

gd32105r-start

  • CODE: .text +80 B (+0.1%, 69,436 B / 262,144 B, total: 26% used)

hc32f334

  • FLASH: .text +80 B (+0.1%, 99,692 B / 131,072 B, total: 76% used)

hpmicro-hpm5100evk

  • ILM: .fast +32 B (+0.1%, 42,672 B / 262,144 B, total: 16% used)
  • XPI0: .fast +32 B, .text +24 B (+0.1%, 102,980 B / 1,048,576 B, total: 10% used)

hpmicro-hpm5301evklite

  • XPI0: .text +24 B (+0.0%, 76,788 B / 1,048,576 B, total: 7% used)

infineon-psoc6

  • flash: .text +80 B (+0.1%, 116,292 B / 262,144 B, total: 44% used)

k230

  • SRAM: .data +16 B, .eh_frame +432 B, .sbss -4 B, .stack -16 B, .text -4,016 B (-0.3%, 1,391,680 B / 60,682,240 B, total: 2% used)

loongson-ls1cdev

  • Code: .text +540 B (+0.1%, 363,948 B)

nordic-nrf51822

  • FLASH: .text +76 B (+0.1%, 52,132 B / 262,144 B, total: 20% used)

nuvoton-m487

  • CODE: .text +332 B (+0.1%, 356,912 B / 524,288 B, total: 68% used)

qemu-virt64-aarch64

  • Code: .eh_frame +80 B, .text +1,360 B (+0.1%, 1,011,196 B)
  • Data: .bss +16 B (+0.0%, 1,068,120 B)

raspberry-pico-rp2040

  • FLASH: .rodata +40 B, .text +40 B (+0.1%, 114,272 B / 2,097,152 B, total: 5% used)

renesas-ra2l1

  • FLASH: .text +188 B (+0.2%, 98,492 B / 262,144 B, total: 38% used)

simulator

  • Code: .rela.dyn +1,728 B, .rodata +224 B, .text +1,644 B (+0.1%, 2,676,654 B)
  • Data: .data +1,824 B (+0.1%, 1,610,852 B)

stm32f407-rt-spark

  • CODE: .text +80 B (+0.1%, 86,168 B / 1,048,576 B, total: 8% used)

stm32l475-atk-pandora-llvm

  • ROM: .text +92 B (+0.1%, 82,804 B / 524,288 B, total: 16% used)

wch-ch32v208w-r0

  • FLASH: .text +88 B (+0.1%, 159,976 B / 491,520 B, total: 33% used)

x86

  • Code: .rodata +32 B, .text +304 B (+0.2%, 199,277 B)

xuantie-e901plus

  • ISRAM: .rodata +48 B, .text +16 B (+0.1%, 46,488 B / 131,072 B, total: 35% used)

No memory changes detected for:


Catch memory regressions in CI with MemBrowse. Free for open source.

@Rbb666 Rbb666 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

释放遗漏的 pts 文件和失败路径修正方向正确,但新增释放与现有目录读取/属性操作之间缺少生命周期同步,会引入并发访问已释放对象的问题。请先处理下方行内意见,并补充目录遍历与 PTY 创建/关闭并发的验证。

另一个非阻塞建议:标题及提交标题建议统一为 [dfs][ptyfs] Release the pts file on unregister。

本次验证为 diff 与调用链静态核对,未进行板级或 QEMU 并发复现。

rid_bitmap_put(ptsno_pool, ptsno);
/* the file is no longer reachable from the root list */
rt_free(pts_file);
rc = 0;

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] 释放 pts_file 前需要同步目录读者的生命周期

这里新增 rt_free(pts_file) 后,仅在 ptyfile_remove_from_root() 中短暂持有 sb->lock 不足以保证没有读者持有这个对象。devpty_deffops_getdents() 在不持有该锁的情况下遍历 root_file.subdirs 并读取 n_file->type/basename;另一个线程关闭 PTY 后可经 ptsdrv_free() 删除并释放同一个节点,目录遍历随后就会访问已释放内存。ptyfile_lookup() 也在返回裸指针前释放锁,ptyfs_ops_setattr() 等调用方仍会在返回后访问它。DFS 外层的 dfs_file_getdents() 和 dfs_file_setattr() 没有与关闭路径共享的全局锁来消除这个窗口。

请让查找/遍历后的使用与摘链释放由同一把锁覆盖,或为 ptyfs_file 引入明确的引用生命周期;仅把 free 移进 remove 的锁区仍不能保护已经取得指针的读者。建议增加一个线程反复打开/关闭 ptmx、另一个线程遍历 /dev/pts(并执行属性查询/修改)的并发验证。

ptyfile_lookup() returned a raw pointer after releasing sb->lock, while
ptyfs_unregister_pts() removed and freed the same node. A concurrent
getdents(), setattr() or vnode lookup could then read freed memory.

Add a reference count protected by sb->lock: lookup takes a reference and
the last ptyfile_put() frees the node. The root list reference is dropped
when the file is removed, so unregistering a pts no longer frees a file
that a reader still uses. getdents() also holds the lock while walking the
subdir list, keeping the entries alive during the copy.
@BernardXiong

Copy link
Copy Markdown
Member Author

已按 [P1] 意见补上生命周期同步,推送到 3cb78faebd:

  • struct ptyfs_file 增加受 sb->lock 保护的引用计数;ptyfile_lookup() 在持锁找到节点时取引用,新增 ptyfile_put() 在最后一个引用释放时 rt_free()。
  • ptyfs_unregister_pts() 依次摘链、释放 ptsno、ptyfile_put();摘链时同时释放根目录列表持有的引用。读者已经取得的指针不会在关闭 PTY 时被释放。
  • devpty_deffops_getdents() 在遍历 root_file.subdirs 期间持有 sb->lock,避免遍历与摘链释放并发;ptyfs_ops_setattr() 的 mode 更新也在锁内完成。
  • ptyfs_ops_lookup()、ptyfs_ops_create_vnode()、ptyfs_ops_setattr()、getdents() 的所有返回路径都会释放 lookup 引用。

验证:bsp/qemu-virt64-aarch64(RT-SMART,启用 DFS v2/PTYFS)编译通过;QEMU 中挂载 ptyfs 后连续 500 次打开/关闭 /dev/ptmx,无内核断言,目录最终只剩 ptmx(无泄漏)。并发 readdir 压力测试受 busybox ash 后台作业影响,未作为结论;后续可以补一个双线程 utest 做回归。

English: lifecycle synchronization added in 3cb78faebd. ptyfs_file now has an sb->lock-protected reference count taken by ptyfile_lookup() and released by ptyfile_put(); unregister drops both the root-list reference and the lookup reference, and getdents() holds the lock while walking the subdir list. Build passes on the RT-SMART aarch64 BSP, and 500 open/close cycles of /dev/ptmx in QEMU left no leaked nodes.

@Rbb666
Rbb666 merged commit 7c974b5 into RT-Thread:master Nov 16, 2026
114 checks passed
@BernardXiong
BernardXiong deleted the fix/ptyfs-pts-file-leak branch November 18, 2026 16:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

component: lwp Component RT-Smart RT-Thread Smart related PR or issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] ptyfs: ptyfs_unregister_pts() never frees the pts file — unprivileged open/close loop exhausts kernel memory and panics the kernel

2 participants