Stage any upload source onto a local disk path so the uploader can stream it
without buffering the whole file. `stageSourceToDisk(source, maxBytes)` unifies:
local file:// / raw path → hardlink into an owned stage dir (O(1); the inode
link pins the bytes against a caller-side unlink/retention), streamed-copy
fallback on any link() failure; http(s):// → stream to a .part file; base64://
→ maxBytes-checked decode to a .part file. Temps are finalized atomically
(fsync file → rename → fsync dir); a TTL reaper reaps crash-orphaned temps while
an in-use `active` set protects live uploads (a hardlink keeps the source's old
mtime, so the staged path is reserved in `active` before it appears on disk).
To avoid divergence, the existing `loadBinarySource` HTTP transport is extracted
into a shared `downloadHttp(source, name, maxBytes, makeSink)` engine (browser
UA, redirect, timeout, Referer retry, declared + streamed size caps, noRetry);
`loadBinarySource` now drives it with an in-memory sink — behavior unchanged.
Additive only — not yet wired into video/file uploads. Tests cover the
retention race (hardlink survives the original's deletion), the reaper's
active-set protection, size caps, symlink resolution, the copy fallback, the
Referer retry reusing the .part, and streamed-overflow .part cleanup.
Single-pass, disk-streaming replacement for the buffered
`computeHashes` + `computeVideoSha1Blocks` (+ offline-file head md5): reads the
file 1 MiB at a time so hashing a multi-GiB upload never buffers the whole file.
Reproduces the buffered helpers exactly — whole-file md5/sha1, the per-1 MiB
intermediate SHA1 state snapshots plus the finalized whole-file SHA1, and an
optional md5 over the first N bytes (`headLimit`, for `md510MCheckSum`).
Additive only — not wired into any caller yet; the old helpers stay in place.
Lands in a new leaf module to avoid a utils↔highway-client import cycle. A
byte-oracle test asserts identity vs the buffered functions across sizes around
the 1 MiB block boundary (empty, sub-block, exact multiples, ±1, multi-block +
tail) and head limits across chunk boundaries.
Prep for streaming large media uploads from disk instead of buffering the
whole file in RAM. Add a `ChunkSource` abstraction — `BufferChunkSource`
(in-memory, byte-for-byte the old path) and `FileChunkSource` (disk-backed,
reads exact-length chunks via a FileHandle, loops on short reads, throws on
unexpected EOF) — and change `uploadHighwayHttp` to consume a `ChunkSource`
instead of a `Uint8Array`. The uploader now owns the source and closes it once
in `finally` (best-effort; never masks a primary upload error).
Pure structural refactor: no streaming is wired yet. All five call sites
(pipeline runPuts, group + private file upload, personal + group avatar) wrap
their bytes in `BufferChunkSource`, so behavior is unchanged. Adds a
byte-oracle test proving FileChunkSource ≡ BufferChunkSource across block
boundaries, plus short-read/EOF/empty-input coverage.
The 100 MB cap that re-routed larger videos to the file-upload pipeline
was a workaround for the width/height=0 → 已过期 bug (fixed in the prior
commit), not a real QQ size limit. Real-machine verified on tempserver:
88 MB / 149 MB / 299 MB / 497 MB videos all upload, render and play as
normal videos in a group. Raise MAX_VIDEO_SIZE to 1.5 GiB (matching the
file-upload ceiling); only videos above that still fall back to the file
path. Collapse the redundant MAX_VIDEO_SIZE_HARD (was also 1.5 GiB) into
the single constant and drop the now-dead 100 MB warn branches.
Note: the whole video is buffered in RAM for the Highway upload, so a
1.5 GiB send costs ~1.5 GiB+ of process memory — that, not a QQ limit,
is what bounds the ceiling.
A group video uploaded with fileInfo width/height = 0x0 reaches receivers
as 文件已过期 even though the resource is fresh and downloadable: QQ-NT
clients (Android especially) can't lay out the video tile without real
dimensions, so they render the expired placeholder. iOS is lenient — it
shows the expired badge but still opens the video, which is exactly the
"iOS 显示过期可以点开、安卓过期点不开" split users reported.
Real-machine capture on tempserver confirmed a working QQ group video's
MsgInfo carries real dimensions (e.g. 296x640) while ours sent 0x0. Fill
width/height from the probed video for group sends; c2c stays at 0 because
the QQ-NT server rejects non-zero dimensions there with a schema mismatch
(no c2c regression coverage yet). Verified end-to-end: a fresh 11 MB group
video that rendered 已过期 now displays and plays.
Older QQ clients (≤9.9.30) deliver a 闪传 (flash transfer) file as a richui
markdown commonElem (svc=45, busId=FlashTransfer); SnowLuma had no decoder for
it, so the message decoded to 0 elements and was reported empty. (Newer
clients ≥9.9.31 send a plain text+link message that already decodes fine.)
Decode svc=45 markdown → parse the FlashTransfer card JSON → emit a flash_file
element, surfaced as the segment the issue asked for:
{ type: "flash_file", data: { title, file_set_id, scene_type } }
- Field names (fileSetId / sceneType / title) confirmed against QQ NT's
flash-transfer manager in wrapper.node.i64. The card is built by the
sender's client so the exact JSON nesting is unknown — values are pulled by
a bounded recursive search rather than a pinned path (handles either shape).
- Non-FlashTransfer markdown is ignored; missing fields default (never empty).
- Manifest: D/S = yes, P/W = by-design-no (receive-only; sending is the
existing send_flash_msg action). Log preview shows [闪传文件:title].
- Tests: card decode (nested + alias + defaults + non-flash) and the
element→segment mapping.
Note: not end-to-end verified on a live message — the svc=45 format only comes
from ≤9.9.30 senders and no such client was available (every test sender here
is ≥9.9.31 → text format). Logic + field names are unit-tested / IDA-confirmed.
typecheck 12 包; protocol 596 / onebot 529 green.
* fix(protocol): retry qzone msglist via user.qzone route on -10000
emotion_cgi_msglist_v6 through the legacy h5.qzone.qq.com →
taotao.qzone.qq.com gateway can be rejected with `-10000 使用人数过多`
for some accounts (issue #182). Real-device captures show the same CGI
via user.qzone.qq.com → taotao.qq.com is not rate-limited, so retry once
through that route when the legacy route returns -10000.
Behaviour-preserving for accounts the legacy route already serves: the
retry only fires on a -10000 envelope; all other codes, transport
errors, and the missing-msglist case are unchanged.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(port-probe): fix port probe logic (parse accounts more than 2) and add description for 4302-4310 port.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Round out the OneBot v11 group-profile fields the issue asked for.
Group (get_group_info / get_group_list, inherited by get_group_info_ex /
get_group_detail_info):
- group_create_time + group_memo come free from the 0xFE5_2 list
(createdTime / announcement), verified live (createdTime=1729505343,
announcement populated).
- group_level lives only in the 0x88D_0 detail; get_group_info fetches it
(memoized 5min) so a joined group doesn't hit the detail every call.
get_group_list leaves it 0 — a per-group detail fetch across the whole
list would be far too costly.
Member (get_group_member_info / get_group_member_list):
- Add area / unfriendly / title_expire_time / card_changeable. QQ NT does
not expose these, so they're fixed placeholders ('' / false / 0 / true),
matching go-cqhttp / NapCat. sex/age stay unknown/0 (real values need a
per-member profile fetch — out of scope here).
typecheck 12 packages; onebot 523 / core 472 green.
* feat(qzone): add set_qzone_msg_right to change a feed's visibility
Add updateQzoneMsgRight in the protocol layer, porting php-qzone's
updateRight: fetch the feed via emotion_cgi_msgdetail_v6, rebuild the
full publish-shaped payload (richval/pic_bo/con), then override
ugc_right/allow_uins and POST to emotion_cgi_update. Expose it through
QzoneApi.updateRight and the OneBot set_qzone_msg_right action, with the
same ugc_right/target_uins validation as send_qzone_msg. Covered by
protocol, core and onebot tests.
* fix(qzone): make qzoneParse more stable and fixed a test example
Add ugc_right and target_uins to the qzone publish path across protocol,
bridge, and the send_qzone_msg OneBot action, regenerate the MCP catalog,
and cover restricted visibility (16/128) with validation tests.
#165: get_record gains an optional `out_format` (mp3/amr/wma/m4a/spx/ogg/
wav/flac, mirroring NapCat). When set, the QQ voice (SILK/AMR) is
transcoded server-side and the result is attached as `data.base64`; the
original info (file/url/file_size) is preserved. Absent ⇒ behaviour
unchanged. This spares ASR consumers from shipping a SILK decoder.
- protocol: convertAudioBytes(bytes, format, deps?) drives the bundled
SILK-capable ffmpeg addon's convertFile through temp files (always
cleaned up), with a format allowlist and an output-size cap. Pure/
injectable, unit-tested with a fake addon (6 cases).
- core: ExtrasApi.convertRecord(source, format) = loadBinarySource (64 MiB
cap) → convertAudioBytes.
- onebot: get_record wires out_format; transcode failure (incl. an addon
lacking SILK decode) degrades to ACTION_FAILED, never crashes.
Adversarially reviewed: no path traversal (format allowlisted), no SSRF
(source is the cached CDN url, not user input), cleanup covers every path,
graceful degradation confirmed. Catalog regenerated. Real-machine e2e
(does the shipped .node actually decode SILK?) is Phase 2.
A face was always built as a legacy FaceElem (face.index). QQ-NT only renders
classic small faces that way; newer 'super'/animated faces sent as face.index
are silently remapped by the server (e.g. 424→168) or shown as a placeholder.
Classify each face id off QQ's own system-face catalog (0x9154_1, ported from
SLE/Lagrange) and pick the matching wire form:
- super (animated, aniSticker pack ≠ (1,1)) → CommonElem 37 + QFaceExtra
- other id ≥ 260 → CommonElem 33 + QSmallFaceExtra
- classic id < 260 → legacy FaceElem
The catalog is fetched once (shared, lazy-warmed off the first face send) and
classify falls back to an id-range guess until it lands. Data-driven so all 66
non-(1,1) super faces encode correctly, not just a hardcoded few.
Verified on a live account: 424→424 (was 168), super 392→392, classic 14→14.
forceFullOnFastPath re-issued the NTV2 upload with tryFastUploadCompleted:
false to try to refresh a possibly-expired video resource. Real-machine
testing on a live account + RE of the QQ NT kernel (wrapper.node) proved it
never works:
- the server's fast-path is driven by md5-metadata existence and ignores
the flag (proton omits a false bool anyway); the retry re-fast-paths
- the kernel itself does no validity check — HandleRspUploadV3 trusts the
response's fileExist and goes straight to SendUploadSuccFTN
So an expired-but-still-indexed video resource is a platform-level
limitation, not something the upload flow can refresh. Removing the retry
drops two wasted OIDB round-trips per cached-video re-send. The underlying
#145 (receiver '资源已过期' for a server-expired resource) remains a
platform issue, tracked separately.
Tests updated to assert the single-request behaviour.
Private (c2c) file sends uploaded the bytes (apply 0xE37_1700 + Highway)
and PbSendMsg returned ok, but the receiver saw "文件传输失败": the message
carried only FileExtra.file (metadata) and not the server-issued download
routing. Group files use a different element, hence c2c-only.
Port NapCat's upload→finalize→send sequence:
- add 0xE37_800 (FinalizeOfflineFile) OIDB service, run after the Highway
PUT to fetch the download metadata (field30)
- extend FileExtra with field6 (PrivateFileExtra.field2) and populate it
from that metadata in sendC2cFile; covers both the url-upload and the
file_id-resend send paths (both funnel through sendC2cFile)
Tests: byte-oracle locks the 0xE37_800 request + FileExtra.field6 tags;
core routing test now asserts finalize runs and field6 is populated.
Needs real-machine verification (send a private file, confirm download).
#153 added a video→file fallback but the group side was dead code: it
lived in element-builder behind `isGroup && element.fileId`, and a video
sent by url/path has no fileId, so oversize group videos just rethrew the
raw Highway error (real bot: `error_code=323` on a 1.4 GB group send).
Move the fallback to the OneBot layer for groups, mirroring the private
path: on a failed Highway send, re-route oversize videos through
groupFile.upload. Verified on a live bot — a 1.4 GB group video now lands
as `[文件:video.mp4]` instead of failing.
Extract the shared heuristic (isHighwaySizeError / videoNeedsFileFallback
/ splitVideoFileFallback) so private and group stay in sync, and drop the
now-unreachable element-builder video catch. Adds send-video-fallback
tests covering both the known-oversize and remote-error-heuristic paths.
Round 3: fix hardcoded 4096 MB in error messages
- Use MAX_VIDEO_SIZE_HARD constant instead of hardcoded 4096 MB
- Both throw sites now derive limit from the same constant
Follow-up to the previous #146 fix, which dropped QQ's compat fallback text by
content-matching the "版本不支持 … 升级" family. RE of the QQ kernel
(wrapper.linux.node) showed that is not how QQ does it:
- the receiver binary contains NONE of these fallback strings, so QQ cannot be
matching by content;
- the on-target wire is [lightApp, text(compat), generalFlags, extraInfo] and the
text element has no distinguishing marker (no pbReserve / attr6Buf);
- QQ NT's kernel codec (msg_codec_mgr) collapses the message to a single ark
element — the sibling text is never emitted. NapCat maps kernel elements 1:1
with no ark-aware skip yet still surfaces only the card, confirming the kernel
drops it upstream.
Switch to that structural rule: when a lightApp/richMsg card is present, drop
sibling plain text outright. It is version/locale-proof (won't break when Tencent
rewords the string) and matches QQ NT + NapCat output exactly. @ mentions /
replies / faces are not plain text and survive; QQ clients can't attach a caption
to a card anyway, so no QQ-origin message loses real text.
Verified live on a logged-in build: get_msg returns a single json segment.
A QQ mini-program / ark share (B站 video, QQ 小程序, …) arrives on the wire
as a lightApp/richMsg ark element followed by a plain text element carrying
QQ's graceful-degradation compat string "当前QQ版本不支持此应用，请升级". That
text is meant only for protocol-old clients that can't render the card; QQ NT
and the kernel-backed bridges (NapCat) surface just the card.
On-target capture (test group) confirmed the text element has no distinguishing
wire marker (no pbReserve / attr6Buf) — it is a bare text sibling of the ark.
SnowLuma was decoding both, so every mini-app card came through as the card
plus a spurious "当前QQ版本不支持" text segment.
Drop the compat string when it sits beside a card, matched narrowly (the
"版本不支持 … 升级/更新" family, <=40 chars) so genuine user captions sent
alongside a card are preserved.
Verified live on a logged-in build: get_msg now returns a single json segment.
Group/c2c video resources expire server-side. When the bot re-sends a
video whose md5 the server already cached, the NTV2 OIDB returns no uKey
for the main file (fast-upload hit) and SnowLuma trusted it — the send
"succeeds" but the receiver sees 资源已过期, because the reused object had
expired. Confirmed by the reporter: same MP4 fast-paths and breaks, but
mutating one byte (new md5) forces a full upload and plays fine.
Add a per-sub `forceFullOnFastPath` flag on MediaSubFileUpload. When a
flagged sub-file (video main) carries real bytes but the server fast-paths
it, runNtv2Upload re-issues the OIDB request with tryFastUploadCompleted
disabled (proton drops a `false` bool, which the server reads as "no
fast-upload"), forcing a fresh resource + full PUT. The thumb stays
fast-path-OK so a cached thumb never re-pushes the main video; forwarding
paths carry no bytes so they never trigger the retry.
Also fixes the observability gap the reporter hit: the main file being
silently skipped on a fast-path now logs explicitly, and so does the
forced-retry, instead of only the thumb PUT showing up.
Note: the retry relies on the server treating an absent
tryFastUploadCompleted as "no fast-upload" — needs real-device
confirmation that the forced round actually returns a uKey.
Inviting an official robot (e.g. 2854207029) into a group made QQ push the
group_member_increase notice twice, so SnowLuma emitted two
notice.group_increase events; a normal member produced one.
Root cause (RE of wrapper.linux.node): QQ NT dedups every received system
message in sys_msg_mgr.cc::ProcessRecvSysMsg (sub_37BADE0) by a global_key
`{peerUid}_{chatType}_{msg_seq}_{random}` (sub_37BC4E0) — an already-seen key
is dropped ("ignore dup msg") before any listener sees it. The per-message
discriminators msg_seq (attr 40003 = contentHead field 5) and random (attr
40002 = contentHead field 4) are exactly head.sequence and head.msgId
(msg_header_codec_helper.cc sub_37C8AD0). Kernel-based bots (NapCat/LLOneBot)
receive events after this dedup; SnowLuma reads the raw OlPush before it.
Mirror it: SysMsgDedup, a per-account bounded ring that drops a duplicate
system push by (msgType, subType, fromUin, sequence, msgId). Push-level (drops
the whole duplicate like NT), live path only (Bridge owns the tracker; the
forward re-parse passes none), system kinds only (chat messages excluded), and
never dedups a push lacking a server identity (sequence or msgId 0).
The server never pushes a member-decrease back to the member that left of its
own accord, so set_group_leave produced no notice.group_decrease — downstream
kept a 'zombie' group in its cache, and a later set_group_add_request failed
with 'matching group request not found'.
bridge.leave() now, after the 0x1097_1 OIDB succeeds, synthesizes a self
group_member_leave (user_id = operator_id = self, sub_type 'leave') through the
event bus — converted to notice.group_decrease by the existing path — and
calls identity.forgetGroup() so get_group_list / member lookups drop it
immediately (in-memory + DB inactive).
Kick / disband already arrive via the GroupMemberDecrease push (PkgType 34).
Limitation: a MANUAL leave in the NTQQ client isn't covered — SL is packet-
based with no kernel callback, and a voluntary self-leave isn't pushed back;
only kernel-based clients (NapCat/LLOneBot) catch that.
Validated by unit tests (bridge leave emits the event + forgets the group).
The 0x8FC_2 body omitted uinName (tag 7), so the server accepted the request
(errorCode 0) but silently never applied the title — looking like the feature
was unsupported. Both NapCat (Oidb.0x8FC_2: uinName=title) and Lagrange
(0x8FC body @7 UinName=title) set it; mirror them by sending uinName = title.
No RE needed — two independent packet-based implementations agree on the field.
Note: setting a special title is group-owner-only, so on-target validation
requires a group the bot owns.
Harvests the sound ideas from PR #128 while KEEPING origSeqs[0] as the c2c
reply sequence (on-target capture in #114/#124 proved origSeqs[0] == the quoted
message's head.sequence; pbReserve.friendSequence is a small unrelated counter
that makes reply.id miss). The PR's friendSequence override is intentionally
NOT taken.
- rich-body-decoder: enrich the reply element with replySenderUin/replyTime and
the quoted message's own decoded elements (SrcMsg.elems, field 5); recover a
c2c quoted FILE from sourceMsg (field 9 -> RichText.notOnlineFile) when elems
carried none, so a quoted file survives into get_msg (#124).
- backfillReplyTarget: tiered fallback — Tier 0 local store, Tier 1 server fetch
(correct seq = origSeqs[0]), Tier 2 reconstruct from the embedded quoted
elements (no round-trip), Tier 3 minimal '[引用消息]' placeholder so get_msg
never returns 'message not found'.
- to-segment: per-element try/catch (logged, not silent) so one bad element
doesn't drop the whole message.
- actions/message: log get_msg misses.
- tests: fix request-actions getGroupInviteCardSequence mock (gap from #125);
cover backfill Tier 2 (reconstruct) + Tier 3 (placeholder); flip the stale
c2c reply test to assert origSeqs[0] (was asserting the disproven
friendSequence).
Validated on-target (Docker): get_msg on group reply, private text reply, and a
private quoted FILE all return correct content (file with name/size/id); full
onebot + protocol suites green.
A QQ NT reply carries the replied sender as a structural auto-mention
(MentionExtra.type=2, uin=0) right after srcMsg, plus a blank separator
text. SnowLuma reported both as a real [CQ:at] + empty/space segment, so
downstreams saw a phantom @ (#127). A real user @ carries a non-zero
MentionExtra.uin, so it is preserved.
- receive (rich-body-decoder): after a reply, drop a type=2/uin=0 mention
and the blank text that follows it; keep non-zero-uin mentions.
- send (element-builder, #129 follow-up): the reply auto-mention added for
reply+@ now uses uin=0 (real uid stays in `uid`) to match QQ's native wire
shape, so the bot's own replies are stripped the same way by receivers and
don't surface a phantom @ either.
Validated on-target (Docker, group): reply w/o @ -> [reply, text] (no
phantom @/empty); reply w/ @ -> [reply, at(real), text]; bot reply+@ still
delivers the @ (uin=0).
When a group message combined a reply with a user @ , the @ showed but
never notified (no blue link, no ping). On-target capture showed SnowLuma
sent [srcMsg, @target(mention), text] — the @ lands directly in the reply's
mention slot right after srcMsg, which the QQ NT group server does not honor.
Mirror Lagrange's ForwardEntity.PackElement: when a reply and an @ coexist
in a group message, emit the replied sender as a trailing mention element
right after srcMsg, so the user's @ is no longer the first post-srcMsg
element and is honored. Gated to the reply+@ case so plain replies are
unchanged.
Validated on-target (Docker, group send): baseline reply+at -> @ not
delivered; fixed -> @ delivered (blue link + notification).
A private (c2c) reply resolved to srcMsg.reserve.friendSequence, but a
received message's message_id is hashed from its head.sequence. On-target
capture showed origSeqs[0] == head.sequence (e.g. 12709) while friendSequence
is a small friend-relationship counter (e.g. 25) — so reply.id != the quoted
message_id. Downstream get_msg(reply_id) then missed:
- #114: reported reply id didn't match the quoted message's real id.
- #124: replying to a File message returned empty quoted content (AstrBot
relies on get_msg(reply_id) to learn the quote was a file; text replies
appeared fine because they read the inline follow-up text instead).
Drop the c2c friendSequence override (reverting cf8e62d's premise, which was
backwards) so both group and c2c replies use origSeqs[0].
Validated on-target (Docker): file message id == reply.id, and
get_msg(reply_id) -> retcode 0, message [{type:file}].
When the bot is invited to a group via a private-chat "qun.invite" ark
card, set_group_add_request failed: handleGroupAddRequest re-derived the
approval tuple from fetchGroupRequests (0x10c8_2), but for a self-invite the
only sequence the server accepts at 0x10c8_1 is the msgseq embedded in the
card's mqqapi://group/invite_join jumpUrl, applied with eventType=2 /
filtered=false. The MSF invite push (PkgType 87) never carries that msgseq,
so approval returned 120161001 'handle async message fail'.
- friend-message decoder: parse the private invite card's jumpUrl and stash
groupUin -> msgseq on the per-instance IdentityService.
- ContactsApi.getGroupInviteCardSequence(groupId) exposes it.
- handleGroupAddRequest: for an invite flag, if a card msgseq is cached,
approve with setAddRequest(groupId, msgseq, 2, approve, reason, false);
otherwise fall back to the existing fetchGroupRequests path.
Validated on-target (Docker): private invite card -> set_group_add_request
approve -> 0x10c8_1 ok -> bot joined the group (get_group_member_info shows
role=member, fresh join_time); no 120161001.
Large private files (e.g. 48 MB .flac via send_private_msg) failed with
"connection closed before response (received 0 bytes, headerSeen=false)".
uploadHighwayHttp sends one fresh TCP connection per 1 MB chunk with no
retry, so a 48 MB file is ~48 sequential connections and a single transient
peer-FIN / reset among them aborted the entire upload. Small files (1 chunk)
always worked; NapCat is unaffected because it delegates to QQ's native
highway client, which retries.
Wrap each chunk's connect + POST in a bounded retry (3 attempts, 300 ms ×
attempt backoff, fresh connection each time). Re-sending the same offset
range is idempotent server-side. Only transport failures retry; a decoded
highway error_code is a definitive server reject and is not retried.
DrSmoothl