codex/async-sqlite-node24
121
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
c8f259ef4a |
test(highway): pin multi-image sequential semantics + richer 921 diag
User reported `highway upload error_code=921` when sending two images
in one OneBot message — image 1 fast-uploaded (server already had the
md5), image 2 (58 bytes) triggered the actual PUT and failed.
Verified the claim from code inspection vs NapCatQQ + Lagrange.Core:
- dev/NapCatQQ/.../highwayContext.ts:62-94 (checkAvailable + prepareUpload)
- dev/NapCatQQ/.../highwayHttpUploader.ts:8-43 (HTTP block upload)
- dev/NapCatQQ/.../highwayContext.ts:219-254 (uploadGroupImage)
- dev/Lagrange.Core/.../HighwayContext.cs:98-180 (UploadSrcByStreamAsync)
Findings:
1. The pipeline runs `runNtv2Upload` sequentially per image (await
inside `buildSendElems`); there's no concurrency or shared state.
2. Each call gets its own OIDB request, its own lazy session fetch,
its own buildHighwayExtend output. No aliasing.
3. The 921 comes from the highway PUT's response head (errorCode),
not from anything client-emitted being malformed.
Verdict: not reproducible from code inspection — almost certainly
server-side (most likely Tencent rejecting the 58-byte image as a
malformed payload; 58B is too small to even be a valid 1×1 PNG).
Hardening:
- Three regression tests pin the sequential two-image semantics so
a future state-leak bug fails loudly:
- extend bytes for image 2 must come from image 2's response,
not image 1's,
- each runNtv2Upload fetches its own non-null session,
- each OIDB request carries its own fileInfo + a unique
clientRandomId.
- The `error_code=N` message now carries segRetCode, cmdId, chunk
size, offset, and the first 8 bytes of the file md5 — without
these the user couldn't tell apart "image rejected" vs "session
expired" vs "rate-limited" on future reports.
If the user reports the same 921 again, the new diagnostic message
should show enough server-side context to triage server-vs-client.
|
||
|
|
cd9403f27a |
fix(forward): align resid+uniseq for nested forwards
Multi-layer 合并转发 was uploading correctly but the outer's
LongMsgResult piggyback was *unreachable* by the receiver because
two things were missing:
1. The forward preview element was emitted as the old XML
`richMsg serviceID=35 m_resid=…` shape, which carries the
inner resId but has NO uniseq field.
2. The piggyback `actionCommand: <uuid>` slot generated a fresh
UUID that was never reflected into the preview either.
Result: QQ-NT receivers saw the inner preview's resid, scanned the
outer's actions[] for a matching uniseq, found none, and either
fell back to a fresh server fetch per layer or dropped the layer
entirely.
Cross-checked against:
- dev/Lagrange.Core/Lagrange.Core/Message/Entity/MultiMsgEntity.cs:43-115
- dev/NapCatQQ/packages/napcat-core/helper/forward-msg-builder.ts:52-122
- dev/NapCatQQ/packages/napcat-onebot/action/msg/SendMsg.ts:208-347
Both upstream impls use the modern `com.tencent.multimsg` LightApp
JSON whose `meta.detail.uniseq` is the round-trip key that links
the inner preview to the matching `actionCommand` in the outer's
LongMsgResult — no separate fetch needed.
Changes:
- `element-builder.makeForwardElem` now emits a deflated
`com.tencent.multimsg` LightApp JSON (LightAppElem, field 51)
instead of the XML RichMsg (field 12). `forwardUuid` on the
MessageElement is baked into `meta.detail.uniseq` and
`extra.filename`. Auto-generates a UUID when absent.
- `MessageElement.forwardUuid?: string` added for the round-trip.
- `forward.uploadRecursive` threads `inner.uuid` into the outer's
preview element AND uses the same uuid for the outer's
piggyback actionCommand — closing the receiver-side lookup loop.
- `forward.fetch` now walks ALL actions (not just `MultiMsg`),
indexes piggybacks by uuid, and pre-populates the cache for
each inner resId discovered in the outer's preview bubbles.
Follow-up `fetch(innerResId)` hits the cache instead of the wire.
- `rich-body-decoder` parses LightApp JSON; when `app ===
com.tencent.multimsg`, emits `{type: 'forward', resId,
forwardUuid}` so the OneBot layer can walk in. Legacy XML
serviceID=35 path stays as a fallback for mobile QQ senders.
Wire-byte tests cover:
- send: lightApp.data is 0x01-prefixed deflated JSON with the
right meta + extra fields
- send/nested: outer LongMsgResult carries `MultiMsg` AND a
uuid-keyed piggyback; the piggyback uuid matches the
preview's `meta.detail.uniseq` baked into the same outer
msgBody
- send/3-deep: outermost action[] has 1 + 2 entries (own
MultiMsg + 2 piggybacks hoisted from deeper levels)
- recv: lightApp `com.tencent.multimsg` → forward element with
both resId and forwardUuid; non-multimsg LightApp falls
through to {type:'json'}; legacy XML path still emits
{type:'forward'}.
641 tests pass (was 628 + 13 new); typecheck clean; lint clean.
|
||
|
|
5724f490b4 |
refactor(oidb): thread ctx into serialize/deserialize for self-contained namespaces
The first pass at namespace-per-cmd left a smell: namespaces that
needed identity/uid resolution had to declare `Deps` AND then rewrite
their own spec inside `invoke` via a closure-over-resolved-value:
// before
export const serialize = (p: Params, targetUid: string) => ({...});
export const invoke = async (deps, params) => {
const targetUid = await deps.resolveUserUid(params.userId);
return invokeOidb(deps, {
...Self,
serialize: p => serialize(p, targetUid), // ← escape hatch
}, params);
};
Reading "what does this cmd do on the wire?" required reading the
namespace AND the closure rewrite — the namespace wasn't really
self-contained, just a forwarder of two pure functions.
Fix: thread `ctx` (the namespace's full capability surface) into
`serialize` and `deserialize` directly. `serialize` may now return
`Promise<TReq>` so identity / uid lookups happen in place:
// after
export const serialize = async (ctx: Deps, p: Params) => ({
groupUin: p.groupId, type: 1,
body: {
targetUid: await ctx.resolveUserUid(p.userId, p.groupId),
duration: p.duration,
},
});
export const invoke = (deps, params) => invokeOidb(deps, Self, params);
OidbCallSpec gains a `TCtx extends OidbSender` generic; invokeOidb's
first arg becomes `ctx: TCtx` (the sender capability is just a slice
of ctx now). buildOidbRequest / parseOidbResponse follow the same
shape. Async serialize is awaited inside invokeOidb.
17 namespaces had closure rewrites; all 17 collapse to plain
`invokeOidb(deps, Self, params)` invokes. Only one closure remains —
`fetch-user-profile` binds `params.uin` into deserialize as a server-
echo fallback; kept as a documented exception (the rare case where
deserialize wants request context, not ctx).
Wire bytes preserved across all migrations. 326 bridge tests + 302
core tests pass; lint clean.
|
||
|
|
9b4ff9c5e9 |
chore(lint): exempt oidb-services from no-namespace + clean stale directives
The namespace-per-cmd OIDB service layer was tripping
`@typescript-eslint/no-namespace` across 59 files; namespace IS
the OidbCallSpec by design (structural typing, one file per
(cmd, subcmd)) so disable the rule for that subtree only. Drop
four stale `eslint-disable-next-line @typescript-eslint/no-empty-
object-type` comments that aren't needed anymore (the empty
`Params {}` shape no longer triggers the rule), and fix an indent
nit in the get-unidirectional-friend-list test.
`npx eslint packages` now reports 0 errors. The 5 remaining
warnings are unused-vars / unused-eslint-disable in unrelated
pre-existing files.
|
||
|
|
521808a6eb |
refactor(oidb): migrate group-file.ts OIDB cmds to namespaces
Splits the 16 OIDB cmds from GroupFileApi into self-contained
namespaces under @snowluma/bridge/oidb-services/group-file/. Adds
@snowluma/bridge/oidb-services/shared with `toInt` / `ensureRetCodeZero`
helpers so namespaces can validate sub-responses without an
@snowluma/core dependency.
Boundaries:
- Single-OIDB methods (publish / getCount / list-page / *url /
delete / move / *folder) become one-line forwarders.
- 3-stage methods (`upload` / `uploadPrivate`) keep their
orchestration on the facade — OIDB preflight goes through the
new namespace, but highway PUT + publish/sendC2cFile + upload-
cache bookkeeping stay where they are.
- `list` keeps its pagination loop on the facade; the page-fetch
namespace is one call.
- The NTV2-rich-media URL fetchers (group/private × ptt/video)
share a deserialize helper (`parseNtv2DownloadUrl`) so all four
namespaces only differ in their scene block.
Wire-byte equivalence preserved. The 0x6D6 family shares one
OidbGroupFileReq envelope; each namespace populates a different
slot (file / download / delete / move) and proton monomorphizes
per call site as before.
|
||
|
|
8d22868777 |
refactor(oidb): migrate group-admin.ts OIDB cmds to namespaces
Splits the 14 OIDB cmds from GroupAdminApi into self-contained
namespaces under @snowluma/bridge/oidb-services/group-admin/. The
facade becomes one-line forwarders; per-cmd capability is narrowed
to OidbSender (+ resolveUserUid / identity where the cmd actually
needs it).
Notes:
- 0x89A_0 hosts MuteAll, SetAddOption, SetSearch, SetGroupName —
same (cmd, subcmd) tuple, four different proto bodies; each gets
its own namespace so proton can monomorphize independently.
- 0x8A0_1 splits KickMember (single) and KickMembers (batch) for
the same reason.
- SetAddRequest carries uinForm=true and dynamic subCommand
(filtered ? 2 : 1) on the same namespace.
- GetAtAllRemain needs the bot's own uin → declares
Pick<BridgeContext, 'identity'> in its Deps.
Wire-byte equivalence preserved; corner case for getAtAllRemain
empty-envelope path now returns {false, 0, 0} instead of throwing —
the new infrastructure normalises empty bodies and the OneBot action
ergonomics are unchanged.
|
||
|
|
014b87c7ce |
refactor(oidb): migrate contacts.ts OIDB cmds to namespaces
6 OIDB cmds moved; pagination + per-group cache + inflight
coalescing stay on the facade because they need cross-call state.
- FetchFriendListPage (0xFD4_1) — facade drives the cursor loop
- FetchGroupList (0xFE5_2) — uinForm; verbatim config blob
- FetchGroupMemberListPage (0xFE7_3) — facade drives the token loop +
owns the TTL cache + inflight
coalescing that keeps busy
OneBot clients from tripping
Tencent risk-control
- FetchUserProfile (0xFE1_2) — uinForm; full property-key
catalogue cross-checked against
LagrangeV2 FetchStrangerService.cs
- FetchGroupRequests (0x10C0_1/_2) — dynamic subcmd via filtered flag
- FetchDownloadRkeys (0x9067_202) — uinForm; respHead.retCode
check moved into deserialize
35 new namespace-level tests covering serialize / deserialize edge
cases (property decoding, sex enum mapping, avatar URL suffix,
empty body handling, pagination cursor behaviour).
Verification:
- typecheck green
- @snowluma/bridge: 30 files / 233 tests
- @snowluma/core: 302 tests
Wire bytes unchanged.
|
||
|
|
80afa31f5c |
refactor(oidb): migrate extras.ts OIDB cmds to namespaces
5 logical OIDB cmds → 4 namespace files (group todo's 3 sub-cmds
collapse into one namespace with a dynamic resolver):
- GroupTodo (0xF90_1/2/3) — set/complete/cancel via the
new `action` discriminator
- GetStrangerStatus (0xFE1_2) — uinForm=true; preserves the
(status*10, ext_status) decode
quirk verbatim
- FetchAiVoiceList (0x929D_0)
- FetchAiVoice (0x929B_0) — single poll; retry loop
stays on the facade so callers
can plug their own back-off
Facade keeps the 6 public methods (setGroupTodo / completeGroupTodo /
cancelGroupTodo / getStrangerStatus / fetchAiVoiceList / fetchAiVoice)
as one-line forwarders; only `getStrangerStatus` wraps in try/catch
to preserve its "return null on transport failure" contract, and
`fetchAiVoice` keeps the polling loop.
30 new namespace-level tests; extras.test.ts moved onto the
sendRawPacket assertion model.
Verification:
- typecheck green
- @snowluma/bridge: 24 files / 195 tests
- @snowluma/core: 302 tests
Wire bytes unchanged.
|
||
|
|
8356121518 |
refactor(oidb): migrate web.ts OIDB cmds to namespaces
2 OIDB cmds moved (the only OIDB calls in this file — the rest are
HTTP cookie / qun.qq.com / qzone web requests):
- ForceFetchClientKey (0x102A_1) — used by ptlogin2 jump flow to
swap clientKey → cookie jar
- GetPskey (0x102A_0) — p_skey fallback when ptlogin2
omits it
The private `forceFetchClientKeyInner` / `getPSkey` helpers in
WebApi are now one-line forwarders into the namespaces. Facade
methods (`getCookies`, `getCsrfToken`, `getCredentials`, etc.) and
all HTTP-only cmds untouched.
12 new namespace-level tests covering deserialize edge cases
(empty server response, missing fields, dropped invalid items)
and e2e wire routing.
Verification:
- typecheck green
- @snowluma/bridge: 20 files / 165 tests
- @snowluma/core: 302 tests
Wire bytes unchanged.
|
||
|
|
58e8081420 |
refactor(oidb): migrate misc.ts OIDB cmds to namespaces
3 OIDB cmds migrated; the 1 non-OIDB cmd (getMiniAppArk via
LightAppSvc) stays on the facade unchanged.
- TranslateEnToZh (0x990_2)
- ClickInlineKeyboardButton (0x112E_1)
- SendGroupSign (0xEB7_1) — preserves historic UPPERCASE
EB7 wire name via the new
`wireName` override (the only
cmd in the codebase with this
quirk; every other cmd uses
the default lowercase).
Tests:
- misc.test.ts moved to bridge.sendRawPacket assertion model
- 18 new namespace-level tests covering serialize, deserialize edge
cases, magic constants, BigInt widening, and the EB7 case quirk
Verification:
- typecheck green
- @snowluma/bridge: 18 files / 153 tests
- @snowluma/core: 302 tests
Wire bytes unchanged.
|
||
|
|
f78431242c |
refactor(oidb): migrate friend.ts to namespaces
3 OIDB cmds moved to self-contained namespaces:
- HandleFriendRequest (0xB5D_44) — digit-only flag still
auto-resolves via resolveUserUid
- DeleteFriend (0x126B_0) — verbatim magic constants in the
nested field2 sub-message
(130/109/8/8/50 — server-side
discriminator, MUST stay byte-
equal to NTQQ wire)
- SetFriendRemark (0xB6E_2)
Cache invalidation (fetchFriendList after deletion) stays in the
facade — that side-effect needs `ctx.apis.contacts`, which namespaces
deliberately can't reach.
Tests migrated to assert against bridge.sendRawPacket. 15 new
namespace-level tests covering each invoke branch + serialize edge
cases.
Verification:
- typecheck green
- @snowluma/bridge: 15 files / 133 tests
- friend.test.ts: 5/5
Wire bytes unchanged.
|
||
|
|
a38f7ea26e |
refactor(oidb): migrate profile.ts OIDB cmds to namespaces
5 of profile.ts's methods were thin OIDB wrappers — migrated to the
new namespace pattern. The 5 non-OIDB methods (setOnlineStatus /
setDiyOnlineStatus / setAvatar / setGroupAvatar / fetchCustomFace)
stay on the facade unchanged — they hit `trpc.qq_new_tech.*`,
Highway, or `Faceroam.OpReq` directly and aren't in scope for the
OIDB-namespace refactor.
New namespaces (each with focused Pick<BridgeContext, …> deps):
- SetProfile (0x112A_2) — needs identity for uin
- SetSelfLongNick (0x112A_2) — same cmd different proto, ditto
- SetInputStatus (0xCD4_1) — needs resolveUserUid
- GetLike (0x7ED_12) — needs identity + resolveUserUid
- GetUnidirectionalFriendList (0xE17_0) — needs identity; non-standard
wire name handled via the new
wireName override on OidbCallSpec
Infrastructure tweak:
- OidbCallSpec gains an optional `wireName(cmd, sub) → string`. Most
cmds use the default `OidbSvcTrpcTcp.0xNNNN_N` scheme; 0xe17_0
routes through `MQUpdateSvc_com_qq_ti.web.OidbSvc.0xe17_0`.
Test mock updates:
- profile.test.ts moves the migrated cmds onto bridge.sendRawPacket
mock (the new path bypasses the bridge-oidb runOidb mock). Non-OIDB
cmds (setOnlineStatus, setAvatar, etc.) keep their existing mocks.
- 36 new namespace-level tests in @snowluma/bridge covering serialize
branches, deserialize edge cases, deps resolution, and e2e wire
routing.
Verification:
- pnpm -r typecheck: green
- @snowluma/bridge: 12 files / 118 tests
- @snowluma/core: 33 files / 302 tests
Wire shape unchanged across migrated cmds — same bytes produced.
|
||
|
|
ff2b902cbd |
refactor(oidb): migrate interaction.ts remaining 3 cmds to namespaces
Completes the InteractionApi facade — every method is now a one-line
forwarder. New namespaces:
- SendPoke (0xED3_1) — group + friend variants discriminated by
which of groupUin/friendUin is set
- SendLike (0x7E5_104) — thumbs-up to a target user
- SetEssence (0xEAC_1/_2) — set/unset essence message, dynamic subcmd
Facade methods reduced to thin forwarders, mirror-fixed comment block
removed (the namespaces now own the wire-shape commentary).
Tests: 28 new in @snowluma/bridge covering each namespace's serialize
edge cases + dynamic subcmd resolution + e2e wire name + envelope body
shape. Cleaned the legacy `vi.mock('bridge-oidb')` setup out of
interaction.test.ts — the post-migration path doesn't route through
the mocked symbols anymore.
Wire behavior unchanged: every cmd produces the same bytes as before,
verified by encode round-trip in the unit tests.
|
||
|
|
0238bcb280 |
refactor(oidb): namespace-per-cmd service layer — vertical slice (3 cmds)
Introduce a new way to organise OIDB cmd implementations:
- one TypeScript namespace per (cmd, subcmd) tuple
- the namespace's exported members ARE the OidbCallSpec (structural
typing — no `implements` clause)
- a single invokeOidb() template method drives every call:
envelope build → encode → send → errorCode check → decode →
deserialize transformation
- each namespace declares its own minimal `Deps` capability via
Pick / intersection on BridgeContext (Interface Segregation):
most cmds need only OidbSender (the `sendRawPacket` method)
The facade Api classes (InteractionApi etc.) stay for now, with the
migrated methods reduced to one-line forwarders that hand off to the
namespace. Callers still write `bridge.apis.X.Y(...)`; the OIDB
wire-encoding work just moved to a self-contained file per cmd.
Vertical slice scope: 3 cmds (the reaction line)
- SetReaction (0x9082_1 / _2; dynamic subcmd)
- FetchReactionSummary (0x9084_1)
- GetEmojiLikes (0x9083_1; legacy stub)
Infrastructure:
- new packages/bridge/src/oidb-service.ts:
OidbSender (sendRawPacket-only capability),
OidbCallSpec<TReq, TResp, TParams, TResult>,
invokeOidb(), buildOidbRequest(), parseOidbResponse(), OidbError
- runOidb() refactored to accept OidbSender instead of BridgeContext
(structurally compatible — no caller changes anywhere)
- new OidbEmpty placeholder type in proto-defs/oidb.ts for cmds whose
response carries no body (proton needs a real message interface,
not `void` / `never`)
- empty-body responses substituted with {} in invokeOidb so
deserialize always receives a defined object
Tests (65 in @snowluma/bridge, all green):
- oidb-service.test.ts (20): every helper branch — static + dynamic
subcmd, uinForm flag, error code propagation as OidbError, empty
body substitution, sender failure modes (success=false /
gotResponse=false / both with and without errorMessage),
timeoutMs propagation
- set-reaction.test.ts: const declarations, dynamic subcmd both
ways, type=1/2 boundary on code length, e2e envelope inspection
- fetch-reaction-summary.test.ts: BigInt serialization, catalog-tail
filtering, default emojiId/Type/lastReactionTime
- get-emoji-likes.test.ts: BigInt serialization, default count /
emojiType, base64 cookie round-trip, uin=0 filtering, isLast
computation, e2e
Existing interaction.test.ts moved off the now-bypassed runOidb mock
onto sendRawPacket mock (the new invokeOidb path routes there).
Workspace-wide typecheck: green. Workspace test totals:
proton 99/99, bridge 65/65, core 302/302, onebot 257/257
(1 pre-existing onebot suite failure, network-manager native addon
load — unrelated to this change.)
|
||
|
|
2638d13d11 |
feat(profile): surface QQ level from OIDB 0xFE1_2 number-property 105
`fetchUserProfile` already requested key 105 in the keys[] array but the decode pass never read it, so callers got nothing. NapCat exposes the same value as `qq_level` on `get_stranger_info`; users coming from NapCat noticed the field was missing on the SnowLuma side. Cross-checked against LagrangeV2 Internal/Services/System/ FetchStrangerService.cs which explicitly labels key 105 = Level and uses `numbers[105]` directly. Changes: - UserProfileInfo: add `level: number`. - fetchUserProfile decode: `info.level = numMap.get(105) ?? 0`. - getStrangerInfo: emit both `qq_level` (NapCat-compat) and `level`. - Test/manager call sites updated for the new required field. |
||
|
|
607a7b319b | chore(release): v1.9.1 | ||
|
|
4e51b4bd7d |
build(sqlite): wrap better-sqlite3 with @snowluma/sqlite + vendor prebuilts
Replaces "runtime npm dependency that gets `npm install`-ed into
dist/" with "TS wrapper that loads the native `.node` from
`dist/native/` like every other native binding in the project".
Why the change
--------------
The previous approach (better-sqlite3 declared in
packages/runtime/package.json, materialized by `npm install
--omit=dev` in dist/) worked but produced an artifact that needed
runtime npm install — a step the rest of SnowLuma's native bindings
don't need (websocket / snowluma-hook / ffmpeg are all standalone
`.node` files copied into `dist/native/` by vite-plugin-cp). The
asymmetry was friction: dev mode required `pnpm dist:install`,
build artifacts carried a `dist/node_modules/` tree the other
modules didn't need, and Docker images had to special-case the
glibc-vs-musl ABI lookup that prebuild-install runs.
The new shape:
- `@snowluma/sqlite` (packages/sqlite/) — TS wrapper. Has
`better-sqlite3` as a devDep (for the JS-layer types + the
actual JS wrapper code that vite inlines). At runtime, calls
`createRequire().require()` against
`<dist|repo>/native/better-sqlite3-v<abi>-<platform>-<arch>.node`
and passes the resolved addon to `new Database(path, {
nativeBinding })`.
- `packages/runtime/native/better-sqlite3-v<abi>-<triplet>.node`
— prebuilt binaries vendored into the repo. Two ABIs ship side-
by-side (v127 = Node 22 LTS used by CI, v137 = Node 24 used by
dev boxes) so `process.versions.modules` picks the matching one
at load time.
- `tools/fetch-sqlite-prebuilts.mjs` — fetches all 5 platforms ×
N ABIs from WiseLibs's GitHub releases. Idempotent: existing
`.node` files skip. Accepts extra ABI args:
`node tools/fetch-sqlite-prebuilts.mjs 12.10.0 v127 v137 v115`.
What was unwound
----------------
- packages/runtime/package.json — dependencies removed (runtime
artifact is now self-contained, no `npm install` step needed).
- root package.json — `dist:install` script removed (no node_modules
to install).
- .github/workflows/dev-build.yml — `Bundle runtime dependencies`
step removed.
- .github/workflows/release.yml — kept the `Bundle runtime
dependencies` step (was a no-op already, but harmless safety net
in case a future runtime dep shows up).
- packages/core/vite.config.ts — `better-sqlite3` no longer
external; vite inlines the JS wrapper (+~33 KB to bundle, ~2.84
MB → 2.87 MB total).
Naming
------
Mirrors the existing `websocket-${triplet}.node` convention (flat
dir under `runtime/native/`, dash-separated `<platform>-<arch>`).
Adds a `v<abi>` segment because better-sqlite3 is NOT N-API — one
binary per Node ABI. Naming:
better-sqlite3-v127-darwin-arm64.node Node 22 / macOS arm
better-sqlite3-v127-darwin-x64.node Node 22 / macOS intel
better-sqlite3-v127-linux-arm64.node Node 22 / Linux arm
better-sqlite3-v127-linux-x64.node Node 22 / Linux x64
better-sqlite3-v127-win32-x64.node Node 22 / Windows x64
better-sqlite3-v137-... (5) Node 24 / same 5 hosts
Total: 10 binaries × ~1.9 MB = ~19 MB committed. Aligned with the
existing native-vendoring footprint (websocket ~2 MB × 5, snowluma-
hook ~10 MB × Windows + Linux, ffmpeg ~10-30 MB × 4).
What changes for consumers
--------------------------
Source-side: nothing. `import Database, { type Database, type
Statement } from '@snowluma/sqlite'` is a drop-in for the previous
`import Database from 'better-sqlite3'`. The wrapper re-exports the
same types and the `Database` callable factory has the same
overload set.
Runtime: `node` finds the binding at
`<dist|repo>/native/better-sqlite3-v${process.versions.modules}-
${process.platform}-${process.arch}.node`. If the ABI doesn't
match a vendored binary, the wrapper throws a clear error pointing
at `tools/fetch-sqlite-prebuilts.mjs` with the missing ABI/triplet.
Bumped better-sqlite3 from 11.10.0 → 12.10.0 because v11 only had
prebuilt up to Node ABI v127 (no v137 = no Node 24). v12.10.0
ships both. Both versions are API-compat (the breaking changes
between v11 and v12 were either internal or in error-handling
paths SnowLuma doesn't exercise).
Verification
------------
- pnpm -r typecheck (12 projects): clean
- @snowluma/core test: 302/302
- @snowluma/onebot test: 250/250 (552/552 combined, unchanged from
the perf-series baseline)
- Bench numbers preserved — wrapper indirection is sub-μs:
storeMeta ~11 μs, storeEvent ~17 μs, findMeta hit ~1.1 μs (all
within noise of the post-RoadMap-#5 measurements).
- Vite build clean, bundle = 2.87 MB / 230 KB gzip
- `dist/index.mjs` no longer contains a `node:sqlite` import (it
did before — media-store hadn't migrated; now neither it nor
message-store reach into the Node builtin).
Bonus side-fix: `packages/onebot/src/media-store.ts` also moved
off `node:sqlite` (caught by the post-build bundle grep — the
file was using `DatabaseSync` + `StatementSync` from the builtin
even though it had already been written to cache its prepared
statements). Its hot paths now go through the same wrapper.
|
||
|
|
a70f194218 |
perf(msg-push): drop Buffer.from() wrap on pkt.body (free allocation)
`PacketInfo.body` is typed as `Uint8Array` in `@snowluma/common/ protocol-types` and arrives as one — the `Buffer.from(uint8Array)` wrap that fed it into `protobuf_decode<PushMsg>(...)` was a no-op copy. `Buffer` is a `Uint8Array` subclass, and proton's decoder accepts the parent type directly, so the wrap added one allocation per incoming push packet without changing behavior. Bench delta (parseMsgPush, pure decode): group_message (text only) 0.29 μs/op → 0.25 μs/op (~1.2x) group_message (5 mixed) 0.50 μs/op → 0.47 μs/op (~1.07x) group_member_join 0.25 μs/op → 0.30 μs/op (within noise) Small absolute gain — but `parseMsgPush` runs once per incoming packet at the very front of the pipeline, so the saved allocation compounds with the per-message GC churn. The text/mixed group- message cases pick up cleanly; the join case sits in noise (it has a smaller body so the Buffer wrap was cheap relative to total). Verification: pnpm -r typecheck clean; 552/552 tests pass. |
||
|
|
0b6541ca36 |
perf(sqlite): swap node:sqlite for better-sqlite3 + cache prepared stmts
Replaces the experimental `node:sqlite` builtin with the stable
`better-sqlite3` wrapper across the two hot SQLite users — the
OneBot message-store and the bridge identity-service. While the API
surface is essentially identical (sync, prepared-statement-driven,
`Database`/`Statement` shape), better-sqlite3 (a) doesn't carry the
"ExperimentalWarning: SQLite is an experimental feature" baggage
that ships with `node:sqlite`, (b) has a more mature C++ binding
that consistently outperforms the builtin by ~30-50%, and (c)
encourages reusing `Statement` objects by design — exactly the win
the second half of this commit is after.
Why both at once
----------------
Caching prepared statements (was P1 in the RoadMap-#5 analysis) is
the largest single optimization opportunity on the SQLite path. The
old shape was the textbook SQLite anti-pattern:
// every call re-parsed the SQL
this.db.prepare(BIG_SQL_STRING).run(...);
Each `prepare(sql)` runs the full SQL parser + planner — for simple
inserts/updates/selects that dominates wall-clock cost. Doing both
the driver swap AND the statement caching in one commit lets the
benchmark numbers attribute the combined effect honestly; splitting
them would have double-charged the same disruption budget.
What changed code-wise
----------------------
**messageStore** — 8 distinct SQL templates, all now `readonly`
fields prepared once in the constructor:
- stmtStoreEvent — INSERT/UPDATE event row + JSON payload
- stmtStoreMeta — INSERT/UPDATE meta-only row
- stmtFindEvent — single-row SELECT by message_hash
- stmtFindMeta — single-row SELECT (full meta columns)
- stmtResolveReplyGroup, stmtResolveReplyPrivate
— 2 variants for the reply-id lookup
- stmtListEventsAnchored, stmtListEventsLatest
— 2 variants for /get_history_msg
**identity-service** — 25+ distinct statements. Used a Map-based
`pstmt(sql)` memoization helper rather than enumerating all 25 as
named fields (call sites stay readable; Map lookup is O(1) hash +
string-eq, negligible next to the SQL parse it replaces). The
helper short-circuits to the cached `Statement` on second-and-later
calls.
API differences from node:sqlite (handled in this commit):
- `import { DatabaseSync } from 'node:sqlite'`
→ `import Database from 'better-sqlite3'`
- `new DatabaseSync(path)` → `new Database(path)`
- `db.exec('PRAGMA journal_mode = WAL')`
→ `db.pragma('journal_mode = WAL')`
- `db.exec('BEGIN' / 'COMMIT' / 'ROLLBACK')` — kept as-is; both
drivers accept raw `exec()` for transaction control. The
`db.transaction()` higher-order wrapper is more idiomatic
but the existing manual control already works and changing
it would invalidate the `transaction()` helper's matched
try/catch shape.
vite build wiring:
- `node:sqlite` removed from the externals list
- `better-sqlite3` added — must stay external because it has a
native binding (`require('./build/Release/better_sqlite3.node')`)
that Vite can't bundle. Vendoring the prebuilt `.node` into
`packages/runtime/native/` is a follow-up packaging task; the
workspace `pnpm install` already places the binding under
`node_modules` so dev + tests work today.
Bench results (RoadMap #5 baseline → after)
-------------------------------------------
@snowluma/bridge — recv pipeline (with SQLite identity writes):
group_member_join 17.5 μs/op → 4.7 μs/op (~3.7x faster)
group_message text 0.42→0.42 μs (no path change)
group_message mixed 0.67→0.67 μs (no path change)
@snowluma/onebot — send pipeline:
messageStore.storeMeta 17.0 μs/op → 10.9 μs/op (~1.6x)
messageStore.storeEvent 23.2 μs/op → 18.2 μs/op (~1.3x)
messageStore.findMeta hit 5.4 μs/op → 1.3 μs/op (~4x)
messageStore.findMeta miss 3.7 μs/op → 0.7 μs/op (~5x)
messageStore.findEvent 3.8 μs/op → 1.7 μs/op (~2.3x)
Read paths gained the most because they're parse-bound (small SQL
+ short execution → the parser was most of the cost). Writes gained
less because the fsync/journal overhead on each commit still
dominates the parse savings. Combined send hot path (storeMeta +
storeEvent, called on every outbound message + the /get_msg-fix
self-event store) drops from ~40 μs to ~29 μs — about a 27%
reduction per message sent.
Verification
------------
- pnpm -r typecheck: all 11 projects clean
- @snowluma/onebot test suite: 250/250
- @snowluma/core test suite: 302/302
- Combined: 552/552 (unchanged baseline)
- Vite build (with stubbed native binary): clean, 2.84 MB / 223 KB
gzip (matches pre-Phase-2 envelope after removing the better-
sqlite3 inline-bundle attempt — externalizing it shaves the
~30 KB regression we saw mid-flight)
Follow-up (next commits in this series)
---------------------------------------
- Skip rememberGroupMemberIdentity DB write when the (uid, uin,
nickname, card) tuple is unchanged in the in-memory cache.
- Drop the `Buffer.from(pkt.body)` allocation in msg-push context.
- Synchronous fast-path for elementsToJson when no async resolver
is wired.
- Vendor `better-sqlite3` prebuilt `.node` into the runtime/native
layout for the production package step.
|
||
|
|
f3f5d27c65 |
perf(bench): bench harness for receive/send chains (RoadMap #5 baseline)
vitest-driven benches at `packages/bridge/bench/recv.bench.ts` and
`packages/onebot/bench/send.bench.ts`. Picks up proton's vite plugin
so call sites get the inlined codecs (running via tsx + proton
runtime-fallback would throw without a pre-built runtime map).
Run via `pnpm -F @snowluma/bridge bench` and
`pnpm -F @snowluma/onebot bench`.
What's measured
---------------
Receive chain:
- parseMsgPush: pure decode + dispatch (no SQLite, no event emit)
* group_message text ~ 0.29 μs/op (3.4M ops/sec)
* group_message 5 mixed ~ 0.50 μs/op (2.0M ops/sec)
* group_member_join ~ 0.25 μs/op (4.1M ops/sec)
- IncomingPacketPipeline.process: full pipeline (with SQLite via
rememberGroupMemberIdentity, BridgeEventBus emit)
* group_message text ~ 0.42 μs/op (2.4M ops/sec)
* group_message 5 mixed ~ 0.67 μs/op (1.5M ops/sec)
* group_member_join ~17.5 μs/op ( 57K ops/sec) ⚠
Send chain:
- parseMessage (OneBot input → MessageElement[])
* CQ string text ~ 0.31 μs/op
* CQ string mixed ~ 1.22 μs/op
* segment array text ~ 0.32 μs/op
* segment array 5 ~ 1.07 μs/op
- messageStore SQLite writes:
* storeMeta ~17.0 μs/op ( 59K ops/sec) ⚠
* storeEvent (~250B JSON) ~23.2 μs/op ( 43K ops/sec) ⚠
- messageStore reads:
* findMeta hit ~ 5.4 μs/op (186K ops/sec)
* findMeta miss ~ 3.7 μs/op (272K ops/sec)
* findEvent hit ~ 3.8 μs/op (262K ops/sec)
Headline finding
----------------
Pure decode/encode is already cheap (sub-μs). The real cost per
sent message is the SQLite write pair (storeMeta + storeEvent =
~40 μs). On receive, group_message itself does NOT trigger
identity writes; only join/leave/admin events do — but those go
through 4-6 db.prepare(sql).run/get(...) calls each, all re-parsing
SQL on every invocation (the canonical SQLite anti-pattern).
The follow-up commits in this series target:
1. Replace `node:sqlite` (experimental, API may shift) with
`better-sqlite3` (stable, sync, ~30-50% faster wire) + cache
prepared statements as readonly fields. Single commit, both
wins together — better-sqlite3's API encourages stmt caching
by design. Expected: storeMeta / storeEvent drop to ~3-5 μs,
group_member_join drops to ~3-5 μs.
2. Skip rememberGroupMemberIdentity DB write when the in-memory
uidByUin / uinByUid cache already has the mapping unchanged.
3. Drop the `Buffer.from(pkt.body)` allocation in msg-push
context (pkt.body is already Uint8Array; proton accepts it).
4. Make `elementsToJson` sync-fast-path when no async resolvers
are wired (saves per-element microtask overhead).
Each commit re-runs these benches and the message should quote the
before/after numbers so the deltas are auditable.
Note: tsx is NOT involved — vitest's bench mode owns the run, and
the proton vite plugin is configured in each `vitest.config.ts`.
Absolute numbers reflect the production code path.
|
||
|
|
f7a03ca05a |
refactor(bridge): extract protocol layer into @snowluma/bridge package
Phase 2 of the 4-package split (RoadMap #3). Moves ~35 files — events / event-bus / identity-service / packet-pipeline / element- builder / bridge-oidb / qq-info / msg-push/* / highway/* / web/* + the bridge-side formatters — out of `packages/core/src/bridge/` into the new `@snowluma/bridge` workspace package. Everything that translates QQ wire bytes ↔ typed events/elements now lives there. Why --- - `packages/core/src/bridge/` was actually two layers: a low-level "protocol decoder + I/O" set (msg-push, highway, packet-pipeline, identity, element-builder) AND the Api-class host (bridge.ts + apis/*). The Api layer can't move without the protocol layer moving first, but the two had grown into one directory. - Bridge-internal consumers (highway uploaders, OIDB envelope helper, element-builder, msg-push parsers) only ever needed `bridge.identity` + `bridge.sendRawPacket` + the upload-metadata cache. Splitting them out lets that minimal surface live in the bridge package as `BridgeContext` while the api-layer's extended context (with `apis: ApiHub`) stays in @snowluma/core. What stays in @snowluma/core (apis layer): - bridge.ts ── the god-class that composes the protocol building blocks + mounts apis - bridge-interface.ts ── public façade for OneBot consumers - manager.ts ── per-uin BridgeManager hub - apis/* ── the 13 typed Api classes - bridge-context.ts (NEW) ── extends @snowluma/bridge's slim BridgeContext with `apis: ApiHub` so the 13 Api classes keep their typed `this.ctx.apis.X.method()` access. Layout ------ Moved via `git mv` (history preserved): events / event-bus / identity-service / packet-pipeline / element-builder / bridge-oidb / qq-info / msg-push/* / highway/* / web/* → @snowluma/bridge/<sub> utils/event-format.ts → @snowluma/bridge/format.ts (the last one renamed because the destination directory only has bridge-related files; `format.ts` reads cleaner than `event-format`. Listed as a rename in git's `RM` output — content unchanged.) Layering fixes (functionality preserved, just type signatures) -------------------------------------------------------------- - `element-builder.SendContext.bridge` → typed as `BridgeContext` instead of `Bridge`. Reads only `bridge.identity` and the upload-cache (both on BridgeContext); the concrete `Bridge` instance still passes via free upcast. - `bridge-oidb.runOidb` signature widened from `Bridge` to `BridgeContext` — same reason. - `highway/{highway-client,pipeline,image-upload,ptt-upload, video-upload}.ts` — same. - `format.ts::formatReply` parameter typed as a new local `ReplyEventLookup` structural type (`{ findEvent(id): JsonObject | null }`) instead of `MessageStore` from @snowluma/onebot. The OneBot `MessageStore` is a structural superset; consumers pass it through unchanged. - JSON value types moved to a new `@snowluma/common/json` subpath (consumed by format.ts) — `onebot/types.ts` re-exports them so external `import { JsonValue } from '<…>/onebot/types'` paths keep working. - `UploadedFileMeta` interface moved from `core/src/bridge/bridge.ts` into `@snowluma/bridge/bridge-context.ts` — element-builder + the forward-builder both read the upload cache through BridgeContext, so the type has to live in @snowluma/bridge. `bridge.ts` keeps its re-export for back-compat. Changes ------- - New package `packages/bridge/` (package.json with per-file subpath exports mirroring legacy `core/src/bridge/<x>` layout 1:1, tsconfig.json, typecheck script). - 35 `.ts` files moved via `git mv` (3 files counted by git as RM — rename + modify because Bridge → BridgeContext type swap). - 150+ import sites rewritten across `packages/core/src` and `packages/core/tests`: `'../bridge/<x>'`, `'../../bridge/<x>'`, `'../src/bridge/<x>'`, `'@/bridge/web/<x>'`, and vitest `vi.mock(...)` literals all collapse to `@snowluma/bridge/<x>`. - `@snowluma/core` depends on `@snowluma/bridge` (workspace:*). - `@snowluma/bridge` depends on `@snowluma/common`, `@snowluma/proto-defs`, `@snowluma/proton`. - `packages/core/src/utils/` directory removed (empty). Verification ------------ - Full workspace `pnpm -r typecheck`: all 10 projects clean. - core test suite: 552 passed / 552 total (pre-existing websocket-darwin-arm64.node addon failure unchanged — same baseline as the previous refactor commits). - Vite build (with stubbed native binary): clean — 143 modules, 2.84 MB bundle / 224 KB gzip, identical envelope to the pre-split build. Phase 3 (next): OneBot + WebUI move into @snowluma/onebot. The runnable `index.ts` stays in @snowluma/core as the application entry, importing `OneBotManager` from `@snowluma/onebot`. |