codex/async-sqlite-node24
361
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
2638d13d11 |
feat(profile): surface QQ level from OIDB 0xFE1_2 number-property 105
`fetchUserProfile` already requested key 105 in the keys[] array but the decode pass never read it, so callers got nothing. NapCat exposes the same value as `qq_level` on `get_stranger_info`; users coming from NapCat noticed the field was missing on the SnowLuma side. Cross-checked against LagrangeV2 Internal/Services/System/ FetchStrangerService.cs which explicitly labels key 105 = Level and uses `numbers[105]` directly. Changes: - UserProfileInfo: add `level: number`. - fetchUserProfile decode: `info.level = numMap.get(105) ?? 0`. - getStrangerInfo: emit both `qq_level` (NapCat-compat) and `level`. - Test/manager call sites updated for the new required field. |
||
|
|
ee0f159b7e |
fix(deps): drop dead hono devDeps from @snowluma/onebot, align core to ^4.12.18
`packages/onebot` declared `hono` and `@hono/node-server` as devDeps but
never imported either — they were leftover scaffolding from when onebot
was carved out of `@snowluma/core`. Combined with dependabot's hono
bump only landing on core's pin (4.12.14 → 4.12.18 in
|
||
|
|
607a7b319b | chore(release): v1.9.1 | ||
|
|
4e51b4bd7d |
build(sqlite): wrap better-sqlite3 with @snowluma/sqlite + vendor prebuilts
Replaces "runtime npm dependency that gets `npm install`-ed into
dist/" with "TS wrapper that loads the native `.node` from
`dist/native/` like every other native binding in the project".
Why the change
--------------
The previous approach (better-sqlite3 declared in
packages/runtime/package.json, materialized by `npm install
--omit=dev` in dist/) worked but produced an artifact that needed
runtime npm install — a step the rest of SnowLuma's native bindings
don't need (websocket / snowluma-hook / ffmpeg are all standalone
`.node` files copied into `dist/native/` by vite-plugin-cp). The
asymmetry was friction: dev mode required `pnpm dist:install`,
build artifacts carried a `dist/node_modules/` tree the other
modules didn't need, and Docker images had to special-case the
glibc-vs-musl ABI lookup that prebuild-install runs.
The new shape:
- `@snowluma/sqlite` (packages/sqlite/) — TS wrapper. Has
`better-sqlite3` as a devDep (for the JS-layer types + the
actual JS wrapper code that vite inlines). At runtime, calls
`createRequire().require()` against
`<dist|repo>/native/better-sqlite3-v<abi>-<platform>-<arch>.node`
and passes the resolved addon to `new Database(path, {
nativeBinding })`.
- `packages/runtime/native/better-sqlite3-v<abi>-<triplet>.node`
— prebuilt binaries vendored into the repo. Two ABIs ship side-
by-side (v127 = Node 22 LTS used by CI, v137 = Node 24 used by
dev boxes) so `process.versions.modules` picks the matching one
at load time.
- `tools/fetch-sqlite-prebuilts.mjs` — fetches all 5 platforms ×
N ABIs from WiseLibs's GitHub releases. Idempotent: existing
`.node` files skip. Accepts extra ABI args:
`node tools/fetch-sqlite-prebuilts.mjs 12.10.0 v127 v137 v115`.
What was unwound
----------------
- packages/runtime/package.json — dependencies removed (runtime
artifact is now self-contained, no `npm install` step needed).
- root package.json — `dist:install` script removed (no node_modules
to install).
- .github/workflows/dev-build.yml — `Bundle runtime dependencies`
step removed.
- .github/workflows/release.yml — kept the `Bundle runtime
dependencies` step (was a no-op already, but harmless safety net
in case a future runtime dep shows up).
- packages/core/vite.config.ts — `better-sqlite3` no longer
external; vite inlines the JS wrapper (+~33 KB to bundle, ~2.84
MB → 2.87 MB total).
Naming
------
Mirrors the existing `websocket-${triplet}.node` convention (flat
dir under `runtime/native/`, dash-separated `<platform>-<arch>`).
Adds a `v<abi>` segment because better-sqlite3 is NOT N-API — one
binary per Node ABI. Naming:
better-sqlite3-v127-darwin-arm64.node Node 22 / macOS arm
better-sqlite3-v127-darwin-x64.node Node 22 / macOS intel
better-sqlite3-v127-linux-arm64.node Node 22 / Linux arm
better-sqlite3-v127-linux-x64.node Node 22 / Linux x64
better-sqlite3-v127-win32-x64.node Node 22 / Windows x64
better-sqlite3-v137-... (5) Node 24 / same 5 hosts
Total: 10 binaries × ~1.9 MB = ~19 MB committed. Aligned with the
existing native-vendoring footprint (websocket ~2 MB × 5, snowluma-
hook ~10 MB × Windows + Linux, ffmpeg ~10-30 MB × 4).
What changes for consumers
--------------------------
Source-side: nothing. `import Database, { type Database, type
Statement } from '@snowluma/sqlite'` is a drop-in for the previous
`import Database from 'better-sqlite3'`. The wrapper re-exports the
same types and the `Database` callable factory has the same
overload set.
Runtime: `node` finds the binding at
`<dist|repo>/native/better-sqlite3-v${process.versions.modules}-
${process.platform}-${process.arch}.node`. If the ABI doesn't
match a vendored binary, the wrapper throws a clear error pointing
at `tools/fetch-sqlite-prebuilts.mjs` with the missing ABI/triplet.
Bumped better-sqlite3 from 11.10.0 → 12.10.0 because v11 only had
prebuilt up to Node ABI v127 (no v137 = no Node 24). v12.10.0
ships both. Both versions are API-compat (the breaking changes
between v11 and v12 were either internal or in error-handling
paths SnowLuma doesn't exercise).
Verification
------------
- pnpm -r typecheck (12 projects): clean
- @snowluma/core test: 302/302
- @snowluma/onebot test: 250/250 (552/552 combined, unchanged from
the perf-series baseline)
- Bench numbers preserved — wrapper indirection is sub-μs:
storeMeta ~11 μs, storeEvent ~17 μs, findMeta hit ~1.1 μs (all
within noise of the post-RoadMap-#5 measurements).
- Vite build clean, bundle = 2.87 MB / 230 KB gzip
- `dist/index.mjs` no longer contains a `node:sqlite` import (it
did before — media-store hadn't migrated; now neither it nor
message-store reach into the Node builtin).
Bonus side-fix: `packages/onebot/src/media-store.ts` also moved
off `node:sqlite` (caught by the post-build bundle grep — the
file was using `DatabaseSync` + `StatementSync` from the builtin
even though it had already been written to cache its prepared
statements). Its hot paths now go through the same wrapper.
|
||
|
|
0b6541ca36 |
perf(sqlite): swap node:sqlite for better-sqlite3 + cache prepared stmts
Replaces the experimental `node:sqlite` builtin with the stable
`better-sqlite3` wrapper across the two hot SQLite users — the
OneBot message-store and the bridge identity-service. While the API
surface is essentially identical (sync, prepared-statement-driven,
`Database`/`Statement` shape), better-sqlite3 (a) doesn't carry the
"ExperimentalWarning: SQLite is an experimental feature" baggage
that ships with `node:sqlite`, (b) has a more mature C++ binding
that consistently outperforms the builtin by ~30-50%, and (c)
encourages reusing `Statement` objects by design — exactly the win
the second half of this commit is after.
Why both at once
----------------
Caching prepared statements (was P1 in the RoadMap-#5 analysis) is
the largest single optimization opportunity on the SQLite path. The
old shape was the textbook SQLite anti-pattern:
// every call re-parsed the SQL
this.db.prepare(BIG_SQL_STRING).run(...);
Each `prepare(sql)` runs the full SQL parser + planner — for simple
inserts/updates/selects that dominates wall-clock cost. Doing both
the driver swap AND the statement caching in one commit lets the
benchmark numbers attribute the combined effect honestly; splitting
them would have double-charged the same disruption budget.
What changed code-wise
----------------------
**messageStore** — 8 distinct SQL templates, all now `readonly`
fields prepared once in the constructor:
- stmtStoreEvent — INSERT/UPDATE event row + JSON payload
- stmtStoreMeta — INSERT/UPDATE meta-only row
- stmtFindEvent — single-row SELECT by message_hash
- stmtFindMeta — single-row SELECT (full meta columns)
- stmtResolveReplyGroup, stmtResolveReplyPrivate
— 2 variants for the reply-id lookup
- stmtListEventsAnchored, stmtListEventsLatest
— 2 variants for /get_history_msg
**identity-service** — 25+ distinct statements. Used a Map-based
`pstmt(sql)` memoization helper rather than enumerating all 25 as
named fields (call sites stay readable; Map lookup is O(1) hash +
string-eq, negligible next to the SQL parse it replaces). The
helper short-circuits to the cached `Statement` on second-and-later
calls.
API differences from node:sqlite (handled in this commit):
- `import { DatabaseSync } from 'node:sqlite'`
→ `import Database from 'better-sqlite3'`
- `new DatabaseSync(path)` → `new Database(path)`
- `db.exec('PRAGMA journal_mode = WAL')`
→ `db.pragma('journal_mode = WAL')`
- `db.exec('BEGIN' / 'COMMIT' / 'ROLLBACK')` — kept as-is; both
drivers accept raw `exec()` for transaction control. The
`db.transaction()` higher-order wrapper is more idiomatic
but the existing manual control already works and changing
it would invalidate the `transaction()` helper's matched
try/catch shape.
vite build wiring:
- `node:sqlite` removed from the externals list
- `better-sqlite3` added — must stay external because it has a
native binding (`require('./build/Release/better_sqlite3.node')`)
that Vite can't bundle. Vendoring the prebuilt `.node` into
`packages/runtime/native/` is a follow-up packaging task; the
workspace `pnpm install` already places the binding under
`node_modules` so dev + tests work today.
Bench results (RoadMap #5 baseline → after)
-------------------------------------------
@snowluma/bridge — recv pipeline (with SQLite identity writes):
group_member_join 17.5 μs/op → 4.7 μs/op (~3.7x faster)
group_message text 0.42→0.42 μs (no path change)
group_message mixed 0.67→0.67 μs (no path change)
@snowluma/onebot — send pipeline:
messageStore.storeMeta 17.0 μs/op → 10.9 μs/op (~1.6x)
messageStore.storeEvent 23.2 μs/op → 18.2 μs/op (~1.3x)
messageStore.findMeta hit 5.4 μs/op → 1.3 μs/op (~4x)
messageStore.findMeta miss 3.7 μs/op → 0.7 μs/op (~5x)
messageStore.findEvent 3.8 μs/op → 1.7 μs/op (~2.3x)
Read paths gained the most because they're parse-bound (small SQL
+ short execution → the parser was most of the cost). Writes gained
less because the fsync/journal overhead on each commit still
dominates the parse savings. Combined send hot path (storeMeta +
storeEvent, called on every outbound message + the /get_msg-fix
self-event store) drops from ~40 μs to ~29 μs — about a 27%
reduction per message sent.
Verification
------------
- pnpm -r typecheck: all 11 projects clean
- @snowluma/onebot test suite: 250/250
- @snowluma/core test suite: 302/302
- Combined: 552/552 (unchanged baseline)
- Vite build (with stubbed native binary): clean, 2.84 MB / 223 KB
gzip (matches pre-Phase-2 envelope after removing the better-
sqlite3 inline-bundle attempt — externalizing it shaves the
~30 KB regression we saw mid-flight)
Follow-up (next commits in this series)
---------------------------------------
- Skip rememberGroupMemberIdentity DB write when the (uid, uin,
nickname, card) tuple is unchanged in the in-memory cache.
- Drop the `Buffer.from(pkt.body)` allocation in msg-push context.
- Synchronous fast-path for elementsToJson when no async resolver
is wired.
- Vendor `better-sqlite3` prebuilt `.node` into the runtime/native
layout for the production package step.
|
||
|
|
f3f5d27c65 |
perf(bench): bench harness for receive/send chains (RoadMap #5 baseline)
vitest-driven benches at `packages/bridge/bench/recv.bench.ts` and
`packages/onebot/bench/send.bench.ts`. Picks up proton's vite plugin
so call sites get the inlined codecs (running via tsx + proton
runtime-fallback would throw without a pre-built runtime map).
Run via `pnpm -F @snowluma/bridge bench` and
`pnpm -F @snowluma/onebot bench`.
What's measured
---------------
Receive chain:
- parseMsgPush: pure decode + dispatch (no SQLite, no event emit)
* group_message text ~ 0.29 μs/op (3.4M ops/sec)
* group_message 5 mixed ~ 0.50 μs/op (2.0M ops/sec)
* group_member_join ~ 0.25 μs/op (4.1M ops/sec)
- IncomingPacketPipeline.process: full pipeline (with SQLite via
rememberGroupMemberIdentity, BridgeEventBus emit)
* group_message text ~ 0.42 μs/op (2.4M ops/sec)
* group_message 5 mixed ~ 0.67 μs/op (1.5M ops/sec)
* group_member_join ~17.5 μs/op ( 57K ops/sec) ⚠
Send chain:
- parseMessage (OneBot input → MessageElement[])
* CQ string text ~ 0.31 μs/op
* CQ string mixed ~ 1.22 μs/op
* segment array text ~ 0.32 μs/op
* segment array 5 ~ 1.07 μs/op
- messageStore SQLite writes:
* storeMeta ~17.0 μs/op ( 59K ops/sec) ⚠
* storeEvent (~250B JSON) ~23.2 μs/op ( 43K ops/sec) ⚠
- messageStore reads:
* findMeta hit ~ 5.4 μs/op (186K ops/sec)
* findMeta miss ~ 3.7 μs/op (272K ops/sec)
* findEvent hit ~ 3.8 μs/op (262K ops/sec)
Headline finding
----------------
Pure decode/encode is already cheap (sub-μs). The real cost per
sent message is the SQLite write pair (storeMeta + storeEvent =
~40 μs). On receive, group_message itself does NOT trigger
identity writes; only join/leave/admin events do — but those go
through 4-6 db.prepare(sql).run/get(...) calls each, all re-parsing
SQL on every invocation (the canonical SQLite anti-pattern).
The follow-up commits in this series target:
1. Replace `node:sqlite` (experimental, API may shift) with
`better-sqlite3` (stable, sync, ~30-50% faster wire) + cache
prepared statements as readonly fields. Single commit, both
wins together — better-sqlite3's API encourages stmt caching
by design. Expected: storeMeta / storeEvent drop to ~3-5 μs,
group_member_join drops to ~3-5 μs.
2. Skip rememberGroupMemberIdentity DB write when the in-memory
uidByUin / uinByUid cache already has the mapping unchanged.
3. Drop the `Buffer.from(pkt.body)` allocation in msg-push
context (pkt.body is already Uint8Array; proton accepts it).
4. Make `elementsToJson` sync-fast-path when no async resolvers
are wired (saves per-element microtask overhead).
Each commit re-runs these benches and the message should quote the
before/after numbers so the deltas are auditable.
Note: tsx is NOT involved — vitest's bench mode owns the run, and
the proton vite plugin is configured in each `vitest.config.ts`.
Absolute numbers reflect the production code path.
|
||
|
|
d983635c8f |
refactor(onebot): inline ctx.bridge.apis.profile/web/groupAlbum direct calls
Remove the remaining ~20 passthrough fields from ApiActionContext spanning four Api areas — most were already dead (no caller) since extended.ts had migrated to direct `ctx.bridge.apis.X` paths in earlier #6 commits. profile area: - setProfile / setOnlineStatus / fetchCustomFace / setGroupAvatar ── all already dead - getProfileLike ── one live caller in extended.ts inlined web area: - getGroupHonorInfo / getGroupEssence / getGroupEssenceAll / sendGroupNotice / getGroupNotice / deleteGroupNotice / getCookiesStr / getCsrfToken / getCredentials / forceFetchClientKey ── all already dead groupAlbum area: - getGroupAlbumList / uploadImageToGroupAlbum / getGroupAlbumMediaList / commentGroupAlbumMedia / deleteGroupAlbumMedia / likeGroupAlbumMedia ── all 7 live callers in actions/group-album.ts inlined to `ctx.bridge.apis.groupAlbum.<m>(...)` friend area: - setFriendRemark ── already dead Behavior preserved: getProfileLike's `userId = 0 → self` semantics keep working because ProfileApi.getLike uses `isSelf = !userId` internally (0 and undefined are equivalent for that check); same default args at the action site (`start = 0`, `count = 10`). Group-album action handlers now wrap the typed Api return values (GroupAlbumMediaResult, GroupAlbumCommentResult) with `as unknown as JsonValue` for okResponse compatibility — the old ctx wrapper was typed `Promise<any>` which silently hid the cast; making the cast explicit doesn't change runtime. Part 4 of the OneBot-side slim-down. 552/552 tests still green. |
||
|
|
8bf2ff3f36 |
refactor(onebot): inline ctx.bridge.apis.friend/interaction/message direct calls
Remove 8 passthrough fields from ApiActionContext spanning three Api
areas:
friend area:
- handleFriendRequest (set_friend_add_request)
- handleDeleteFriend (delete_friend)
- setFriendRemark already kept the field name but now unused
interaction area:
- sendFriendPoke / sendGroupPoke (friend_poke, group_poke, send_poke)
- getEmojiLikes (was already dead — extended.ts used the direct
path; field removed)
message area:
- markGroupMsgAsRead / markPrivateMsgAsRead (already dead —
mark_msg_as_read sites use the module helper)
`handleGroupRequest` stays on ctx because `handleGroupAddRequest` is
a module helper that does the flag-parsing + group-request roster
lookup before routing to `bridge.apis.groupAdmin.setAddRequest` —
real composition, not a passthrough.
Behavior preserved: same flag parsing (`asString/asNumber/asBoolean`),
same `isGroup` flag flatten into `sendPoke(isGroup, peerUin, target?)`,
same default `block=false` for delete_friend.
Part 3 of the OneBot-side slim-down. 552/552 tests still green.
|
||
|
|
4c4ce60bf2 |
refactor(onebot): inline ctx.bridge.apis.groupFile direct calls
Remove 6 passthrough fields from ApiActionContext for the group-file area (`uploadGroupFile / uploadPrivateFile / getGroupFileUrl / createGroupFileFolder / getPrivateFileUrl / getGroupFileCount`). Action sites (`actions/group-file.ts` and one site in `actions/extended.ts`) now call `ctx.bridge.apis.groupFile.<method>` directly. `getGroupFiles` stays on the ctx because its module helper (`getGroupFiles(bridge, groupId, folderId)` in `modules/contact-actions.ts`) does identity-lookup enrichment of uploader/creator names — not a pure passthrough. Behavior preserved: same default values (`name = ''`, `folderId = '/'`, `busId = 102`, `uploadFile = true`) — they were baked into the ctx wrapper before, now they're explicit in the action site (`asString` returns `''`, `... || '/'`, `... || 102`, `asBoolean(..., true)`). The result-unwrap (`fileId` extraction from `UploadFileResult`) moves from the ctx wrapper into the action handler. Part 2 of the OneBot-side slim-down. 552/552 tests still green. |
||
|
|
cb698d2435 |
refactor(onebot): inline ctx.bridge.apis.groupAdmin direct calls
Remove the 6 pure pass-through fields from `ApiActionContext` for the group-admin area (`setGroupKick / setGroupKickMembers / setGroupBan / setGroupWholeBan / setGroupLeave / setGroupRemark`). All 5 call sites in `actions/group-admin.ts` now invoke `ctx.bridge.apis.groupAdmin.<m>` directly. `setGroupRemark` had zero remaining callers (extended.ts was already on the direct path) so the field is simply deleted. Part 1 of RoadMap #6 OneBot-side slim-down. Core side's typed Api hub (bridge.apis.X) has been in place since the #6 commit series; OneBot actions were still routing through a 90+ field bag-of- functions on `ApiActionContext`. Most of those fields were 1:1 shims like `setGroupKick: (g,u,r) => bridge.apis.groupAdmin.kickMember(g,u,r)` — pure indirection with no composition logic. This commit kills the group-admin chunk; the friend / interaction / profile / web / album / groupFile / message chunks land in follow-up commits. Behavior preserved byte-for-byte: same parameter parsing (`asNumber/asString/asBoolean`), same default values, same 1800s mute fallback, same error responses. 552/552 tests still green. |
||
|
|
9a30df9907 |
refactor(onebot): extract OneBot protocol translation into @snowluma/onebot
Phase 3 of the 4-package split (RoadMap #3). Moves the entire `packages/core/src/onebot/` subtree (43 source files + 20 test files spanning network adapters, action handlers, event converters, message parser, message store, media store, instance + manager) into the new `@snowluma/onebot` workspace package. The remaining `@snowluma/core` now hosts only the api-layer (Bridge class + 13 Api classes), the hook tooling, the WebUI server, and the runnable `index.ts`. Why --- - The OneBot adapter is the public-facing OneBot11 protocol surface; it has its own version cadence and its own consumers (third-party bots talking OneBot WS / HTTP). Splitting it lets that cadence move independently of the Api-on-ctx layer. - The two were sharing the `packages/core/` build pipeline only because they happened to live in the same directory. Nothing in the api-layer reaches into OneBot internals. - Mirrors NapCat / Lagrange.OneBot's repo split (their OneBot adapter is a separate assembly from the Core protocol library). Layout ------ Moved via `git mv` (history preserved): packages/core/src/onebot/* → @snowluma/onebot/src/<x> packages/core/tests/onebot/* (6 files) → @snowluma/onebot/tests/ packages/core/tests/{message-parser,message-store,event-pipeline, media-store,media-url-resolver,event-converter, http-server-hot-reload,http-server-content-type, api-handler-logging,message-id,group-msg-emoji-like, config,media-indexer,network-manager}.test.ts (14 files) → @snowluma/onebot/tests/ What stays in @snowluma/core ---------------------------- - `bridge/{bridge,bridge-interface,bridge-context,manager}.ts` - `bridge/apis/*` (the 13 typed Api classes) - `hook/*` (debugger/injector — tooling for the core protocol layer) - `webui/*` (admin UI server — application wiring, not the OneBot adapter itself) - `index.ts` (application entry that wires BridgeManager + OneBotManager + HookManager + WebUI together) Changes ------- - New package `packages/onebot/` (package.json with subpath exports mirroring legacy `core/src/onebot/<x>` 1:1, tsconfig.json, vitest.config.ts with `SNOWLUMA_LOG_LEVEL=debug` env so the action- dispatch logging tests see debug-level subscribers — copied verbatim from core's vitest.config.ts). - @snowluma/core grows new subpath exports for the api-layer files OneBot's type-only imports need: `./bridge`, `./bridge-interface`, `./bridge-context`, `./manager`, `./apis`. - `@snowluma/onebot` depends on `@snowluma/bridge`, `@snowluma/common`, `@snowluma/core`, `@snowluma/proto-defs`, `@snowluma/proton`, `@snowluma/websocket`. - `@snowluma/core` depends on `@snowluma/onebot` (for index.ts + webui/server.ts wiring). - 100+ import sites rewritten via sed: * OneBot → core: `'../bridge/bridge'` → `'@snowluma/core/bridge'`, `'../bridge/manager'` → `'@snowluma/core/manager'`, `'../bridge/bridge-interface'` → `'@snowluma/core/bridge-interface'`, `'@/bridge/bridge'` (vite path alias) → `'@snowluma/core/bridge'`. * Core → OneBot: `'./onebot/manager'` → `'@snowluma/onebot/manager'`, `'../onebot/config'` → `'@snowluma/onebot/config'`, etc. * Test relative paths: `'../../src/onebot/<x>'` → `'../src/<x>'` after the test file move (one level shallower). - New `packages/onebot/src/env.d.ts` declaring `__APP_VERSION__` and `__BUILD_WEBUI__` (the vite-injected globals `info.ts` reads). Workspace cycle --------------- core ↔ onebot is allowed by pnpm workspaces. The runtime direction is one-way: core → onebot (index.ts constructs OneBotManager, webui/server reads OneBot config). OneBot only reaches into core for TYPES (Bridge, BridgeInterface, BridgeManager, ClientKeyInfo, DownloadRKeyInfo); those imports use `import type` so they compile out at runtime and don't introduce a real cycle. Verification ------------ - Full workspace `pnpm -r typecheck`: all 11 projects clean. - `@snowluma/core` test suite: 302/302 pass (33 test files). - `@snowluma/onebot` test suite: 250/250 pass (20 test files; pre-existing `network-manager.test.ts` native-addon failure unchanged — same baseline as before the split). - Combined: 552/552 — identical pass count to the pre-Phase-3 baseline. - Vite build with stubbed native binary: clean, 2.84 MB bundle / 224 KB gzip (identical envelope). |