Remove the privacy-scan command; releases run an optional owner-supplied check instead

waybetter release and the remote builds no longer scan artifacts themselves: WAYBETTER_RELEASE_CHECK
can name a command that gets every artifact path before anything is copied or signed (a non-zero
exit stops the release). The command, its tests and every mention in the docs, the template and
the architecture diagram are gone.
This commit is contained in:
CRTkafa 2026-10-11 07:32:38 +08:00
parent 7cd6c99007
commit 612322a45e
13 changed files with 48 additions and 1108 deletions

View file

@ -25,7 +25,7 @@ encrypted sync and signed updates; and the tooling to scaffold, sign, scan and r
| **Themes** | One JSON document: colours with contrast enforced, fonts per role (Google Fonts downloaded once, after asking), shapes, layout, decoration, icon packs, per-pane and per-account themes, a Theme Builder with a doctor report. | | **Themes** | One JSON document: colours with contrast enforced, fonts per role (Google Fonts downloaded once, after asking), shapes, layout, decoration, icon packs, per-pane and per-account themes, a Theme Builder with a doctor report. |
| **String packs** | Data that re-words the site's own interface: a language it does not offer, a fix, your own wording. Never code. | | **String packs** | Data that re-words the site's own interface: a language it does not offer, a fix, your own wording. Never code. |
| **A signed Store** | One Ed25519-signed catalog for every wayBetter app, `waybetter://install` links, drag-and-drop and Open With, all through one install screen that shows the source, SHA-256, permissions and code first. | | **A signed Store** | One Ed25519-signed catalog for every wayBetter app, `waybetter://install` links, drag-and-drop and Open With, all through one install screen that shows the source, SHA-256, permissions and code first. |
| **Release tooling** | `waybetter store`, `release`, `privacy-scan`, `build-env`, `i18n` and `strings`: sign the catalog, build binaries free of the builder's paths, and refuse to ship a build with a privacy hit. | | **Release tooling** | `waybetter store`, `release`, `build-env`, `i18n` and `strings`: sign the catalog and build binaries free of the builder's paths. |
## How it fits together ## How it fits together
@ -71,7 +71,7 @@ mode reloads them as you save:
| Package | Where | What it is | | Package | Where | What it is |
| --- | --- | --- | | --- | --- | --- |
| [`waybetter`](https://www.npmjs.com/package/waybetter) | `packages/waybetter` | The npm package: the built runtime and shell (plain ESM with `.d.ts`), JSON Schemas for themes, icon packs, catalogs and i18n, the locales, and the `waybetter` command (`dev`, `store`, `release`, `i18n`, `strings`, `privacy-scan`, `build-env`) | | [`waybetter`](https://www.npmjs.com/package/waybetter) | `packages/waybetter` | The npm package: the built runtime and shell (plain ESM with `.d.ts`), JSON Schemas for themes, icon packs, catalogs and i18n, the locales, and the `waybetter` command (`dev`, `store`, `release`, `i18n`, `strings`, `build-env`) |
| `create-waybetter` | `packages/create-waybetter` | `npm create waybetter`: scaffolds a plugin (JS or TypeScript), a theme or a whole app. No dependencies | | `create-waybetter` | `packages/create-waybetter` | `npm create waybetter`: scaffolds a plugin (JS or TypeScript), a theme or a whole app. No dependencies |
| `@waybetter/runtime` | `packages/runtime` | The page runtime (published inside `waybetter`) | | `@waybetter/runtime` | `packages/runtime` | The page runtime (published inside `waybetter`) |
| `@waybetter/shell` | `packages/shell` | Shell models, frame, panel and Store screens (published inside `waybetter`) | | `@waybetter/shell` | `packages/shell` | Shell models, frame, panel and Store screens (published inside `waybetter`) |
@ -191,8 +191,7 @@ The site's page is not trusted, and neither is a plugin. The rules:
that host commands stay shell-only or tab-only, that site commands are granted only on the that host commands stay shell-only or tab-only, that site commands are granted only on the
site's URLs, and that each command's first statement checks its caller. The scaffold includes site's URLs, and that each command's first statement checks its caller. The scaffold includes
such a test. such a test.
- `waybetter privacy-scan` and `waybetter build-env` keep a build free of the builder's paths and - `waybetter build-env` keeps a build free of the builder's paths.
names; `waybetter release` refuses to release an artifact with a hit.
## Optional command groups ## Optional command groups
@ -222,7 +221,7 @@ crates/waybetter-host/ Rust host crate (features: tauri, net, app)
packages/runtime/ page runtime (published inside `waybetter`) packages/runtime/ page runtime (published inside `waybetter`)
packages/shell/ shell models, shell frame, panel, store screens (published inside `waybetter`) packages/shell/ shell models, shell frame, panel, store screens (published inside `waybetter`)
packages/waybetter/ the npm package: built runtime and shell, JSON Schemas, locales, packages/waybetter/ the npm package: built runtime and shell, JSON Schemas, locales,
and the `waybetter` command (store, release, i18n, strings, privacy-scan, and the `waybetter` command (store, release, i18n, strings,
build-env, dev) build-env, dev)
packages/create-waybetter/ `npm create waybetter` packages/create-waybetter/ `npm create waybetter`
docs/ ARCHITECTURE.md, STORE.md, STRING-PACKS.md docs/ ARCHITECTURE.md, STORE.md, STRING-PACKS.md

View file

@ -1,4 +1,4 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 1200 760" width="1200" height="760" role="img" aria-label="wayBetter architecture: one Tauri 2 window with a trusted shell webview and one webview per site tab, a Rust host underneath, your adapter on the side, and the waybetter command for store, release and privacy tooling"> <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 1200 760" width="1200" height="760" role="img" aria-label="wayBetter architecture: one Tauri 2 window with a trusted shell webview and one webview per site tab, a Rust host underneath, your adapter on the side, and the waybetter command for store, release and build tooling">
<defs> <defs>
<linearGradient id="bg" x1="0" y1="0" x2="0" y2="1"> <linearGradient id="bg" x1="0" y1="0" x2="0" y2="1">
<stop offset="0" stop-color="#0b1020"/> <stop offset="0" stop-color="#0b1020"/>
@ -122,6 +122,6 @@
<text x="860" y="610">waybetter store build|sign|verify</text> <text x="860" y="610">waybetter store build|sign|verify</text>
<text x="860" y="636">waybetter strings check</text> <text x="860" y="636">waybetter strings check</text>
<text x="860" y="662">waybetter release</text> <text x="860" y="662">waybetter release</text>
<text x="860" y="688">waybetter privacy-scan</text> <text x="860" y="688">waybetter build-env</text>
</g> </g>
</svg> </svg>

Before

Width:  |  Height:  |  Size: 7.4 KiB

After

Width:  |  Height:  |  Size: 7.4 KiB

Before After
Before After

View file

@ -129,7 +129,7 @@ packages/shell/src/ shell core (TypeScript) and the shell frame (React)
packages/waybetter/ the published npm package (runtime + shell as ESM + .d.ts, theme.schema.json, packages/waybetter/ the published npm package (runtime + shell as ESM + .d.ts, theme.schema.json,
store-catalog.schema.json, the `waybetter store build|sign|verify`, store-catalog.schema.json, the `waybetter store build|sign|verify`,
`waybetter release windows|macos`, `waybetter i18n check|extract`, `waybetter release windows|macos`, `waybetter i18n check|extract`,
`waybetter privacy-scan` and `waybetter build-env` commands, `waybetter/build`, locales/ (51 catalogs and translator notes), `waybetter build-env` commands, `waybetter/build`, locales/ (51 catalogs and translator notes),
windows/ (the installer template and its language files)) windows/ (the installer template and its language files))
dev-tools/ development harnesses for a running or built app (perf probes, close and dev-tools/ development harnesses for a running or built app (perf probes, close and
stress checks); not published (dev-tools/README.md) stress checks); not published (dev-tools/README.md)
@ -459,8 +459,8 @@ The npm package adds the app-side tooling:
`<outDir>/upload`, `latest.json` (other platforms of the version kept); macOS: signs a universal `<outDir>/upload`, `latest.json` (other platforms of the version kept); macOS: signs a universal
build's updater bundle and adds both darwin platforms. It never uploads: it prints the steps. build's updater bundle and adds both darwin platforms. It never uploads: it prints the steps.
The key is read only into the build's environment or handed to the Tauri signer by path. The key is read only into the build's environment or handed to the Tauri signer by path.
- `waybetter build-env -- <command>` and `waybetter privacy-scan`: nothing an app publishes may - `waybetter build-env -- <command>`: nothing an app publishes should carry the builder's
name the person who built it. A Rust binary embeds the absolute paths of its sources (the Cargo paths. A Rust binary embeds the absolute paths of its sources (the Cargo
registry, the rustup toolchain, the workspace: about 1,400 strings holding the account name in registry, the rustup toolchain, the workspace: about 1,400 strings holding the account name in
one app), so `build-env` runs a build with those roots remapped to `/home`, `/src`, `/cargo`, one app), so `build-env` runs a build with those roots remapped to `/home`, `/src`, `/cargo`,
`/rustup` and `/target` (`--remap-path-prefix` appended to the flags already set, `/rustup` and `/target` (`--remap-path-prefix` appended to the flags already set,
@ -468,13 +468,9 @@ The npm package adds the app-side tooling:
prefix map, so there the C and C++ a build script compiles (a CMake library, the cc crate) lose prefix map, so there the C and C++ a build script compiles (a CMake library, the cc crate) lose
the same roots through `/d1trimfile:<root>` in the `CL` variable, and any PDB they link through the same roots through `/d1trimfile:<root>` in the `CL` variable, and any PDB they link through
`/PDBALTPATH` in `LINK`; cl.exe and link.exe read both themselves, whatever runs them: `/PDBALTPATH` in `LINK`; cl.exe and link.exe read both themselves, whatever runs them:
`waybetter build-env -- tauri build`, on every platform. `privacy-scan` is the proof: it searches `waybetter build-env -- tauri build`, on every platform. `waybetter release` builds with
files, binaries (UTF-8, UTF-16), archives (npm tarballs, zip, tar, .deb, DMG; NSIS installers and `build-env` and runs an optional release check on every artifact (`WAYBETTER_RELEASE_CHECK`, a
AppImages through 7-Zip), git-tracked files and, with `--history`, every blob, path, commit command given the artifact paths; a non-zero exit stops the release). The `.pdb`
message and identity of a repository for the terms in a file kept outside every repository
(`"forbiddenTerms"` in `waybetter.release.json`, or `WAYBETTER_FORBIDDEN_TERMS`) plus the
machine's account name, and prints hits masked (`m***`). `waybetter release` builds with
`build-env` and refuses to release an artifact with a hit or one it could not open. The `.pdb`
next to a Windows binary still holds every path: never publish it. next to a Windows binary still holds every path: never publish it.
- `waybetter strings check <pack>`: a string pack (data that replaces the wrapped site's own UI - `waybetter strings check <pack>`: a string pack (data that replaces the wrapped site's own UI
strings) against the site's string inventory: only inventory keys, parameter names, plural and strings) against the site's string inventory: only inventory keys, parameter names, plural and

View file

@ -1214,8 +1214,7 @@ function readme(o) {
'', '',
'Build with `npm run desktop:build` (or `npm run release`), not a bare `tauri build`: a Rust binary', 'Build with `npm run desktop:build` (or `npm run release`), not a bare `tauri build`: a Rust binary',
'embeds the paths it was built from, and those hold your account name. `waybetter build-env` maps', 'embeds the paths it was built from, and those hold your account name. `waybetter build-env` maps',
'them to `/home`, `/src`, `/cargo` and `/rustup`; `npm run privacy:scan` and the release check the', 'them to `/home`, `/src`, `/cargo` and `/rustup`.',
'result (`waybetter privacy-scan --help`).',
'', '',
'```text', '```text',
'```', '```',
@ -1266,7 +1265,6 @@ export function appFiles(o) {
dev: 'waybetter dev --watch "node scripts/build.mjs --watch"', dev: 'waybetter dev --watch "node scripts/build.mjs --watch"',
'dev:stable': 'waybetter dev --stable --watch "node scripts/build.mjs --watch"', 'dev:stable': 'waybetter dev --stable --watch "node scripts/build.mjs --watch"',
'desktop:build': 'waybetter build-env -- tauri build', 'desktop:build': 'waybetter build-env -- tauri build',
'privacy:scan': 'waybetter privacy-scan --tracked .',
'test:rust': 'cargo test --manifest-path src-tauri/Cargo.toml --no-default-features', 'test:rust': 'cargo test --manifest-path src-tauri/Cargo.toml --no-default-features',
release: 'waybetter release windows', release: 'waybetter release windows',
'release:macos': 'waybetter release macos', 'release:macos': 'waybetter release macos',

View file

@ -120,9 +120,8 @@ const Shell = createShell({ ...site, panel: { id: 'app', title: 'wayBetter', com
- `waybetter store sign [--init] --key <private key>` / `waybetter store verify --pub <key file>`: Ed25519 signature over the exact catalog bytes; any listed public key verifies (rotation). The private key is never printed or copied. - `waybetter store sign [--init] --key <private key>` / `waybetter store verify --pub <key file>`: Ed25519 signature over the exact catalog bytes; any listed public key verifies (rotation). The private key is never printed or copied.
- `waybetter strings check <pack> [--inventory <file>] [--verbose]`: validates a string pack (data that replaces the wrapped site's own UI strings: `waybetter/runtime/stringPacks`, `waybetter/runtime/stringPackHook`, `waybetter/runtime/textNet`) against the site's string inventory and prints its coverage; the inventories otherwise come from `waybetter.strings.json` in the app's root. - `waybetter strings check <pack> [--inventory <file>] [--verbose]`: validates a string pack (data that replaces the wrapped site's own UI strings: `waybetter/runtime/stringPacks`, `waybetter/runtime/stringPackHook`, `waybetter/runtime/textNet`) against the site's string inventory and prints its coverage; the inventories otherwise come from `waybetter.strings.json` in the app's root.
- `waybetter i18n check [--write] [--verbose] [locale…]`: checks an app's catalogs (and the framework's in the same languages) with the `waybetter/i18n/check` rules, bundled plugins' inline English and the generated files; `--write` regenerates `languages.json` and the native localization files (macOS, Linux). `waybetter i18n extract [--missing]`: keys the code looks up that English lacks, and English keys without a literal lookup. Both read `waybetter.i18n.json` in the app's root (`locales`, `sources`, `nativeSources`, `bundledPlugins`, `runtimeKeys`, `supported`, `native`; schema: `i18n-config.schema.json`). `supported` lists the languages in the supported tier; every other catalog gets `tier: "community"` in the generated `languages.json` (`languages.schema.json`), and an app resolves its automatic language choices only to supported ones (`createLanguage({ supported })`). - `waybetter i18n check [--write] [--verbose] [locale…]`: checks an app's catalogs (and the framework's in the same languages) with the `waybetter/i18n/check` rules, bundled plugins' inline English and the generated files; `--write` regenerates `languages.json` and the native localization files (macOS, Linux). `waybetter i18n extract [--missing]`: keys the code looks up that English lacks, and English keys without a literal lookup. Both read `waybetter.i18n.json` in the app's root (`locales`, `sources`, `nativeSources`, `bundledPlugins`, `runtimeKeys`, `supported`, `native`; schema: `i18n-config.schema.json`). `supported` lists the languages in the supported tier; every other catalog gets `tier: "community"` in the generated `languages.json` (`languages.schema.json`), and an app resolves its automatic language choices only to supported ones (`createLanguage({ supported })`).
- `waybetter release windows|macos`: signed native releases and the update manifest (`waybetter.release.json`). The Windows build runs path-free (below) and every artifact must pass the privacy scan before anything is copied or signed. - `waybetter release windows|macos`: signed native releases and the update manifest (`waybetter.release.json`). The Windows build runs path-free (below); `WAYBETTER_RELEASE_CHECK` can name a command run on every artifact before anything is copied or signed.
- `waybetter build-env -- <command>`: runs a build (`tauri build`, `cargo build`) so the binary holds none of the machine's paths: the home folder, the workspace, `CARGO_HOME`, `RUSTUP_HOME` and `CARGO_TARGET_DIR` become `/home`, `/src`, `/cargo`, `/rustup` and `/target` (`--remap-path-prefix` appended to the flags already in `RUSTFLAGS`/`CARGO_ENCODED_RUSTFLAGS`, `/PDBALTPATH` on MSVC; for C and C++ a build script compiles, `-ffile-prefix-map` on gcc and clang and, on MSVC, `/d1trimfile:<root>` in `CL` and `/PDBALTPATH` in `LINK`, which reach the cc crate, CMake and MSBuild alike). Without a command it prints that environment (`--json`). - `waybetter build-env -- <command>`: runs a build (`tauri build`, `cargo build`) so the binary holds none of the machine's paths: the home folder, the workspace, `CARGO_HOME`, `RUSTUP_HOME` and `CARGO_TARGET_DIR` become `/home`, `/src`, `/cargo`, `/rustup` and `/target` (`--remap-path-prefix` appended to the flags already in `RUSTFLAGS`/`CARGO_ENCODED_RUSTFLAGS`, `/PDBALTPATH` on MSVC; for C and C++ a build script compiles, `-ffile-prefix-map` on gcc and clang and, on MSVC, `/d1trimfile:<root>` in `CL` and `/PDBALTPATH` in `LINK`, which reach the cc crate, CMake and MSBuild alike). Without a command it prints that environment (`--json`).
- `waybetter privacy-scan <paths…> [--terms <file>] [--tracked] [--history]`: searches files, binaries (UTF-8 and UTF-16), archives (zip, tar, gzip, zstd, .deb, DMG; NSIS installers, AppImages and xz with 7-Zip or the matching tool), git-tracked files and a repository's whole history for forbidden terms: the lines of a file kept outside every repository (`--terms`, `WAYBETTER_FORBIDDEN_TERMS`, or `"forbiddenTerms"` in `waybetter.release.json`) plus this machine's account name. A name is a hit as a whole word in any case; a name of three characters or fewer only in a path or as the capitalized word. Output shows file and line or offset with the term masked (`m***`); exit 1 on a hit, 2 when a container could not be opened. Reviewed false positives: `"privacyAllowlist"`, lines of `<file glob>: <regex over the context, the term written <term>>`.
- `windows/languages/<Language>.nsh`: the Windows installer's messages for the languages Tauri does not ship, next to the installer template (`windows/installer.nsi`). - `windows/languages/<Language>.nsh`: the Windows installer's messages for the languages Tauri does not ship, next to the installer template (`windows/installer.nsi`).
## Plugin UI kit (`runtime/ui`) ## Plugin UI kit (`runtime/ui`)

View file

@ -1,7 +1,7 @@
#!/usr/bin/env node #!/usr/bin/env node
// The `waybetter` command: `waybetter store build|sign|verify` (cli/store.mjs), // The `waybetter` command: `waybetter store build|sign|verify` (cli/store.mjs),
// `waybetter release windows|macos|linux` (cli/release.mjs), `waybetter manifests` (cli/manifests.mjs), `waybetter i18n check|extract` // `waybetter release windows|macos|linux` (cli/release.mjs), `waybetter manifests` (cli/manifests.mjs), `waybetter i18n check|extract`
// (cli/i18n.mjs), `waybetter privacy-scan` (cli/privacy.mjs) and `waybetter build-env` // (cli/i18n.mjs), `waybetter build-env`
// (cli/buildenv.mjs), `waybetter dev` (cli/dev.mjs: watchers + tauri dev), `waybetter strings check` // (cli/buildenv.mjs), `waybetter dev` (cli/dev.mjs: watchers + tauri dev), `waybetter strings check`
// (cli/strings.mjs). // (cli/strings.mjs).
import { readFileSync } from 'node:fs'; import { readFileSync } from 'node:fs';
@ -9,7 +9,6 @@ import { BUILD_ENV_USAGE, buildEnvCommand } from '../cli/buildenv.mjs';
import { DEV_USAGE, devCommand } from '../cli/dev.mjs'; import { DEV_USAGE, devCommand } from '../cli/dev.mjs';
import { MANIFESTS_USAGE, manifestsCommand } from '../cli/manifests.mjs'; import { MANIFESTS_USAGE, manifestsCommand } from '../cli/manifests.mjs';
import { I18N_USAGE, i18nCommand } from '../cli/i18n.mjs'; import { I18N_USAGE, i18nCommand } from '../cli/i18n.mjs';
import { PRIVACY_USAGE, privacyCommand } from '../cli/privacy.mjs';
import { STORE_USAGE, storeCommand } from '../cli/store.mjs'; import { STORE_USAGE, storeCommand } from '../cli/store.mjs';
import { STRINGS_USAGE, stringsCommand } from '../cli/strings.mjs'; import { STRINGS_USAGE, stringsCommand } from '../cli/strings.mjs';
import { RELEASE_USAGE, releaseCommand } from '../cli/release.mjs'; import { RELEASE_USAGE, releaseCommand } from '../cli/release.mjs';
@ -26,8 +25,6 @@ try {
process.exitCode = await i18nCommand(rest); process.exitCode = await i18nCommand(rest);
} else if (area === 'strings') { } else if (area === 'strings') {
process.exitCode = await stringsCommand(rest); process.exitCode = await stringsCommand(rest);
} else if (area === 'privacy-scan') {
process.exitCode = privacyCommand(rest);
} else if (area === 'build-env') { } else if (area === 'build-env') {
process.exitCode = buildEnvCommand(rest); process.exitCode = buildEnvCommand(rest);
} else if (area === 'dev') { } else if (area === 'dev') {
@ -35,7 +32,7 @@ try {
} else if (area === '--version' || area === '-v') { } else if (area === '--version' || area === '-v') {
console.log(JSON.parse(readFileSync(new URL('../package.json', import.meta.url), 'utf8')).version); console.log(JSON.parse(readFileSync(new URL('../package.json', import.meta.url), 'utf8')).version);
} else { } else {
console.log(`${STORE_USAGE}\n\n${RELEASE_USAGE}\n\n${MANIFESTS_USAGE}\n\n${I18N_USAGE}\n\n${STRINGS_USAGE}\n\n${PRIVACY_USAGE}\n\n${BUILD_ENV_USAGE} console.log(`${STORE_USAGE}\n\n${RELEASE_USAGE}\n\n${MANIFESTS_USAGE}\n\n${I18N_USAGE}\n\n${STRINGS_USAGE}\n\n${BUILD_ENV_USAGE}
${DEV_USAGE}`); ${DEV_USAGE}`);
if (area && area !== '--help' && area !== '-h') process.exitCode = 1; if (area && area !== '--help' && area !== '-h') process.exitCode = 1;

View file

@ -18,7 +18,7 @@
// waybetter build-env -- tauri build run a command with that environment // waybetter build-env -- tauri build run a command with that environment
// waybetter build-env [--json] show it // waybetter build-env [--json] show it
// //
// `waybetter release windows` builds with it. It is not a proof: `waybetter privacy-scan` is. // `waybetter release windows` builds with it.
import { spawnSync } from 'node:child_process'; import { spawnSync } from 'node:child_process';
import { existsSync } from 'node:fs'; import { existsSync } from 'node:fs';
import { homedir } from 'node:os'; import { homedir } from 'node:os';

View file

@ -1,768 +0,0 @@
// `waybetter privacy-scan`: the release gate that keeps personal names and account names out of
// everything an app publishes. It reads a list of forbidden terms (one per line, kept OUTSIDE every
// repository), always adds this machine's account name and home folder name, and searches files,
// folders, archives, git-tracked files and (with --history) a whole repository for them.
//
// - Text and binary alike: every file is searched as bytes, in UTF-8, UTF-16LE and UTF-16BE.
// - Containers are opened: zip (and .nupkg/.jar), tar, gzip, zstd, ar/.deb, and with an external
// tool xz/bzip2 (xz, bzip2 or 7-Zip), NSIS installers, AppImage/squashfs (7-Zip or unsquashfs)
// and DMG (zlib images natively, others with 7-Zip). A container that could not be opened is an
// error, not a pass.
// - A term is never printed. A hit shows the file, line:column or byte offset, and the term as its
// first letter plus `***`; the context shown has every term masked the same way.
//
// Matching: a term is a hit as a whole word, case-insensitively (letters with diacritics count as
// their base letter). A term of three characters or fewer is usually also an ordinary word fragment
// (`max(`, `max-width`), so it is a hit only in a path (`Users\<it>`, `/home/<it>`, `\<it>\`,
// `/<it>/`, `Users-<it>-`) or as the capitalized standalone word.
//
// Reviewed false positives go in an allowlist file inside the repository: lines of
// `<file glob>: <regular expression>`, where the expression is tested against the hit's context with
// the term replaced by `<term>` (another hit nearby reads `<other>`). The file never contains a term (the scan refuses one that does).
import { spawnSync } from 'node:child_process';
import { closeSync, existsSync, lstatSync, mkdtempSync, openSync, readFileSync, readSync, readdirSync, rmSync, writeFileSync } from 'node:fs';
import { homedir, tmpdir, userInfo } from 'node:os';
import { basename, dirname, join, relative, resolve } from 'node:path';
import * as zlib from 'node:zlib';
export const TERMS_ENV = 'WAYBETTER_FORBIDDEN_TERMS';
export const SEVEN_ZIP_ENV = 'WAYBETTER_7Z';
const CONFIG_FILE = 'waybetter.release.json';
export const PRIVACY_USAGE = `Usage:
waybetter privacy-scan <paths...> [--terms <file>] [--allow <file>] [--tracked] [--untracked]
[--history] [--all] [--json] [--if-terms] [--no-auto]
Searches files, folders and archives for forbidden terms (personal names, account names) and
exits 1 on a hit, 2 when something could not be opened. Terms: --terms, else the file named by
${TERMS_ENV}, else "forbiddenTerms" in ${CONFIG_FILE}; this machine's account name and
home folder name are always added (--no-auto: not). A term is never printed.
--tracked each path is inside a git work tree: only the files git tracks (and their names)
--untracked with --tracked: also files git neither tracks nor ignores
--history each path is a repository: every blob, path, commit message, tag and identity
--allow reviewed false positives: lines of "<file glob>: <regex over the context, the
term written <term>>" (default: "privacyAllowlist" in ${CONFIG_FILE})
--all list every hit (default: the first 5 of each file)
--if-terms do nothing, quietly, when no terms file exists (for a contributor's test run)`;
class PrivacyError extends Error {}
const fail = message => { throw new PrivacyError(message); };
// ---------------------------------------------------------------------------------------------
// Terms
/** Account names that say nothing about a person: never added automatically. */
const GENERIC_ACCOUNTS = new Set(['root', 'user', 'users', 'admin', 'administrator', 'runner', 'runneradmin', 'ubuntu', 'build',
'builder', 'ci', 'docker', 'node', 'vagrant', 'default', 'nobody', 'home', 'dev', 'developer', 'owner', 'guest', 'test',
'app', 'work', 'vscode', 'codespace', 'gitpod', 'circleci', 'jenkins', 'buildkite', 'ec2-user', 'azureuser', 'pi', 'public']);
/** The lines of a terms file: one term each, `#` comments and blank lines skipped. */
export function parseTerms(text) {
return text.split(/\r?\n/).map(line => line.trim()).filter(line => line && !line.startsWith('#'));
}
const expandHome = value => value.replace(/^~(?=$|[\\/])/, homedir());
/** The nearest waybetter.release.json's privacy settings (paths made absolute), or {}. */
export function privacyConfig(cwd) {
for (let dir = resolve(cwd); ; dir = dirname(dir)) {
const file = join(dir, CONFIG_FILE);
if (existsSync(file)) {
let raw;
try { raw = JSON.parse(readFileSync(file, 'utf8')); } catch { return {}; }
const at = value => (typeof value === 'string' && value.trim() ? resolve(dir, expandHome(value.trim())) : undefined);
return { root: dir, forbiddenTerms: at(raw.forbiddenTerms), privacyAllowlist: at(raw.privacyAllowlist) };
}
if (dirname(dir) === dir) return {};
}
}
/**
* The terms to search for: the file's lines plus (unless `auto` is false) this machine's account
* name and home folder name. `file` may be missing (then only the automatic ones).
* Returns `{ terms: [{ text, rule }], fromFile }`; rule is "word" or "account" (see the header).
*/
export function collectTerms({ file, auto = true, extra = [] } = {}) {
const seen = new Map();
const add = (text, automatic) => {
const value = text.trim();
const key = value.toLowerCase();
if (value.length < 2 || seen.has(key)) return;
if (automatic && GENERIC_ACCOUNTS.has(key)) return;
seen.set(key, { text: value, rule: [...value].length <= 3 ? 'account' : 'word' });
};
let fromFile = 0;
if (file) {
for (const term of parseTerms(readFileSync(file, 'utf8'))) add(term, false);
fromFile = seen.size;
}
for (const term of extra) add(term, false);
if (auto) {
let user = '';
try { user = userInfo().username; } catch { /* no passwd entry */ }
add(user, true);
add(basename(homedir()), true);
}
return { terms: [...seen.values()], fromFile };
}
// ---------------------------------------------------------------------------------------------
// Matching (on bytes, read as a latin1 string so one regular expression sees every encoding)
const ENCODINGS = ['utf8', 'utf16le', 'utf16be'];
const FOLDS = { a: 'àáâãäåā', c: 'çćč', e: 'èéêëēęě', g: 'ğ', i: 'ìíîïıī', l: 'ł', n: 'ñńň', o: 'òóôõöøō', r: 'ř', s: 'şšśș', t: 'țť', u: 'ùúûüūů', y: 'ýÿ', z: 'žźż' };
function bytesOf(text, encoding) {
if (encoding === 'utf8') return Buffer.from(text, 'utf8');
const bytes = Buffer.from(text, 'utf16le');
return encoding === 'utf16be' ? bytes.swap16() : bytes;
}
const hex = bytes => [...bytes].map(byte => `\\x${byte.toString(16).padStart(2, '0')}`).join('');
/** One character in one encoding; `fold`: either case, and the letter's diacritic forms. */
function unit(char, encoding, fold) {
const forms = new Set([char]);
if (fold) {
const lower = char.toLowerCase();
for (const form of [lower, char.toUpperCase(), ...(FOLDS[lower] ?? '')]) {
forms.add(form);
forms.add(form.toUpperCase());
}
if (lower === 'i') forms.add('İ');
}
const encoded = [...new Set([...forms].filter(form => [...form].length === 1).map(form => hex(bytesOf(form, encoding))))];
return encoded.length === 1 ? encoded[0] : `(?:${encoded.join('|')})`;
}
const literal = (text, encoding, fold) => [...text].map(char => unit(char, encoding, fold)).join('');
const oneOf = (texts, encoding, fold) => `(?:${texts.map(text => literal(text, encoding, fold)).join('|')})`;
const ALNUM = 'A-Za-z0-9';
const BEFORE = { utf8: `(?<![${ALNUM}])`, utf16le: `(?<![${ALNUM}]\\x00)`, utf16be: `(?<!\\x00[${ALNUM}])` };
const AFTER = { utf8: `(?![${ALNUM}])`, utf16le: `(?![${ALNUM}]\\x00)`, utf16be: `(?!\\x00[${ALNUM}])` };
const SEPARATORS = ['\\', '/', '%5C', '%2F'];
/**
* The alternatives (regular expression sources) that make `term` a hit. `firm` leaves out the one
* that three random bytes satisfy (the capitalized word in UTF-8), see `stringLike`.
*/
function alternatives(term, firm = false) {
const out = [];
for (const encoding of ENCODINGS) {
const word = literal(term.text, encoding, true);
if (term.rule === 'word') {
out.push(`${BEFORE[encoding]}${word}${AFTER[encoding]}`);
continue;
}
const sep = oneOf(SEPARATORS, encoding, true);
// In a path: between separators, right after a Users/home folder, or in a dash-flattened path.
out.push(`(?<=${sep})${word}(?=${sep})`);
out.push(`(?<=${oneOf(['Users', 'home'], encoding, true)}${sep}+)${word}${AFTER[encoding]}`);
out.push(`(?<=${literal('Users-', encoding, true)})${word}(?=${literal('-', encoding, false)})`);
// The capitalized standalone word.
const [first, ...rest] = [...term.text];
const capitalized = literal(first.toUpperCase(), encoding, false) + literal(rest.join('').toLowerCase(), encoding, false);
if (!firm || encoding !== 'utf8') out.push(`${BEFORE[encoding]}${capitalized}${AFTER[encoding]}`);
}
return out;
}
/**
* Whether the bytes at [start, end) sit in text rather than in compressed or random data: part of a
* run of 12 or more printable characters, or a field between two NUL bytes. Megabytes of media hold
* every three-letter sequence by chance; they do not hold it inside a sentence.
*/
function stringLike(text, start, end) {
const printable = code => (code >= 0x20 && code <= 0x7e) || code === 9 || code === 10 || code === 13;
let run = end - start;
for (let at = start - 1; at >= 0 && run < 12 && printable(text.charCodeAt(at)); at -= 1) run += 1;
for (let at = end; at < text.length && run < 12 && printable(text.charCodeAt(at)); at += 1) run += 1;
if (run >= 12) return true;
return (start === 0 || text.charCodeAt(start - 1) === 0) && (end === text.length || text.charCodeAt(end) === 0);
}
/** First letter + `***`: the only form of a term that is ever printed. */
export const maskTerm = text => `${[...text][0]}***`;
/** A matcher over bytes for a term list. */
export function createMatcher(terms) {
const each = terms.map(term => ({ term, mask: maskTerm(term.text), source: alternatives(term).join('|') }));
const all = each.length ? new RegExp(each.map(entry => `(?:${entry.source})`).join('|'), 'g') : undefined;
const single = each.map(entry => new RegExp(entry.source, 'g'));
const firm = each.map(entry => (entry.term.rule === 'account' ? new RegExp(alternatives(entry.term, true).join('|'), 'g') : undefined));
const startsAt = (expression, window, at) => {
expression.lastIndex = at;
return expression.exec(window)?.index === at;
};
/** Every hit in `buffer`: `{ offset, length, mask, encoding }`, in order. `binary`: not a text file. */
function find(buffer, binary = false) {
if (!all || !buffer.length) return [];
const text = buffer.latin1Slice(0, buffer.length);
const hits = [];
all.lastIndex = 0;
let end = -1;
for (let match = all.exec(text); match; match = all.exec(text)) {
if (match.index >= end) {
end = match.index + match[0].length;
// Which term: the one whose own expression matches at this offset (seen with its context).
const from = Math.max(0, match.index - 64);
const window = text.slice(from, end + 8);
const which = single.findIndex(expression => startsAt(expression, window, match.index - from));
const mask = each[which]?.mask ?? '?***';
if (binary && firm[which] && !startsAt(firm[which], window, match.index - from) && !stringLike(text, match.index, end)) continue;
// (UTF-16 text matches in one byte order or, a byte earlier, in the other: the label names neither.)
hits.push({ offset: match.index, length: match[0].length, mask, encoding: match[0].includes('\x00') ? 'utf16' : 'utf8' });
}
if (match[0].length === 0) all.lastIndex += 1;
}
return hits;
}
/** `text` (a latin1 view of bytes) with every hit replaced by `mark` (default: the term's mask). */
function redact(text, mark) {
let out = text;
for (let i = 0; i < single.length; i += 1) {
single[i].lastIndex = 0;
out = out.replace(single[i], mark ?? each[i].mask);
}
return out;
}
return { find, redact, size: each.length };
}
// ---------------------------------------------------------------------------------------------
// Allowlist
const globToRegExp = glob => new RegExp(`^${glob.split('**').map(part => part.split('*').map(piece =>
piece.replace(/[.+^${}()|[\]\\]/g, '\\$&').replaceAll('?', '[^/]')).join('[^/]*')).join('.*')}$`);
/** Reads an allowlist file: `<file glob>: <regex>` lines. It must not contain a term itself. */
export function loadAllowlist(file, matcher) {
if (!file || !existsSync(file)) return [];
const raw = readFileSync(file);
if (matcher.find(raw).length) fail(`${file}: the allowlist must not contain a term (write <term> in its place)`);
const rules = [];
raw.toString('utf8').split(/\r?\n/).forEach((line, index) => {
const text = line.trim();
if (!text || text.startsWith('#')) return;
const cut = text.indexOf(': ');
if (cut <= 0) fail(`${file}:${index + 1}: expected "<file glob>: <regex>"`);
let pattern;
try { pattern = new RegExp(text.slice(cut + 2).trim()); } catch (error) { fail(`${file}:${index + 1}: ${error.message}`); }
rules.push({ glob: globToRegExp(text.slice(0, cut).trim()), pattern, line: index + 1, used: 0 });
});
return rules;
}
// ---------------------------------------------------------------------------------------------
// Containers
const startsWith = (buffer, bytes, at = 0) => buffer.length >= at + bytes.length && bytes.every((byte, i) => buffer[at + i] === byte);
const ascii = (buffer, from, length) => buffer.latin1Slice(Math.min(from, buffer.length), Math.min(from + length, buffer.length)).replace(/\0.*$/s, '');
function* tarEntries(buffer) {
let at = 0;
let longName;
while (at + 512 <= buffer.length) {
const header = buffer.subarray(at, at + 512);
if (header.every(byte => byte === 0)) break;
const sizeField = header.subarray(124, 136);
const size = sizeField[0] & 0x80
? Number(sizeField.subarray(4).readBigUInt64BE(0))
: parseInt(ascii(header, 124, 12).trim() || '0', 8) || 0;
const type = String.fromCharCode(header[156] || 48);
const data = buffer.subarray(at + 512, at + 512 + size);
at += 512 + Math.ceil(size / 512) * 512;
if (type === 'L') { longName = data.toString('utf8').replace(/\0+$/, ''); continue; }
if (type === 'x' || type === 'g') {
const path = /\d+ path=([^\n]*)\n/.exec(data.toString('utf8'));
if (type === 'x' && path) longName = path[1];
yield { name: `${longName ?? 'pax'} (pax header)`, data };
continue;
}
const prefix = ascii(header, 345, 155);
const name = longName ?? (prefix ? `${prefix}/${ascii(header, 0, 100)}` : ascii(header, 0, 100));
longName = undefined;
// The header also carries a link target and the owner's user and group names.
yield { name, data: type === '0' || type === '7' ? data : undefined, meta: [ascii(header, 157, 100), ascii(header, 265, 32), ascii(header, 297, 32)] };
}
}
function* zipEntries(buffer, note) {
const end = buffer.lastIndexOf(Buffer.from([0x50, 0x4b, 0x05, 0x06]));
if (end < 0) { note('zip: no central directory'); return; }
let count = buffer.readUInt16LE(end + 10);
let at = buffer.readUInt32LE(end + 16);
if (at === 0xffffffff || count === 0xffff) {
const locator = end - 20;
if (locator < 0 || !startsWith(buffer, [0x50, 0x4b, 0x06, 0x07], locator)) { note('zip64: no locator'); return; }
const record = Number(buffer.readBigUInt64LE(locator + 8));
count = Number(buffer.readBigUInt64LE(record + 32));
at = Number(buffer.readBigUInt64LE(record + 48));
}
yield { name: '(zip directory)', data: buffer.subarray(at, end + 22 + buffer.readUInt16LE(end + 20)), leaf: true };
for (let i = 0; i < count; i += 1) {
if (!startsWith(buffer, [0x50, 0x4b, 0x01, 0x02], at)) { note('zip: broken central directory'); return; }
const method = buffer.readUInt16LE(at + 10);
let size = buffer.readUInt32LE(at + 20);
let full = buffer.readUInt32LE(at + 24);
const nameLength = buffer.readUInt16LE(at + 28);
const extraLength = buffer.readUInt16LE(at + 30);
const commentLength = buffer.readUInt16LE(at + 32);
let local = buffer.readUInt32LE(at + 42);
const name = buffer.toString('utf8', at + 46, at + 46 + nameLength);
for (let extra = at + 46 + nameLength; extra + 4 <= at + 46 + nameLength + extraLength;) {
const id = buffer.readUInt16LE(extra);
const length = buffer.readUInt16LE(extra + 2);
if (id === 1) {
let field = extra + 4;
if (full === 0xffffffff) { full = Number(buffer.readBigUInt64LE(field)); field += 8; }
if (size === 0xffffffff) { size = Number(buffer.readBigUInt64LE(field)); field += 8; }
if (local === 0xffffffff) local = Number(buffer.readBigUInt64LE(field));
}
extra += 4 + length;
}
at += 46 + nameLength + extraLength + commentLength;
if (name.endsWith('/')) continue;
const start = local + 30 + buffer.readUInt16LE(local + 26) + buffer.readUInt16LE(local + 28);
const packed = buffer.subarray(start, start + size);
if (method === 0) yield { name, data: packed };
else if (method === 8) {
try { yield { name, data: zlib.inflateRawSync(packed) }; } catch (error) { note(`zip: ${name}: ${error.message}`); }
} else note(`zip: ${name}: compression method ${method}`);
}
}
function* arEntries(buffer) {
let at = 8;
while (at + 60 <= buffer.length) {
const name = ascii(buffer, at, 16).trim().replace(/\/$/, '');
const size = parseInt(ascii(buffer, at + 48, 10).trim(), 10) || 0;
yield { name: `${name} (ar header)`, data: buffer.subarray(at, at + 60), leaf: true };
yield { name, data: buffer.subarray(at + 60, at + 60 + size) };
at += 60 + size + (size % 2);
}
}
/** The decompressed partitions of a DMG whose chunks are raw or zlib; undefined when it has others. */
function dmgPartitions(buffer) {
const trailer = buffer.subarray(buffer.length - 512);
const dataFork = Number(trailer.readBigUInt64BE(24));
const xmlAt = Number(trailer.readBigUInt64BE(216));
const xml = buffer.toString('latin1', xmlAt, xmlAt + Number(trailer.readBigUInt64BE(224)));
const blocks = /<key>blkx<\/key>([\s\S]*?)<\/array>/.exec(xml)?.[1] ?? '';
const partitions = [];
for (const [, encoded] of blocks.matchAll(/<data>([\s\S]*?)<\/data>/g)) {
const table = Buffer.from(encoded.replace(/\s+/g, ''), 'base64');
if (ascii(table, 0, 4) !== 'mish') continue;
const parts = [];
for (let i = 0; i < table.readUInt32BE(200); i += 1) {
const entry = 204 + i * 40;
const type = table.readUInt32BE(entry);
const from = dataFork + Number(table.readBigUInt64BE(entry + 24));
const packed = buffer.subarray(from, from + Number(table.readBigUInt64BE(entry + 32)));
if (type === 1) parts.push(packed);
else if (type === 0x80000005) parts.push(zlib.inflateSync(packed));
else if (type === 0x80000006 || type === 0x80000007 || type === 0x80000008) return undefined;
}
partitions.push(Buffer.concat(parts));
}
return partitions;
}
/** The offset where an ELF file ends (an AppImage's squashfs payload starts there). */
function elfEnd(buffer) {
if (buffer[4] !== 2 || buffer[5] !== 1) return undefined;
return Number(buffer.readBigUInt64LE(0x28)) + buffer.readUInt16LE(0x3a) * buffer.readUInt16LE(0x3c);
}
/** Whether bytes start like a container this scan would open. */
const isContainer = head => startsWith(head, [0x1f, 0x8b]) || startsWith(head, [0x50, 0x4b, 0x03, 0x04]) || ascii(head, 257, 5) === 'ustar'
|| ascii(head, 0, 8) === '!<arch>\n' || startsWith(head, [0x28, 0xb5, 0x2f, 0xfd]) || startsWith(head, [0xfd, 0x37, 0x7a, 0x58, 0x5a, 0x00])
|| ascii(head, 0, 4) === 'hsqs' || startsWith(head, [0x4d, 0x5a]) || startsWith(head, [0x7f, 0x45, 0x4c, 0x46]);
let sevenZipPath;
function sevenZip(env) {
if (sevenZipPath !== undefined) return sevenZipPath;
const candidates = [env[SEVEN_ZIP_ENV], '7z', '7zz', '7za'];
if (process.platform === 'win32') {
for (const base of [env.ProgramFiles, env['ProgramFiles(x86)'], env.ProgramW6432]) if (base) candidates.push(join(base, '7-Zip', '7z.exe'));
}
sevenZipPath = candidates.filter(Boolean).find(file => spawnSync(file, ['i'], { stdio: 'ignore', windowsHide: true }).status === 0) ?? '';
return sevenZipPath;
}
/** Runs a filter (`xz -dc`, `bzip2 -dc`) over bytes; undefined when the tool is missing or fails. */
function pipe(tool, args, input) {
const result = spawnSync(tool, args, { input, maxBuffer: 2 ** 31 - 1, windowsHide: true });
return result.status === 0 ? result.stdout : undefined;
}
// ---------------------------------------------------------------------------------------------
// The scan
const CHUNK = 64 * 1024 * 1024;
const OVERLAP = 4096;
const utf8 = latin1 => Buffer.from(latin1, 'latin1').toString('utf8');
const printable = text => text.replace(/\0/g, '').replace(/[\x01-\x1f\x7f-\x9f]+/g, ' ').trim();
/**
* A scan over any number of inputs. `report()` gives
* `{ hits, files: [{ file, count, masks, hits: [{ where, mask, encoding, context }] }], unopened, scanned, bytes }`.
*/
export function createScan({ terms, allow = [], env = process.env } = {}) {
const matcher = createMatcher(terms);
const files = new Map();
const unopened = [];
let scanned = 0;
let bytes = 0;
function record(name, hit, where, context, allowContext) {
const file = safeName(name);
for (const rule of allow) {
if (rule.glob.test(file) && rule.pattern.test(allowContext)) { rule.used += 1; return; }
}
let entry = files.get(file);
if (!entry) files.set(file, entry = { file, count: 0, masks: {}, hits: [] });
entry.count += 1;
entry.masks[hit.mask] = (entry.masks[hit.mask] ?? 0) + 1;
entry.hits.push({ where, mask: hit.mask, encoding: hit.encoding, context });
}
const safeName = name => printable(utf8(matcher.redact(Buffer.from(name, 'utf8').latin1Slice())));
/** The allowlist's view of a hit: its surroundings with this hit written <term>, any other <other>. */
const marked = (buffer, from, to, hit) => printable(`${matcher.redact(buffer.latin1Slice(from, hit.offset), '<other>')}<term>${matcher.redact(buffer.latin1Slice(hit.offset + hit.length, to), '<other>')}`);
/** Searches bytes; `base`: their offset inside the file, `skip`: hits below this offset were seen. */
function bytesOfFile(file, buffer, { base = 0, skip = 0, textual } = {}) {
const isText = textual ?? !buffer.subarray(0, 8000).includes(0);
const hits = matcher.find(buffer, !isText);
if (!hits.length) return;
let line = 1;
let lineStart = 0;
let cursor = 0;
for (const hit of hits) {
if (hit.offset < skip) continue;
let where;
let from;
let to;
if (isText && hit.encoding === 'utf8') {
for (let next = buffer.indexOf(10, cursor); next >= 0 && next < hit.offset; next = buffer.indexOf(10, cursor)) {
line += 1;
lineStart = next + 1;
cursor = next + 1;
}
where = `${line}:${hit.offset - lineStart + 1}`;
const lineEnd = buffer.indexOf(10, hit.offset);
from = Math.max(lineStart, hit.offset - 100);
to = Math.min(lineEnd < 0 ? buffer.length : lineEnd, hit.offset + hit.length + 100);
} else {
where = `@0x${(base + hit.offset).toString(16)}`;
from = Math.max(0, hit.offset - 160);
to = Math.min(buffer.length, hit.offset + hit.length + 160);
}
const near = buffer.latin1Slice(Math.max(from, hit.offset - 60), Math.min(to, hit.offset + hit.length + 60));
record(file, hit, where, printable(matcher.redact(near)), marked(buffer, from, to, hit));
}
}
/** A name (a path inside an archive, a git path, an identity) searched as text. */
function text(file, value, label = 'name') {
const buffer = Buffer.from(value, 'utf8');
for (const hit of matcher.find(buffer)) {
const shown = printable(utf8(matcher.redact(buffer.latin1Slice(0, buffer.length))));
record(file, hit, label, shown.slice(0, 200), marked(buffer, 0, buffer.length, hit));
}
}
/** Extracts with 7-Zip (or unsquashfs) into a temporary folder, scans it, removes it. */
function extracted(file, buffer, depth, { type, tool = 'container' } = {}) {
const dir = mkdtempSync(join(tmpdir(), 'wb-privacy-'));
try {
const source = join(dir, 'in');
const out = join(dir, 'out');
writeFileSync(source, buffer);
const seven = sevenZip(env);
let done = false;
if (seven) {
done = spawnSync(seven, ['x', '-y', '-bd', `-o${out}`, ...(type ? [`-t${type}`] : []), source], { stdio: 'ignore', windowsHide: true }).status === 0;
}
if (!done && type === 'SquashFS') {
done = spawnSync('unsquashfs', ['-no-progress', '-d', out, source], { stdio: 'ignore' }).status === 0;
}
if (!done || !existsSync(out)) {
unopened.push(`${file}: ${tool} not opened (install 7-Zip, or set ${SEVEN_ZIP_ENV})`);
return;
}
folder(out, `${file}!`, depth + 1);
} finally {
rmSync(dir, { recursive: true, force: true });
}
}
/** One file's bytes: searched, and opened when they are a container. */
function content(file, buffer, depth = 0) {
const note = message => unopened.push(`${file}: ${message}`);
const inner = (entries) => {
for (const entry of entries) {
const name = `${file}!/${safeName(entry.name)}`;
text(name, entry.name);
for (const value of entry.meta ?? []) text(`${name} (tar header)`, value, 'owner');
if (entry.leaf) bytesOfFile(name, entry.data, { textual: false });
else if (entry.data) content(name, entry.data, depth + 1);
}
};
const filtered = (data, what) => {
if (data) content(file, data, depth + 1);
else note(`${what} not opened`);
};
if (depth === 0) { scanned += 1; bytes += buffer.length; }
if (depth > 8) { bytesOfFile(file, buffer); return; }
try {
if (startsWith(buffer, [0x1f, 0x8b])) {
bytesOfFile(`${file} (gzip header)`, buffer.subarray(0, 512), { textual: false });
let data;
try { data = zlib.gunzipSync(buffer); } catch (error) { note(`gzip: ${error.message}`); }
if (data) content(file, data, depth + 1);
} else if (startsWith(buffer, [0x50, 0x4b, 0x03, 0x04])) {
inner(zipEntries(buffer, note));
} else if (buffer.length > 512 && ascii(buffer, 257, 5) === 'ustar') {
inner(tarEntries(buffer));
} else if (ascii(buffer, 0, 8) === '!<arch>\n') {
inner(arEntries(buffer));
} else if (startsWith(buffer, [0x28, 0xb5, 0x2f, 0xfd])) {
let data;
try { data = zlib.zstdDecompressSync?.(buffer); } catch { /* reported below */ }
filtered(data ?? pipe('zstd', ['-dc'], buffer), 'zstd');
} else if (startsWith(buffer, [0xfd, 0x37, 0x7a, 0x58, 0x5a, 0x00])) {
filtered(pipe('xz', ['-dc'], buffer), 'xz (install xz)');
} else if (startsWith(buffer, [0x42, 0x5a, 0x68]) && buffer[3] >= 0x31 && buffer[3] <= 0x39 && startsWith(buffer, [0x31, 0x41, 0x59, 0x26, 0x53, 0x59], 4)) {
filtered(pipe('bzip2', ['-dc'], buffer), 'bzip2 (install bzip2)');
} else if (ascii(buffer, 0, 4) === 'hsqs') {
extracted(file, buffer, depth, { type: 'SquashFS', tool: 'squashfs' });
} else if (buffer.length > 512 && ascii(buffer, buffer.length - 512, 4) === 'koly') {
const partitions = dmgPartitions(buffer);
if (partitions) partitions.forEach((data, i) => bytesOfFile(`${file}!/partition ${i}`, data, { textual: false }));
else extracted(file, buffer, depth, { tool: 'dmg' });
} else if (startsWith(buffer, [0x7f, 0x45, 0x4c, 0x46]) && ascii(buffer, 8, 2) === 'AI' && buffer[10] === 2) {
bytesOfFile(file, buffer);
const end = elfEnd(buffer);
if (end && ascii(buffer, end, 4) === 'hsqs') extracted(file, buffer.subarray(end), depth, { type: 'SquashFS', tool: 'AppImage' });
else note('AppImage: payload not found');
} else if (startsWith(buffer, [0x4d, 0x5a]) && buffer.includes('NullsoftInst')) {
bytesOfFile(file, buffer);
extracted(file, buffer, depth, { type: 'Nsis', tool: 'NSIS installer' });
} else {
bytesOfFile(file, buffer);
}
} catch (error) {
if (error instanceof PrivacyError) throw error;
note(`not opened (${error.message})`);
bytesOfFile(file, buffer);
}
}
/** A file on disk under the name `file`. Large plain files are read in overlapping chunks. */
function diskFile(path, file, depth = 0) {
const size = lstatSync(path).size;
if (size <= CHUNK * 4) { content(file, readFileSync(path), depth); return; }
scanned += 1;
bytes += size;
const handle = openSync(path, 'r');
const buffer = Buffer.alloc(CHUNK + OVERLAP);
let container = false;
try {
container = isContainer(buffer.subarray(0, readSync(handle, buffer, 0, 512, 0)));
for (let at = 0; at < size; at += CHUNK) {
const from = Math.max(0, at - OVERLAP);
const read = readSync(handle, buffer, 0, Math.min(CHUNK + (at - from), size - from), from);
bytesOfFile(file, buffer.subarray(0, read), { base: from, skip: at - from, textual: false });
}
} finally {
closeSync(handle);
}
if (container) unopened.push(`${file}: a container larger than ${(CHUNK * 4) / 2 ** 20} MB, searched as plain bytes only`);
}
function folder(path, prefix, depth = 0) {
for (const entry of readdirSync(path, { withFileTypes: true }).sort((a, b) => (a.name < b.name ? -1 : 1))) {
const full = join(path, entry.name);
const file = `${prefix}/${safeName(entry.name)}`;
text(file, entry.name);
if (entry.isDirectory()) { if (entry.name !== '.git') folder(full, file, depth); }
else if (entry.isFile()) diskFile(full, file, depth);
}
}
/** A file, folder or archive, shown relative to `cwd`. */
function path(target, cwd = process.cwd()) {
const full = resolve(cwd, target);
if (!existsSync(full)) fail(`not found: ${safeName(target)}`);
// Outside the working directory only the name is shown (the folders above it are not published).
const inside = relative(cwd, full);
const outside = inside.startsWith('..') || resolve(cwd, inside) !== full;
const shown = safeName(outside ? basename(full) : inside.replaceAll('\\', '/') || '.');
text(shown, outside ? basename(full) : inside);
if (lstatSync(full).isDirectory()) folder(full, shown === '.' ? '' : shown);
else diskFile(full, shown);
}
const git = (repo, args, options = {}) => {
const result = spawnSync('git', ['-C', repo, ...args], { maxBuffer: 2 ** 31 - 1, windowsHide: true, ...options });
if (result.status !== 0) fail(`git ${args[0]} failed in ${safeName(repo)}: ${String(result.stderr ?? '').trim().split('\n')[0]}`);
return result.stdout;
};
/** The files git tracks under `target` (and, with `untracked`, the ones it neither tracks nor ignores). */
function tracked(target, { cwd = process.cwd(), untracked = false } = {}) {
const dir = resolve(cwd, target);
const list = git(dir, ['ls-files', '-z', ...(untracked ? ['--cached', '--others', '--exclude-standard'] : [])]).toString('utf8').split('\0').filter(Boolean);
for (const name of [...new Set(list)]) {
const full = join(dir, name);
const shown = safeName(relative(cwd, full).replaceAll('\\', '/'));
text(shown, name);
let stat;
try { stat = lstatSync(full); } catch { continue; } // deleted in the work tree
if (stat.isFile()) diskFile(full, shown);
}
}
/** Everything a repository would publish: blobs, paths, commit and tag messages, identities, ref names. */
function history(target, { cwd = process.cwd() } = {}) {
const dir = resolve(cwd, target);
const label = safeName(relative(cwd, dir).replaceAll('\\', '/') || '.');
const names = new Map();
for (const line of git(dir, ['rev-list', '--all', '--objects']).toString('utf8').split('\n')) {
const space = line.indexOf(' ');
if (space > 0 && line.length > space + 1 && !names.has(line.slice(0, space))) names.set(line.slice(0, space), line.slice(space + 1));
}
for (const [, name] of names) text(`${label} history: ${safeName(name)}`, name, 'path');
const ids = [...names.keys()];
const kinds = git(dir, ['cat-file', '--batch-check=%(objectname) %(objecttype) %(objectsize)'], { input: `${ids.join('\n')}\n` }).toString('utf8').split('\n');
const blobs = kinds.map(line => line.split(' ')).filter(parts => parts[1] === 'blob');
// In batches, so a repository with large media never sits in memory whole.
for (let i = 0; i < blobs.length;) {
const batch = [];
for (let size = 0; i < blobs.length && (size < CHUNK * 4 || !batch.length); i += 1) {
batch.push(blobs[i]);
size += Number(blobs[i][2]);
}
const out = git(dir, ['cat-file', '--batch'], { input: `${batch.map(parts => parts[0]).join('\n')}\n` });
let at = 0;
for (const [id, , size] of batch) {
at = out.indexOf(10, at) + 1;
content(`${label} history: ${safeName(names.get(id))} (blob ${id.slice(0, 10)})`, out.subarray(at, at + Number(size)));
at += Number(size) + 1;
}
}
const log = git(dir, ['log', '--all', '--format=%H%x00%an%x00%ae%x00%cn%x00%ce%x00%B%x01']).toString('utf8');
for (const entry of log.split('\x01')) {
const [id, author, authorMail, committer, committerMail, message] = entry.replace(/^\n/, '').split('\0');
if (!id) continue;
const commit = `${label} history: commit ${id.slice(0, 10)}`;
text(commit, `${author} <${authorMail}>`, 'author');
text(commit, `${committer} <${committerMail}>`, 'committer');
text(commit, message ?? '', 'message');
}
const refs = git(dir, ['for-each-ref', '--format=%(refname)%00%(taggername) %(taggeremail)%00%(contents)%01']).toString('utf8');
for (const entry of refs.split('\x01')) {
const [ref, tagger, message] = entry.replace(/^\n/, '').split('\0');
if (!ref) continue;
text(`${label} history: ${safeName(ref)}`, ref, 'ref');
if (tagger?.trim()) text(`${label} history: ${safeName(ref)}`, tagger, 'tagger');
if (ref.startsWith('refs/tags/') && message) text(`${label} history: ${safeName(ref)}`, message, 'tag message');
}
}
function report() {
const list = [...files.values()];
return { hits: list.reduce((sum, entry) => sum + entry.count, 0), files: list, unopened: [...unopened], scanned, bytes, terms: matcher.size };
}
return { path, tracked, history, content, text, report };
}
/** The report as lines (hits masked). `all`: every hit instead of the first five of each file. */
export function formatReport(report, { all = false, label = 'privacy-scan' } = {}) {
const lines = [];
for (const entry of report.files) {
const masks = Object.entries(entry.masks).map(([mask, count]) => `${mask} x${count}`).join(', ');
lines.push(`${entry.file}: ${entry.count} ${entry.count === 1 ? 'hit' : 'hits'} (${masks})`);
for (const hit of all ? entry.hits : entry.hits.slice(0, 5)) {
lines.push(` ${hit.where} ${hit.mask}${hit.encoding === 'utf8' ? '' : ` [${hit.encoding}]`} ${hit.context.slice(0, 160)}`);
}
if (!all && entry.hits.length > 5) lines.push(` ... ${entry.hits.length - 5} more (--all)`);
}
for (const item of report.unopened) lines.push(`not opened: ${item}`);
const size = report.bytes >= 2 ** 20 ? `${(report.bytes / 2 ** 20).toFixed(1)} MB` : `${report.bytes} bytes`;
lines.push(`${label}: ${report.hits} ${report.hits === 1 ? 'hit' : 'hits'} in ${report.files.length} ${report.files.length === 1 ? 'file' : 'files'}`
+ `${report.unopened.length ? `, ${report.unopened.length} not opened` : ''} (${report.scanned} files, ${size}, ${report.terms} terms)`);
return lines;
}
/**
* Scans artifacts and throws when one holds a term or could not be opened (the release gate).
* `paths`: files or folders; `terms`/`allowlist`: files (both optional).
*/
export function assertClean(paths, { terms, allowlist, cwd = process.cwd(), env = process.env, log = console.log, label = 'privacy-scan' } = {}) {
const file = env[TERMS_ENV] || terms;
if (file && !existsSync(file)) fail(`terms file not found (${TERMS_ENV} or "forbiddenTerms"): fix the path, it is the release gate`);
const collected = collectTerms({ file });
const matcher = createMatcher(collected.terms);
const scan = createScan({ terms: collected.terms, allow: loadAllowlist(allowlist, matcher), env });
for (const target of paths) scan.path(target, cwd);
const report = scan.report();
for (const line of formatReport(report, { label })) log(line);
if (report.hits) fail(`${report.hits} forbidden ${report.hits === 1 ? 'term' : 'terms'} in the artifacts: nothing was released`);
if (report.unopened.length) fail(`${report.unopened.length} ${report.unopened.length === 1 ? 'container' : 'containers'} could not be opened, so the artifacts are not proven clean`);
return report;
}
function flag(args, name) {
const i = args.indexOf(name);
if (i < 0) return undefined;
const [, value] = args.splice(i, 2);
if (value === undefined) fail(`${name} needs a value`);
return value;
}
function toggle(args, name) {
const i = args.indexOf(name);
if (i >= 0) args.splice(i, 1);
return i >= 0;
}
/** `waybetter privacy-scan ...`; returns the exit code (0 clean, 1 hits, 2 something not opened). */
export function privacyCommand(argv, { cwd = process.cwd(), env = process.env, log = console.log } = {}) {
const args = [...argv];
if (toggle(args, '--help') || toggle(args, '-h')) { log(PRIVACY_USAGE); return 0; }
const config = privacyConfig(cwd);
const explicit = flag(args, '--terms');
const allowFile = flag(args, '--allow');
const options = Object.fromEntries(['--tracked', '--untracked', '--history', '--all', '--json', '--if-terms', '--no-auto'].map(name => [name, toggle(args, name)]));
const unknown = args.find(arg => arg.startsWith('--'));
if (unknown) fail(`unknown option ${unknown}`);
const candidate = explicit ? resolve(cwd, expandHome(explicit)) : env[TERMS_ENV] ? resolve(cwd, expandHome(env[TERMS_ENV])) : config.forbiddenTerms;
const file = candidate && existsSync(candidate) ? candidate : undefined;
if (!file && options['--if-terms']) return 0;
if (explicit && !file) fail('--terms: file not found');
const collected = collectTerms({ file, auto: !options['--no-auto'] });
if (!collected.terms.length) fail('no terms: pass --terms <file> or set ' + TERMS_ENV);
const matcher = createMatcher(collected.terms);
const allow = loadAllowlist(allowFile ? resolve(cwd, allowFile) : config.privacyAllowlist, matcher);
const scan = createScan({ terms: collected.terms, allow, env });
const targets = args.length ? args : ['.'];
for (const target of targets) {
if (options['--history']) scan.history(target, { cwd });
if (options['--tracked']) scan.tracked(target, { cwd, untracked: options['--untracked'] });
if (!options['--history'] && !options['--tracked']) scan.path(target, cwd);
}
const report = scan.report();
if (options['--json']) log(JSON.stringify(report, null, 2));
else {
for (const line of formatReport(report, { all: options['--all'] })) log(line);
if (!file) log(`privacy-scan: no terms file, only this machine's account name was searched (--terms, ${TERMS_ENV})`);
}
return report.hits ? 1 : report.unopened.length ? 2 : 0;
}

View file

@ -17,17 +17,13 @@
// "prepareCommand": "bun install --frozen-lockfile", // remote/WSL builds: runs first in the unpacked sources // "prepareCommand": "bun install --frozen-lockfile", // remote/WSL builds: runs first in the unpacked sources
// "distribution": { "publisher": "CRT", "id": "CRT.Name", "homepage": "https://...", "license": "...", "shortDescription": "...", "downloadBase": "https://..." }, // "distribution": { "publisher": "CRT", "id": "CRT.Name", "homepage": "https://...", "license": "...", "shortDescription": "...", "downloadBase": "https://..." },
// // manifests written after each release step (see cli/manifests.mjs) // // manifests written after each release step (see cli/manifests.mjs)
// "exportExclude": ["verification"], // remote/WSL builds: app-repo paths left out of the upload // "exportExclude": ["verification"] // remote/WSL builds: app-repo paths left out of the upload
// "forbiddenTerms": "~/.better-twttr-keys/forbidden-names.txt", // optional, outside the repo
// "privacyAllowlist": "privacy-allowlist.txt" // optional, reviewed false positives
// } // }
// //
// Privacy: the Windows build runs with the path-free build environment (cli/buildenv.mjs), and // The Windows build runs with the path-free build environment (cli/buildenv.mjs). Every artifact
// every artifact (the installer and what it installs, the main binary, the front end, the dmg and // (the installer, the main binary, the front end, the dmg and the updater bundle) goes through the
// the updater bundle) goes through `waybetter privacy-scan` (cli/privacy.mjs) before anything is // optional release check (cli/releasecheck.mjs, WAYBETTER_RELEASE_CHECK) before anything is
// copied, signed or written: a forbidden term, or a container that cannot be opened (the NSIS // copied, signed or written.
// installer needs 7-Zip), stops the release. The terms are the file "forbiddenTerms" names (or
// WAYBETTER_FORBIDDEN_TERMS) plus this machine's account name and home folder name.
// //
// The private key is read only into the build's environment (Windows) or handed to the Tauri // The private key is read only into the build's environment (Windows) or handed to the Tauri
// signer by path (macOS). Nothing here prints, copies or moves it. // signer by path (macOS). Nothing here prints, copies or moves it.
@ -37,7 +33,7 @@ import { homedir } from 'node:os';
import { dirname, isAbsolute, join, relative, resolve } from 'node:path'; import { dirname, isAbsolute, join, relative, resolve } from 'node:path';
import { pathToFileURL } from 'node:url'; import { pathToFileURL } from 'node:url';
import { pathFreeBuildEnv } from './buildenv.mjs'; import { pathFreeBuildEnv } from './buildenv.mjs';
import { assertClean } from './privacy.mjs'; import { runReleaseCheck } from './releasecheck.mjs';
import { CHANNELS, resolveSpec, writeManifests } from './manifests.mjs'; import { CHANNELS, resolveSpec, writeManifests } from './manifests.mjs';
import { buildLinuxWsl, buildMacRemote, scanBuilt } from './remote.mjs'; import { buildLinuxWsl, buildMacRemote, scanBuilt } from './remote.mjs';
@ -65,10 +61,9 @@ export const RELEASE_USAGE = `Usage:
linux-x86_64 to latest.json (run the Windows step first). linux-x86_64 to latest.json (run the Windows step first).
Config: ${CONFIG_FILE} in the app's root (product, artifact, desktopDir, keysDir, keysDirEnv, Config: ${CONFIG_FILE} in the app's root (product, artifact, desktopDir, keysDir, keysDirEnv,
cdnBase, apiBase, outDir, optional manifestValidator, forbiddenTerms, privacyAllowlist). Env: cdnBase, apiBase, outDir, optional manifestValidator). Env: CARGO_TARGET_DIR, CDN_BASE,
CARGO_TARGET_DIR, CDN_BASE, DEPLOY_TARGET, CDN_DIR, RELEASES_DIR, WAYBETTER_FORBIDDEN_TERMS, DEPLOY_TARGET, CDN_DIR, RELEASES_DIR, WAYBETTER_RELEASE_CHECK (an optional command run on every
WAYBETTER_7Z. The build is path-free (waybetter build-env) and every artifact must pass artifact first). The build is path-free (waybetter build-env). Nothing is ever uploaded.`;
waybetter privacy-scan. Nothing is ever uploaded.`;
class ReleaseError extends Error {} class ReleaseError extends Error {}
const fail = message => { throw new ReleaseError(message); }; const fail = message => { throw new ReleaseError(message); };
@ -127,8 +122,6 @@ export function normalizeReleaseConfig(raw, root) {
distribution: raw.distribution, distribution: raw.distribution,
exportExclude: Array.isArray(raw.exportExclude) ? raw.exportExclude.map(String) : [], exportExclude: Array.isArray(raw.exportExclude) ? raw.exportExclude.map(String) : [],
manifestValidator: validator && { module: at(validator.module), export: validator.export, hint: validator.hint }, manifestValidator: validator && { module: at(validator.module), export: validator.export, hint: validator.hint },
forbiddenTerms: raw.forbiddenTerms === undefined ? undefined : at(home(text('forbiddenTerms'))),
privacyAllowlist: raw.privacyAllowlist === undefined ? undefined : at(text('privacyAllowlist')),
}; };
} }
@ -179,9 +172,9 @@ function readVersion(config) {
return { version, productName: conf.productName, binary, dist }; return { version, productName: conf.productName, binary, dist };
} }
/** The release gate: no artifact may hold a forbidden term (see cli/privacy.mjs). */ /** The optional release check (cli/releasecheck.mjs) over every artifact that exists. */
function gate(config, paths, env, log) { function gate(config, paths, env, log) {
assertClean(paths.filter(path => path && existsSync(path)), { terms: config.forbiddenTerms, allowlist: config.privacyAllowlist, cwd: config.root, env, log, label: 'release: privacy-scan' }); runReleaseCheck(paths.filter(path => path && existsSync(path)), { env, log, label: 'release: check' });
} }
function option(args, name) { function option(args, name) {

View file

@ -0,0 +1,15 @@
// An optional, owner-supplied check run on every artifact before a release copies, signs or writes
// anything: WAYBETTER_RELEASE_CHECK names a command (run through the shell) that gets the artifact
// paths as arguments. A non-zero exit stops the release. Unset, nothing runs.
import { spawnSync } from 'node:child_process';
export const RELEASE_CHECK_ENV = 'WAYBETTER_RELEASE_CHECK';
export function runReleaseCheck(paths, { env = process.env, log = console.log, label = 'release check' } = {}) {
const command = env[RELEASE_CHECK_ENV];
if (!command || !paths.length) return;
const quoted = paths.map(path => `'${String(path).replaceAll("'", "'\\''")}'`).join(' ');
log(`${label}: ${paths.length} file(s)`);
const result = spawnSync(`${command} ${quoted}`, { shell: true, stdio: 'inherit', env });
if (result.status !== 0) throw new Error(`${label} failed (exit ${result.status ?? result.signal})`);
}

View file

@ -1,7 +1,7 @@
// Remote and WSL builds for `waybetter release macos|linux`: the committed state of an app (its // Remote and WSL builds for `waybetter release macos|linux`: the committed state of an app (its
// git HEAD plus every submodule at the commit the app records) is exported, sent to the machine // git HEAD plus every submodule at the commit the app records) is exported, sent to the machine
// that builds it, built there through `waybetter build-env` (no build path in the binary) and the // that builds it, built there through `waybetter build-env` (no build path in the binary) and the
// artifacts come back to be scanned (`waybetter privacy-scan`) and, later, signed on this machine. // artifacts come back to be checked (the optional release check) and, later, signed on this machine.
// Nothing here touches the working tree the developer has open, and no key ever leaves this // Nothing here touches the working tree the developer has open, and no key ever leaves this
// machine: the remote build is unsigned (updater artifacts off), signing is `waybetter release`. // machine: the remote build is unsigned (updater artifacts off), signing is `waybetter release`.
// //
@ -18,7 +18,7 @@ import { spawnSync } from 'node:child_process';
import { existsSync, mkdirSync, mkdtempSync, readdirSync, readFileSync, rmSync, statSync, writeFileSync } from 'node:fs'; import { existsSync, mkdirSync, mkdtempSync, readdirSync, readFileSync, rmSync, statSync, writeFileSync } from 'node:fs';
import { tmpdir } from 'node:os'; import { tmpdir } from 'node:os';
import { basename, join, relative, resolve } from 'node:path'; import { basename, join, relative, resolve } from 'node:path';
import { assertClean } from './privacy.mjs'; import { runReleaseCheck } from './releasecheck.mjs';
class RemoteError extends Error {} class RemoteError extends Error {}
const fail = message => { throw new RemoteError(message); }; const fail = message => { throw new RemoteError(message); };
@ -250,11 +250,11 @@ export function buildLinuxWsl({ config, distro = 'Ubuntu-22.04', remoteDir, ref
const relTauri = (config, name) => `${relative(config.root, config.tauriDir).replaceAll('\\', '/')}/${name}`; const relTauri = (config, name) => `${relative(config.root, config.tauriDir).replaceAll('\\', '/')}/${name}`;
/** Scans built artifacts here (the terms never leave this machine) and prints their sizes. */ /** Prints the built artifacts' sizes and runs the optional release check on them here. */
export function scanBuilt(config, files, { env = process.env, log = console.log } = {}) { export function scanBuilt(config, files, { env = process.env, log = console.log } = {}) {
for (const file of files) { for (const file of files) {
if (!existsSync(file)) fail(`expected build output missing: ${file}`); if (!existsSync(file)) fail(`expected build output missing: ${file}`);
log(`built: ${basename(file)} ${(statSync(file).size / 1048576).toFixed(1)} MB`); log(`built: ${basename(file)} ${(statSync(file).size / 1048576).toFixed(1)} MB`);
} }
assertClean(files, { terms: config.forbiddenTerms, allowlist: config.privacyAllowlist, cwd: config.root, env, log, label: 'release: privacy-scan' }); runReleaseCheck(files, { env, log, label: 'release: check' });
} }

View file

@ -3,7 +3,7 @@
"version": "0.7.0", "version": "0.7.0",
"license": "GPL-3.0-or-later", "license": "GPL-3.0-or-later",
"type": "module", "type": "module",
"description": "wayBetter: the framework for apps that run the apps you already use in a light native (Tauri) window and extend them in place. Page runtime (DOM/request hubs, plugins, plugin UI kit with themed dialogs and toasts, theme API and theme JSON Schema, theme color model, link routing, account reporting, deck columns, i18n: ICU-lite messages, locale fallback, plugin strings, catalog checks and JSON Schemas), shell models (address bar, panes, decks, profiles, palette, updates), the shell frame (createShell: tabs, splits, address bar, titlebar, palette, account switcher, decks, workspaces, overlays, the language state; React + dockview peers), the panel (createPanel: an app's own tab with Appearance and the Theme Builder, plugins and custom CSS, the Store, Make, accounts, settings, backup and sync, developer mode, updates; the app adds its sections) and the Store: install screen and Store view React components (optional React peer), their state and catalog models, the store catalog JSON Schema and the `waybetter store build|sign|verify` command, `waybetter release windows|macos` (signed native releases and the update manifest, configured by waybetter.release.json; path-free builds and a privacy gate), `waybetter privacy-scan` (forbidden names in files, binaries, archives and git history), `waybetter build-env` (a build without the machine's paths) and `waybetter/build` (the page runtime bundle, with an optional esbuild peer). No dependencies.", "description": "wayBetter: the framework for apps that run the apps you already use in a light native (Tauri) window and extend them in place. Page runtime (DOM/request hubs, plugins, plugin UI kit with themed dialogs and toasts, theme API and theme JSON Schema, theme color model, link routing, account reporting, deck columns, i18n: ICU-lite messages, locale fallback, plugin strings, catalog checks and JSON Schemas), shell models (address bar, panes, decks, profiles, palette, updates), the shell frame (createShell: tabs, splits, address bar, titlebar, palette, account switcher, decks, workspaces, overlays, the language state; React + dockview peers), the panel (createPanel: an app's own tab with Appearance and the Theme Builder, plugins and custom CSS, the Store, Make, accounts, settings, backup and sync, developer mode, updates; the app adds its sections) and the Store: install screen and Store view React components (optional React peer), their state and catalog models, the store catalog JSON Schema and the `waybetter store build|sign|verify` command, `waybetter release windows|macos` (signed native releases and the update manifest, configured by waybetter.release.json; path-free builds), `waybetter build-env` (a build without the machine's paths) and `waybetter/build` (the page runtime bundle, with an optional esbuild peer). No dependencies.",
"keywords": [ "keywords": [
"waybetter", "waybetter",
"tauri", "tauri",
@ -190,7 +190,6 @@
"./cli/store": "./cli/store.mjs", "./cli/store": "./cli/store.mjs",
"./cli/release": "./cli/release.mjs", "./cli/release": "./cli/release.mjs",
"./cli/i18n": "./cli/i18n.mjs", "./cli/i18n": "./cli/i18n.mjs",
"./cli/privacy": "./cli/privacy.mjs",
"./cli/buildenv": "./cli/buildenv.mjs", "./cli/buildenv": "./cli/buildenv.mjs",
"./build": "./build/runtime.mjs", "./build": "./build/runtime.mjs",
"./package.json": "./package.json" "./package.json": "./package.json"

View file

@ -1,288 +0,0 @@
// `waybetter privacy-scan` and `waybetter build-env`: matching rules, masking, containers, git,
// the allowlist, exit codes, the path-free build flags and the release gate. The terms here are
// invented ("zorblat", "qix"); no real name belongs in a test.
import assert from 'node:assert/strict';
import { execFileSync } from 'node:child_process';
import { existsSync, mkdirSync, mkdtempSync, writeFileSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import test from 'node:test';
import { fileURLToPath } from 'node:url';
import { deflateRawSync, gzipSync } from 'node:zlib';
import { pathFreeBuildEnv } from '../cli/buildenv.mjs';
import { collectTerms, createMatcher, createScan, formatReport, loadAllowlist, privacyCommand } from '../cli/privacy.mjs';
import { releaseCommand } from '../cli/release.mjs';
const BIN = fileURLToPath(new URL('../bin/waybetter.mjs', import.meta.url));
const TERMS = [{ text: 'zorblat', rule: 'word' }, { text: 'qix', rule: 'account' }];
const dir = () => mkdtempSync(join(tmpdir(), 'wb-privacy-test-'));
const hitsIn = text => createMatcher(TERMS).find(Buffer.isBuffer(text) ? text : Buffer.from(text)).length;
/** A tar archive of `{ name, data, user }` entries (ustar headers, like npm pack writes). */
function tar(entries) {
const blocks = [];
for (const { name, data, user = '' } of entries) {
const header = Buffer.alloc(512);
header.write(name, 0);
header.write('0000644\0', 100);
header.write(`${data.length.toString(8).padStart(11, '0')}\0`, 124);
header.write('0', 156);
header.write('ustar\0' + '00', 257);
header.write(user, 265);
blocks.push(header, data, Buffer.alloc((512 - (data.length % 512)) % 512));
}
return Buffer.concat([...blocks, Buffer.alloc(1024)]);
}
/** A zip archive of `{ name, data, stored }` entries. */
function zip(entries) {
const locals = [];
const central = [];
let offset = 0;
for (const { name, data, stored } of entries) {
const packed = stored ? data : deflateRawSync(data);
const local = Buffer.alloc(30 + name.length);
local.writeUInt32LE(0x04034b50, 0);
local.writeUInt16LE(stored ? 0 : 8, 8);
local.writeUInt32LE(packed.length, 18);
local.writeUInt32LE(data.length, 22);
local.writeUInt16LE(name.length, 26);
local.write(name, 30);
const entry = Buffer.alloc(46 + name.length);
entry.writeUInt32LE(0x02014b50, 0);
entry.writeUInt16LE(stored ? 0 : 8, 10);
entry.writeUInt32LE(packed.length, 20);
entry.writeUInt32LE(data.length, 24);
entry.writeUInt16LE(name.length, 28);
entry.writeUInt32LE(offset, 42);
entry.write(name, 46);
locals.push(local, packed);
central.push(entry);
offset += local.length + packed.length;
}
const directory = Buffer.concat(central);
const end = Buffer.alloc(22);
end.writeUInt32LE(0x06054b50, 0);
end.writeUInt16LE(entries.length, 8);
end.writeUInt16LE(entries.length, 10);
end.writeUInt32LE(directory.length, 12);
end.writeUInt32LE(offset, 16);
return Buffer.concat([...locals, directory, end]);
}
test('a name is a hit as a whole word, in any case, with diacritics; never inside another word', () => {
for (const text of ['by zorblat', 'ZORBLAT', 'Zorblat!', 'x_zorblat_y', 'a-zorblat-b', '@zorblat', 'zórblat', 'ZÖRBLAT', 'dir/zorblat/file']) {
assert.equal(hitsIn(text), 1, text);
}
for (const text of ['zorblats', 'azorblat', 'zorblat2', 'zorbla t']) assert.equal(hitsIn(text), 0, text);
});
test('a short account name is a hit only in a path or as the capitalized word', () => {
for (const text of ['qix(a, b)', 'qix-width: 3px', 'Math.qix', 'const qix = 1', 'QIX_SIZE', 'a qix b', 'qixed', 'src/qixes/a']) {
assert.equal(hitsIn(text), 0, text);
}
for (const text of ['C:\\Users\\qix\\Desktop', 'c:\\users\\QIX\\x', 'C:\\\\Users\\\\qix\\\\x', '/Users/qix', '/home/qix/.cargo', 'a\\qix\\b', 'a/qix/b',
'C--Users-qix-Desktop', 'file:///C:/Users/qix/x', 'C%3A%5CUsers%5Cqix%5Cx', 'by Qix.', 'Qix']) {
assert.equal(hitsIn(text), 1, text);
}
});
test('binary content: UTF-16 in either byte order, with the offset', () => {
const wide = Buffer.concat([Buffer.from([1, 2, 0, 0, 3, 0]), Buffer.from('C:\\Users\\qix\\AppData zorblat', 'utf16le'), Buffer.from([0, 0, 9])]);
assert.equal(hitsIn(wide), 2);
assert.equal(hitsIn(Buffer.from('\u0000:/zorblat/x', 'utf16le').swap16()), 1);
const scan = createScan({ terms: TERMS });
scan.content('app.exe', wide);
const [file] = scan.report().files;
assert.deepEqual(file.hits.map(hit => hit.encoding), ['utf16', 'utf16']);
assert.match(file.hits[0].where, /^@0x[0-9a-f]+$/);
});
test('binary content: three chance bytes are not a name; a field or a sentence is', () => {
const noise = Buffer.from([0, 0x91, 0xfe, 0x07, 0xc4]);
const count = (...parts) => {
const scan = createScan({ terms: TERMS });
scan.content('image.png', Buffer.concat([noise, ...parts.map(part => (typeof part === 'string' ? Buffer.from(part) : part)), noise]));
return scan.report().hits;
};
assert.equal(count('Qix'), 0);
assert.equal(count('k;Qix_'), 0);
assert.equal(count(Buffer.from([0]), 'Qix', Buffer.from([0])), 1);
assert.equal(count('Author: Qix, 2026'), 1);
assert.equal(count('C:\\Users\\qix\\x'), 1);
assert.equal(count('zorblat'), 1);
});
test('the report masks every term: first letter and ***, never the word', () => {
const scan = createScan({ terms: TERMS });
scan.content('notes/zorblat.txt', Buffer.from('first line\nwritten by Zorblat on C:\\Users\\qix\\notes, zorblat again\n'));
scan.text('notes/zorblat.txt', 'notes/zorblat.txt');
const report = scan.report();
assert.equal(report.hits, 4);
assert.equal(report.files[0].hits[0].where, '2:12');
const text = formatReport(report, { all: true }).join('\n');
assert.doesNotMatch(text, /zorblat|qix/i);
assert.match(text, /^notes\/z\*\*\*\.txt: 4 hits \(z\*\*\* x3, q\*\*\* x1\)/);
assert.match(text, /written by z\*\*\* on C:\\Users\\q\*\*\*\\notes/);
});
test('containers are opened: tar.gz (and its owner names), zip stored and deflated, nested', () => {
const inner = tar([{ name: 'package/readme.md', data: Buffer.from('made by zorblat') }, { name: 'package/clean.js', data: Buffer.from('ok'), user: 'zorblat' }]);
const archive = zip([
{ name: 'app/a.txt', data: Buffer.from('/home/qix/project'), stored: true },
{ name: 'app/b.bin', data: Buffer.concat([Buffer.alloc(40), Buffer.from('zorblat')]) },
{ name: 'app/zorblat.cfg', data: Buffer.from('nothing') },
{ name: 'app/pkg.tgz', data: gzipSync(inner) },
]);
const scan = createScan({ terms: TERMS });
scan.content('bundle.zip', archive);
const found = Object.fromEntries(scan.report().files.map(entry => [entry.file, entry.count]));
assert.deepEqual(found, {
'bundle.zip!/(zip directory)': 1,
'bundle.zip!/app/a.txt': 1,
'bundle.zip!/app/b.bin': 1,
'bundle.zip!/app/z***.cfg': 1,
'bundle.zip!/app/pkg.tgz!/package/readme.md': 1,
'bundle.zip!/app/pkg.tgz!/package/clean.js (tar header)': 1,
});
assert.equal(scan.report().files.at(-1).hits[0].where, 'owner');
assert.deepEqual(scan.report().unopened, []);
const clean = createScan({ terms: TERMS });
clean.content('ok.tgz', gzipSync(tar([{ name: 'package/index.js', data: Buffer.from('Math.max(qix, 1)') }])));
assert.equal(clean.report().hits, 0);
});
test('a container that cannot be opened is reported, not passed', () => {
const scan = createScan({ terms: TERMS });
const broken = gzipSync(Buffer.from('zorblat')).subarray(0, 14);
scan.content('broken.gz', broken);
assert.equal(scan.report().hits, 0);
assert.equal(scan.report().unopened.length, 1);
});
test('terms file: comments skipped, short terms get the account rule, generic accounts never added', () => {
const root = dir();
writeFileSync(join(root, 'terms.txt'), '# names\nzorblat\n\n Qix \nzorblat\n');
const { terms, fromFile } = collectTerms({ file: join(root, 'terms.txt'), auto: false });
assert.deepEqual(terms, [{ text: 'zorblat', rule: 'word' }, { text: 'Qix', rule: 'account' }]);
assert.equal(fromFile, 2);
assert.ok(!collectTerms({}).terms.some(term => ['root', 'runner', 'user'].includes(term.text.toLowerCase())));
});
test('allowlist: a reviewed context is skipped; the file may not hold a term', () => {
const root = dir();
writeFileSync(join(root, 'allow.txt'), '# a generated identifier\nsrc/**/*.rs: <term>-Id\n');
const matcher = createMatcher(TERMS);
const allow = loadAllowlist(join(root, 'allow.txt'), matcher);
const scan = createScan({ terms: TERMS, allow });
scan.content('src/net/cookie.rs', Buffer.from('header("Qix-Id=3600")\n// Qix wrote this\n'));
scan.content('docs/cookie.md', Buffer.from('Qix-Id\n'));
const report = scan.report();
assert.deepEqual(report.files.map(entry => [entry.file, entry.hits.map(hit => hit.where)]), [['src/net/cookie.rs', ['2:4']], ['docs/cookie.md', ['1:1']]]);
writeFileSync(join(root, 'bad.txt'), 'src/**: by zorblat\n');
assert.throws(() => loadAllowlist(join(root, 'bad.txt'), matcher), /must not contain a term/);
});
test('git: tracked files, then the history (blobs, paths, messages, identities)', () => {
const root = dir();
const git = (...args) => execFileSync('git', ['-C', root, '-c', 'user.name=Zorblat Dev', '-c', 'user.email=dev@example.test', '-c', 'core.autocrlf=false', '-c', `core.hooksPath=${join(root, '.no-hooks')}`, ...args], { stdio: 'pipe' });
git('init', '-q');
writeFileSync(join(root, 'notes.txt'), 'C:\\Users\\qix\\secret\n');
writeFileSync(join(root, 'ignored.log'), 'zorblat\n');
writeFileSync(join(root, '.gitignore'), '*.log\n');
git('add', '.');
git('commit', '-q', '-m', 'first, thanks zorblat');
writeFileSync(join(root, 'notes.txt'), 'C:\\Users\\<user>\\secret\n');
git('commit', '-q', '-am', 'paths made neutral');
const tracked = createScan({ terms: TERMS });
tracked.tracked('.', { cwd: root });
assert.equal(tracked.report().hits, 0);
const history = createScan({ terms: TERMS });
history.history('.', { cwd: root });
const where = history.report().files.flatMap(entry => entry.hits.map(hit => `${entry.file.replace(/[0-9a-f]{10}/, 'ID')} ${hit.where}`)).sort();
assert.deepEqual(where, [
'. history: commit ID author', '. history: commit ID author',
'. history: commit ID committer', '. history: commit ID committer',
'. history: commit ID message',
'. history: notes.txt (blob ID) 1:10',
]);
});
test('the command: exit 1 on a hit, 0 when clean, quiet with --if-terms and no terms file', () => {
const root = dir();
mkdirSync(join(root, 'dist'));
writeFileSync(join(root, 'terms.txt'), 'zorblat\n');
writeFileSync(join(root, 'dist/app.js'), 'console.log("clean")\n');
const lines = [];
const run = args => privacyCommand(args, { cwd: root, env: {}, log: line => lines.push(line) });
assert.equal(run(['dist', '--terms', 'terms.txt', '--no-auto']), 0);
assert.match(lines.at(-1), /^privacy-scan: 0 hits in 0 files \(1 files, /);
writeFileSync(join(root, 'dist/app.js.map'), '{"sources":["/home/zorblat/app/src/main.ts"]}');
assert.equal(run(['dist', '--terms', 'terms.txt', '--no-auto']), 1);
assert.doesNotMatch(lines.join('\n'), /zorblat/);
lines.length = 0;
assert.equal(run(['dist', '--if-terms']), 0);
assert.deepEqual(lines, []);
assert.throws(() => run(['dist', '--terms', 'missing.txt']), /not found/);
assert.equal(privacyCommand(['dist'], { cwd: root, env: { WAYBETTER_FORBIDDEN_TERMS: join(root, 'terms.txt') }, log: () => {} }), 1);
const out = execFileSync(process.execPath, [BIN, '--help'], { encoding: 'utf8' });
assert.match(out, /waybetter privacy-scan/);
assert.match(out, /waybetter build-env/);
});
test('build-env: roots remapped general to specific, existing flags kept, per platform', () => {
const windows = pathFreeBuildEnv({ root: 'C:\\Users\\dev\\app', home: 'C:\\Users\\dev', platform: 'win32', env: { RUSTFLAGS: '-Ctarget-cpu=native --cfg foo' } });
const flags = windows.env.CARGO_ENCODED_RUSTFLAGS.split('\x1f');
assert.deepEqual(flags.slice(0, 2), ['-Ctarget-cpu=native', '--cfg']);
assert.ok(flags.indexOf('--remap-path-prefix=C:\\Users\\dev=/home') < flags.indexOf('--remap-path-prefix=C:\\Users\\dev\\app=/src'));
assert.ok(flags.indexOf('--remap-path-prefix=C:\\Users\\dev\\app=/src') < flags.indexOf('--remap-path-prefix=C:\\Users\\dev\\.cargo=/cargo'));
assert.ok(flags.includes('--remap-path-prefix=\\\\?\\C:\\Users\\dev\\.rustup=/rustup'));
assert.equal(flags.at(-1), '-Clink-arg=/PDBALTPATH:%_PDB%');
assert.equal(windows.env.CFLAGS, undefined);
// MSVC C and C++: cl.exe takes the first /d1trimfile that matches, so the longest root leads.
assert.deepEqual(windows.env.CL.split(' '), ['C:/Users/dev/.rustup', 'C:/Users/dev/.cargo', 'C:/Users/dev/app', 'C:/Users/dev'].map(dir => `"/d1trimfile:${dir.replaceAll('/', '\\')}/"`));
assert.equal(windows.env.LINK, '/PDBALTPATH:%_PDB%');
const kept = pathFreeBuildEnv({ root: 'D:\\my apps\\app', home: 'C:\\Users\\dev', platform: 'win32', env: { CL: '/MP', LINK: '/DEBUG:NONE', CARGO_TARGET_DIR: 'D:\\t' } });
assert.ok(kept.env.CL.startsWith('/MP "/d1trimfile:C:\\Users\\dev\\.rustup/" '));
assert.ok(kept.env.CL.includes(' "/d1trimfile:D:\\my apps\\app/" ') && kept.env.CL.endsWith(' "/d1trimfile:D:\\t/"'));
assert.equal(kept.env.LINK, '/DEBUG:NONE /PDBALTPATH:%_PDB%');
const gnu = pathFreeBuildEnv({ root: 'C:\\Users\\dev\\app', home: 'C:\\Users\\dev', platform: 'win32', env: { CARGO_BUILD_TARGET: 'x86_64-pc-windows-gnu' } });
assert.equal(gnu.env.CL, undefined);
assert.equal(gnu.env.LINK, undefined);
const encoded = pathFreeBuildEnv({ root: '/home/dev/app', home: '/home/dev', platform: 'linux', env: { CARGO_ENCODED_RUSTFLAGS: '-Cfoo\x1f--remap-path-prefix=/home/dev=/home', CARGO_HOME: '/opt/cargo', CARGO_TARGET_DIR: '/tmp/t', CFLAGS: '-O2' } });
assert.deepEqual(encoded.env.CARGO_ENCODED_RUSTFLAGS.split('\x1f'), ['-Cfoo', '--remap-path-prefix=/home/dev=/home', '--remap-path-prefix=/home/dev/app=/src',
'--remap-path-prefix=/tmp/t=/target', '--remap-path-prefix=/opt/cargo=/cargo', '--remap-path-prefix=/home/dev/.rustup=/rustup']);
assert.match(encoded.env.CFLAGS, /^-O2 -ffile-prefix-map=\/home\/dev=\/home .*-ffile-prefix-map=\/opt\/cargo=\/cargo/);
assert.equal(encoded.env.CXXFLAGS.split(' ').length, 5);
assert.equal(encoded.env.CL, undefined);
});
test('release: an artifact holding a term stops the release before anything is written', async () => {
const root = dir();
const tauri = join(root, 'apps/desktop/src-tauri');
const nsis = join(root, 'target/release/bundle/nsis');
mkdirSync(tauri, { recursive: true });
mkdirSync(nsis, { recursive: true });
writeFileSync(join(root, 'terms.txt'), 'zorblat\n');
writeFileSync(join(root, 'waybetter.release.json'), JSON.stringify({ product: 'Test', artifact: 'Test', desktopDir: 'apps/desktop', keysDir: 'keys',
cdnBase: 'https://cdn.example.test', apiBase: 'https://api.example.test', forbiddenTerms: 'terms.txt' }));
for (const file of ['package.json', 'apps/desktop/package.json']) writeFileSync(join(root, file), JSON.stringify({ version: '1.0.0' }));
writeFileSync(join(tauri, 'tauri.conf.json'), JSON.stringify({ version: '1.0.0', productName: 'Test App' }));
writeFileSync(join(tauri, 'Cargo.toml'), '[package]\nname = "t"\nversion = "1.0.0"\n');
writeFileSync(join(nsis, 'Test App_1.0.0_x64-setup.exe.sig'), 'SIG\n');
const lines = [];
const release = () => releaseCommand(['windows', '--skip-build'], { cwd: root, env: { CARGO_TARGET_DIR: join(root, 'target') }, log: line => lines.push(line) });
writeFileSync(join(nsis, 'Test App_1.0.0_x64-setup.exe'), 'MZ... C:\\Users\\zorblat\\.cargo\\registry\\src\\lib.rs ...');
await assert.rejects(release(), /1 forbidden term in the artifacts: nothing was released/);
assert.ok(!existsSync(join(root, 'releases')));
assert.doesNotMatch(lines.join('\n'), /zorblat/);
assert.match(lines.join('\n'), /Test App_1\.0\.0_x64-setup\.exe: 1 hit \(z\*\*\* x1\)/);
writeFileSync(join(nsis, 'Test App_1.0.0_x64-setup.exe'), 'MZ... /cargo/registry/src/lib.rs ...');
await release();
assert.ok(existsSync(join(root, 'releases/native/latest.json')));
});