Remove the privacy-scan command; releases run an optional owner-supplied check instead
waybetter release and the remote builds no longer scan artifacts themselves: WAYBETTER_RELEASE_CHECK can name a command that gets every artifact path before anything is copied or signed (a non-zero exit stops the release). The command, its tests and every mention in the docs, the template and the architecture diagram are gone.
This commit is contained in:
parent
7cd6c99007
commit
612322a45e
13 changed files with 48 additions and 1108 deletions
|
|
@ -25,7 +25,7 @@ encrypted sync and signed updates; and the tooling to scaffold, sign, scan and r
|
||||||
| **Themes** | One JSON document: colours with contrast enforced, fonts per role (Google Fonts downloaded once, after asking), shapes, layout, decoration, icon packs, per-pane and per-account themes, a Theme Builder with a doctor report. |
|
| **Themes** | One JSON document: colours with contrast enforced, fonts per role (Google Fonts downloaded once, after asking), shapes, layout, decoration, icon packs, per-pane and per-account themes, a Theme Builder with a doctor report. |
|
||||||
| **String packs** | Data that re-words the site's own interface: a language it does not offer, a fix, your own wording. Never code. |
|
| **String packs** | Data that re-words the site's own interface: a language it does not offer, a fix, your own wording. Never code. |
|
||||||
| **A signed Store** | One Ed25519-signed catalog for every wayBetter app, `waybetter://install` links, drag-and-drop and Open With, all through one install screen that shows the source, SHA-256, permissions and code first. |
|
| **A signed Store** | One Ed25519-signed catalog for every wayBetter app, `waybetter://install` links, drag-and-drop and Open With, all through one install screen that shows the source, SHA-256, permissions and code first. |
|
||||||
| **Release tooling** | `waybetter store`, `release`, `privacy-scan`, `build-env`, `i18n` and `strings`: sign the catalog, build binaries free of the builder's paths, and refuse to ship a build with a privacy hit. |
|
| **Release tooling** | `waybetter store`, `release`, `build-env`, `i18n` and `strings`: sign the catalog and build binaries free of the builder's paths. |
|
||||||
|
|
||||||
## How it fits together
|
## How it fits together
|
||||||
|
|
||||||
|
|
@ -71,7 +71,7 @@ mode reloads them as you save:
|
||||||
|
|
||||||
| Package | Where | What it is |
|
| Package | Where | What it is |
|
||||||
| --- | --- | --- |
|
| --- | --- | --- |
|
||||||
| [`waybetter`](https://www.npmjs.com/package/waybetter) | `packages/waybetter` | The npm package: the built runtime and shell (plain ESM with `.d.ts`), JSON Schemas for themes, icon packs, catalogs and i18n, the locales, and the `waybetter` command (`dev`, `store`, `release`, `i18n`, `strings`, `privacy-scan`, `build-env`) |
|
| [`waybetter`](https://www.npmjs.com/package/waybetter) | `packages/waybetter` | The npm package: the built runtime and shell (plain ESM with `.d.ts`), JSON Schemas for themes, icon packs, catalogs and i18n, the locales, and the `waybetter` command (`dev`, `store`, `release`, `i18n`, `strings`, `build-env`) |
|
||||||
| `create-waybetter` | `packages/create-waybetter` | `npm create waybetter`: scaffolds a plugin (JS or TypeScript), a theme or a whole app. No dependencies |
|
| `create-waybetter` | `packages/create-waybetter` | `npm create waybetter`: scaffolds a plugin (JS or TypeScript), a theme or a whole app. No dependencies |
|
||||||
| `@waybetter/runtime` | `packages/runtime` | The page runtime (published inside `waybetter`) |
|
| `@waybetter/runtime` | `packages/runtime` | The page runtime (published inside `waybetter`) |
|
||||||
| `@waybetter/shell` | `packages/shell` | Shell models, frame, panel and Store screens (published inside `waybetter`) |
|
| `@waybetter/shell` | `packages/shell` | Shell models, frame, panel and Store screens (published inside `waybetter`) |
|
||||||
|
|
@ -191,8 +191,7 @@ The site's page is not trusted, and neither is a plugin. The rules:
|
||||||
that host commands stay shell-only or tab-only, that site commands are granted only on the
|
that host commands stay shell-only or tab-only, that site commands are granted only on the
|
||||||
site's URLs, and that each command's first statement checks its caller. The scaffold includes
|
site's URLs, and that each command's first statement checks its caller. The scaffold includes
|
||||||
such a test.
|
such a test.
|
||||||
- `waybetter privacy-scan` and `waybetter build-env` keep a build free of the builder's paths and
|
- `waybetter build-env` keeps a build free of the builder's paths.
|
||||||
names; `waybetter release` refuses to release an artifact with a hit.
|
|
||||||
|
|
||||||
## Optional command groups
|
## Optional command groups
|
||||||
|
|
||||||
|
|
@ -222,7 +221,7 @@ crates/waybetter-host/ Rust host crate (features: tauri, net, app)
|
||||||
packages/runtime/ page runtime (published inside `waybetter`)
|
packages/runtime/ page runtime (published inside `waybetter`)
|
||||||
packages/shell/ shell models, shell frame, panel, store screens (published inside `waybetter`)
|
packages/shell/ shell models, shell frame, panel, store screens (published inside `waybetter`)
|
||||||
packages/waybetter/ the npm package: built runtime and shell, JSON Schemas, locales,
|
packages/waybetter/ the npm package: built runtime and shell, JSON Schemas, locales,
|
||||||
and the `waybetter` command (store, release, i18n, strings, privacy-scan,
|
and the `waybetter` command (store, release, i18n, strings,
|
||||||
build-env, dev)
|
build-env, dev)
|
||||||
packages/create-waybetter/ `npm create waybetter`
|
packages/create-waybetter/ `npm create waybetter`
|
||||||
docs/ ARCHITECTURE.md, STORE.md, STRING-PACKS.md
|
docs/ ARCHITECTURE.md, STORE.md, STRING-PACKS.md
|
||||||
|
|
|
||||||
|
|
@ -1,4 +1,4 @@
|
||||||
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 1200 760" width="1200" height="760" role="img" aria-label="wayBetter architecture: one Tauri 2 window with a trusted shell webview and one webview per site tab, a Rust host underneath, your adapter on the side, and the waybetter command for store, release and privacy tooling">
|
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 1200 760" width="1200" height="760" role="img" aria-label="wayBetter architecture: one Tauri 2 window with a trusted shell webview and one webview per site tab, a Rust host underneath, your adapter on the side, and the waybetter command for store, release and build tooling">
|
||||||
<defs>
|
<defs>
|
||||||
<linearGradient id="bg" x1="0" y1="0" x2="0" y2="1">
|
<linearGradient id="bg" x1="0" y1="0" x2="0" y2="1">
|
||||||
<stop offset="0" stop-color="#0b1020"/>
|
<stop offset="0" stop-color="#0b1020"/>
|
||||||
|
|
@ -122,6 +122,6 @@
|
||||||
<text x="860" y="610">waybetter store build|sign|verify</text>
|
<text x="860" y="610">waybetter store build|sign|verify</text>
|
||||||
<text x="860" y="636">waybetter strings check</text>
|
<text x="860" y="636">waybetter strings check</text>
|
||||||
<text x="860" y="662">waybetter release</text>
|
<text x="860" y="662">waybetter release</text>
|
||||||
<text x="860" y="688">waybetter privacy-scan</text>
|
<text x="860" y="688">waybetter build-env</text>
|
||||||
</g>
|
</g>
|
||||||
</svg>
|
</svg>
|
||||||
|
|
|
||||||
|
Before Width: | Height: | Size: 7.4 KiB After Width: | Height: | Size: 7.4 KiB |
|
|
@ -129,7 +129,7 @@ packages/shell/src/ shell core (TypeScript) and the shell frame (React)
|
||||||
packages/waybetter/ the published npm package (runtime + shell as ESM + .d.ts, theme.schema.json,
|
packages/waybetter/ the published npm package (runtime + shell as ESM + .d.ts, theme.schema.json,
|
||||||
store-catalog.schema.json, the `waybetter store build|sign|verify`,
|
store-catalog.schema.json, the `waybetter store build|sign|verify`,
|
||||||
`waybetter release windows|macos`, `waybetter i18n check|extract`,
|
`waybetter release windows|macos`, `waybetter i18n check|extract`,
|
||||||
`waybetter privacy-scan` and `waybetter build-env` commands, `waybetter/build`, locales/ (51 catalogs and translator notes),
|
`waybetter build-env` commands, `waybetter/build`, locales/ (51 catalogs and translator notes),
|
||||||
windows/ (the installer template and its language files))
|
windows/ (the installer template and its language files))
|
||||||
dev-tools/ development harnesses for a running or built app (perf probes, close and
|
dev-tools/ development harnesses for a running or built app (perf probes, close and
|
||||||
stress checks); not published (dev-tools/README.md)
|
stress checks); not published (dev-tools/README.md)
|
||||||
|
|
@ -459,8 +459,8 @@ The npm package adds the app-side tooling:
|
||||||
`<outDir>/upload`, `latest.json` (other platforms of the version kept); macOS: signs a universal
|
`<outDir>/upload`, `latest.json` (other platforms of the version kept); macOS: signs a universal
|
||||||
build's updater bundle and adds both darwin platforms. It never uploads: it prints the steps.
|
build's updater bundle and adds both darwin platforms. It never uploads: it prints the steps.
|
||||||
The key is read only into the build's environment or handed to the Tauri signer by path.
|
The key is read only into the build's environment or handed to the Tauri signer by path.
|
||||||
- `waybetter build-env -- <command>` and `waybetter privacy-scan`: nothing an app publishes may
|
- `waybetter build-env -- <command>`: nothing an app publishes should carry the builder's
|
||||||
name the person who built it. A Rust binary embeds the absolute paths of its sources (the Cargo
|
paths. A Rust binary embeds the absolute paths of its sources (the Cargo
|
||||||
registry, the rustup toolchain, the workspace: about 1,400 strings holding the account name in
|
registry, the rustup toolchain, the workspace: about 1,400 strings holding the account name in
|
||||||
one app), so `build-env` runs a build with those roots remapped to `/home`, `/src`, `/cargo`,
|
one app), so `build-env` runs a build with those roots remapped to `/home`, `/src`, `/cargo`,
|
||||||
`/rustup` and `/target` (`--remap-path-prefix` appended to the flags already set,
|
`/rustup` and `/target` (`--remap-path-prefix` appended to the flags already set,
|
||||||
|
|
@ -468,13 +468,9 @@ The npm package adds the app-side tooling:
|
||||||
prefix map, so there the C and C++ a build script compiles (a CMake library, the cc crate) lose
|
prefix map, so there the C and C++ a build script compiles (a CMake library, the cc crate) lose
|
||||||
the same roots through `/d1trimfile:<root>` in the `CL` variable, and any PDB they link through
|
the same roots through `/d1trimfile:<root>` in the `CL` variable, and any PDB they link through
|
||||||
`/PDBALTPATH` in `LINK`; cl.exe and link.exe read both themselves, whatever runs them:
|
`/PDBALTPATH` in `LINK`; cl.exe and link.exe read both themselves, whatever runs them:
|
||||||
`waybetter build-env -- tauri build`, on every platform. `privacy-scan` is the proof: it searches
|
`waybetter build-env -- tauri build`, on every platform. `waybetter release` builds with
|
||||||
files, binaries (UTF-8, UTF-16), archives (npm tarballs, zip, tar, .deb, DMG; NSIS installers and
|
`build-env` and runs an optional release check on every artifact (`WAYBETTER_RELEASE_CHECK`, a
|
||||||
AppImages through 7-Zip), git-tracked files and, with `--history`, every blob, path, commit
|
command given the artifact paths; a non-zero exit stops the release). The `.pdb`
|
||||||
message and identity of a repository for the terms in a file kept outside every repository
|
|
||||||
(`"forbiddenTerms"` in `waybetter.release.json`, or `WAYBETTER_FORBIDDEN_TERMS`) plus the
|
|
||||||
machine's account name, and prints hits masked (`m***`). `waybetter release` builds with
|
|
||||||
`build-env` and refuses to release an artifact with a hit or one it could not open. The `.pdb`
|
|
||||||
next to a Windows binary still holds every path: never publish it.
|
next to a Windows binary still holds every path: never publish it.
|
||||||
- `waybetter strings check <pack>`: a string pack (data that replaces the wrapped site's own UI
|
- `waybetter strings check <pack>`: a string pack (data that replaces the wrapped site's own UI
|
||||||
strings) against the site's string inventory: only inventory keys, parameter names, plural and
|
strings) against the site's string inventory: only inventory keys, parameter names, plural and
|
||||||
|
|
|
||||||
|
|
@ -1214,8 +1214,7 @@ function readme(o) {
|
||||||
'',
|
'',
|
||||||
'Build with `npm run desktop:build` (or `npm run release`), not a bare `tauri build`: a Rust binary',
|
'Build with `npm run desktop:build` (or `npm run release`), not a bare `tauri build`: a Rust binary',
|
||||||
'embeds the paths it was built from, and those hold your account name. `waybetter build-env` maps',
|
'embeds the paths it was built from, and those hold your account name. `waybetter build-env` maps',
|
||||||
'them to `/home`, `/src`, `/cargo` and `/rustup`; `npm run privacy:scan` and the release check the',
|
'them to `/home`, `/src`, `/cargo` and `/rustup`.',
|
||||||
'result (`waybetter privacy-scan --help`).',
|
|
||||||
'',
|
'',
|
||||||
'```text',
|
'```text',
|
||||||
'```',
|
'```',
|
||||||
|
|
@ -1266,7 +1265,6 @@ export function appFiles(o) {
|
||||||
dev: 'waybetter dev --watch "node scripts/build.mjs --watch"',
|
dev: 'waybetter dev --watch "node scripts/build.mjs --watch"',
|
||||||
'dev:stable': 'waybetter dev --stable --watch "node scripts/build.mjs --watch"',
|
'dev:stable': 'waybetter dev --stable --watch "node scripts/build.mjs --watch"',
|
||||||
'desktop:build': 'waybetter build-env -- tauri build',
|
'desktop:build': 'waybetter build-env -- tauri build',
|
||||||
'privacy:scan': 'waybetter privacy-scan --tracked .',
|
|
||||||
'test:rust': 'cargo test --manifest-path src-tauri/Cargo.toml --no-default-features',
|
'test:rust': 'cargo test --manifest-path src-tauri/Cargo.toml --no-default-features',
|
||||||
release: 'waybetter release windows',
|
release: 'waybetter release windows',
|
||||||
'release:macos': 'waybetter release macos',
|
'release:macos': 'waybetter release macos',
|
||||||
|
|
|
||||||
|
|
@ -120,9 +120,8 @@ const Shell = createShell({ ...site, panel: { id: 'app', title: 'wayBetter', com
|
||||||
- `waybetter store sign [--init] --key <private key>` / `waybetter store verify --pub <key file>`: Ed25519 signature over the exact catalog bytes; any listed public key verifies (rotation). The private key is never printed or copied.
|
- `waybetter store sign [--init] --key <private key>` / `waybetter store verify --pub <key file>`: Ed25519 signature over the exact catalog bytes; any listed public key verifies (rotation). The private key is never printed or copied.
|
||||||
- `waybetter strings check <pack> [--inventory <file>] [--verbose]`: validates a string pack (data that replaces the wrapped site's own UI strings: `waybetter/runtime/stringPacks`, `waybetter/runtime/stringPackHook`, `waybetter/runtime/textNet`) against the site's string inventory and prints its coverage; the inventories otherwise come from `waybetter.strings.json` in the app's root.
|
- `waybetter strings check <pack> [--inventory <file>] [--verbose]`: validates a string pack (data that replaces the wrapped site's own UI strings: `waybetter/runtime/stringPacks`, `waybetter/runtime/stringPackHook`, `waybetter/runtime/textNet`) against the site's string inventory and prints its coverage; the inventories otherwise come from `waybetter.strings.json` in the app's root.
|
||||||
- `waybetter i18n check [--write] [--verbose] [locale…]`: checks an app's catalogs (and the framework's in the same languages) with the `waybetter/i18n/check` rules, bundled plugins' inline English and the generated files; `--write` regenerates `languages.json` and the native localization files (macOS, Linux). `waybetter i18n extract [--missing]`: keys the code looks up that English lacks, and English keys without a literal lookup. Both read `waybetter.i18n.json` in the app's root (`locales`, `sources`, `nativeSources`, `bundledPlugins`, `runtimeKeys`, `supported`, `native`; schema: `i18n-config.schema.json`). `supported` lists the languages in the supported tier; every other catalog gets `tier: "community"` in the generated `languages.json` (`languages.schema.json`), and an app resolves its automatic language choices only to supported ones (`createLanguage({ supported })`).
|
- `waybetter i18n check [--write] [--verbose] [locale…]`: checks an app's catalogs (and the framework's in the same languages) with the `waybetter/i18n/check` rules, bundled plugins' inline English and the generated files; `--write` regenerates `languages.json` and the native localization files (macOS, Linux). `waybetter i18n extract [--missing]`: keys the code looks up that English lacks, and English keys without a literal lookup. Both read `waybetter.i18n.json` in the app's root (`locales`, `sources`, `nativeSources`, `bundledPlugins`, `runtimeKeys`, `supported`, `native`; schema: `i18n-config.schema.json`). `supported` lists the languages in the supported tier; every other catalog gets `tier: "community"` in the generated `languages.json` (`languages.schema.json`), and an app resolves its automatic language choices only to supported ones (`createLanguage({ supported })`).
|
||||||
- `waybetter release windows|macos`: signed native releases and the update manifest (`waybetter.release.json`). The Windows build runs path-free (below) and every artifact must pass the privacy scan before anything is copied or signed.
|
- `waybetter release windows|macos`: signed native releases and the update manifest (`waybetter.release.json`). The Windows build runs path-free (below); `WAYBETTER_RELEASE_CHECK` can name a command run on every artifact before anything is copied or signed.
|
||||||
- `waybetter build-env -- <command>`: runs a build (`tauri build`, `cargo build`) so the binary holds none of the machine's paths: the home folder, the workspace, `CARGO_HOME`, `RUSTUP_HOME` and `CARGO_TARGET_DIR` become `/home`, `/src`, `/cargo`, `/rustup` and `/target` (`--remap-path-prefix` appended to the flags already in `RUSTFLAGS`/`CARGO_ENCODED_RUSTFLAGS`, `/PDBALTPATH` on MSVC; for C and C++ a build script compiles, `-ffile-prefix-map` on gcc and clang and, on MSVC, `/d1trimfile:<root>` in `CL` and `/PDBALTPATH` in `LINK`, which reach the cc crate, CMake and MSBuild alike). Without a command it prints that environment (`--json`).
|
- `waybetter build-env -- <command>`: runs a build (`tauri build`, `cargo build`) so the binary holds none of the machine's paths: the home folder, the workspace, `CARGO_HOME`, `RUSTUP_HOME` and `CARGO_TARGET_DIR` become `/home`, `/src`, `/cargo`, `/rustup` and `/target` (`--remap-path-prefix` appended to the flags already in `RUSTFLAGS`/`CARGO_ENCODED_RUSTFLAGS`, `/PDBALTPATH` on MSVC; for C and C++ a build script compiles, `-ffile-prefix-map` on gcc and clang and, on MSVC, `/d1trimfile:<root>` in `CL` and `/PDBALTPATH` in `LINK`, which reach the cc crate, CMake and MSBuild alike). Without a command it prints that environment (`--json`).
|
||||||
- `waybetter privacy-scan <paths…> [--terms <file>] [--tracked] [--history]`: searches files, binaries (UTF-8 and UTF-16), archives (zip, tar, gzip, zstd, .deb, DMG; NSIS installers, AppImages and xz with 7-Zip or the matching tool), git-tracked files and a repository's whole history for forbidden terms: the lines of a file kept outside every repository (`--terms`, `WAYBETTER_FORBIDDEN_TERMS`, or `"forbiddenTerms"` in `waybetter.release.json`) plus this machine's account name. A name is a hit as a whole word in any case; a name of three characters or fewer only in a path or as the capitalized word. Output shows file and line or offset with the term masked (`m***`); exit 1 on a hit, 2 when a container could not be opened. Reviewed false positives: `"privacyAllowlist"`, lines of `<file glob>: <regex over the context, the term written <term>>`.
|
|
||||||
- `windows/languages/<Language>.nsh`: the Windows installer's messages for the languages Tauri does not ship, next to the installer template (`windows/installer.nsi`).
|
- `windows/languages/<Language>.nsh`: the Windows installer's messages for the languages Tauri does not ship, next to the installer template (`windows/installer.nsi`).
|
||||||
|
|
||||||
## Plugin UI kit (`runtime/ui`)
|
## Plugin UI kit (`runtime/ui`)
|
||||||
|
|
|
||||||
|
|
@ -1,7 +1,7 @@
|
||||||
#!/usr/bin/env node
|
#!/usr/bin/env node
|
||||||
// The `waybetter` command: `waybetter store build|sign|verify` (cli/store.mjs),
|
// The `waybetter` command: `waybetter store build|sign|verify` (cli/store.mjs),
|
||||||
// `waybetter release windows|macos|linux` (cli/release.mjs), `waybetter manifests` (cli/manifests.mjs), `waybetter i18n check|extract`
|
// `waybetter release windows|macos|linux` (cli/release.mjs), `waybetter manifests` (cli/manifests.mjs), `waybetter i18n check|extract`
|
||||||
// (cli/i18n.mjs), `waybetter privacy-scan` (cli/privacy.mjs) and `waybetter build-env`
|
// (cli/i18n.mjs), `waybetter build-env`
|
||||||
// (cli/buildenv.mjs), `waybetter dev` (cli/dev.mjs: watchers + tauri dev), `waybetter strings check`
|
// (cli/buildenv.mjs), `waybetter dev` (cli/dev.mjs: watchers + tauri dev), `waybetter strings check`
|
||||||
// (cli/strings.mjs).
|
// (cli/strings.mjs).
|
||||||
import { readFileSync } from 'node:fs';
|
import { readFileSync } from 'node:fs';
|
||||||
|
|
@ -9,7 +9,6 @@ import { BUILD_ENV_USAGE, buildEnvCommand } from '../cli/buildenv.mjs';
|
||||||
import { DEV_USAGE, devCommand } from '../cli/dev.mjs';
|
import { DEV_USAGE, devCommand } from '../cli/dev.mjs';
|
||||||
import { MANIFESTS_USAGE, manifestsCommand } from '../cli/manifests.mjs';
|
import { MANIFESTS_USAGE, manifestsCommand } from '../cli/manifests.mjs';
|
||||||
import { I18N_USAGE, i18nCommand } from '../cli/i18n.mjs';
|
import { I18N_USAGE, i18nCommand } from '../cli/i18n.mjs';
|
||||||
import { PRIVACY_USAGE, privacyCommand } from '../cli/privacy.mjs';
|
|
||||||
import { STORE_USAGE, storeCommand } from '../cli/store.mjs';
|
import { STORE_USAGE, storeCommand } from '../cli/store.mjs';
|
||||||
import { STRINGS_USAGE, stringsCommand } from '../cli/strings.mjs';
|
import { STRINGS_USAGE, stringsCommand } from '../cli/strings.mjs';
|
||||||
import { RELEASE_USAGE, releaseCommand } from '../cli/release.mjs';
|
import { RELEASE_USAGE, releaseCommand } from '../cli/release.mjs';
|
||||||
|
|
@ -26,8 +25,6 @@ try {
|
||||||
process.exitCode = await i18nCommand(rest);
|
process.exitCode = await i18nCommand(rest);
|
||||||
} else if (area === 'strings') {
|
} else if (area === 'strings') {
|
||||||
process.exitCode = await stringsCommand(rest);
|
process.exitCode = await stringsCommand(rest);
|
||||||
} else if (area === 'privacy-scan') {
|
|
||||||
process.exitCode = privacyCommand(rest);
|
|
||||||
} else if (area === 'build-env') {
|
} else if (area === 'build-env') {
|
||||||
process.exitCode = buildEnvCommand(rest);
|
process.exitCode = buildEnvCommand(rest);
|
||||||
} else if (area === 'dev') {
|
} else if (area === 'dev') {
|
||||||
|
|
@ -35,7 +32,7 @@ try {
|
||||||
} else if (area === '--version' || area === '-v') {
|
} else if (area === '--version' || area === '-v') {
|
||||||
console.log(JSON.parse(readFileSync(new URL('../package.json', import.meta.url), 'utf8')).version);
|
console.log(JSON.parse(readFileSync(new URL('../package.json', import.meta.url), 'utf8')).version);
|
||||||
} else {
|
} else {
|
||||||
console.log(`${STORE_USAGE}\n\n${RELEASE_USAGE}\n\n${MANIFESTS_USAGE}\n\n${I18N_USAGE}\n\n${STRINGS_USAGE}\n\n${PRIVACY_USAGE}\n\n${BUILD_ENV_USAGE}
|
console.log(`${STORE_USAGE}\n\n${RELEASE_USAGE}\n\n${MANIFESTS_USAGE}\n\n${I18N_USAGE}\n\n${STRINGS_USAGE}\n\n${BUILD_ENV_USAGE}
|
||||||
|
|
||||||
${DEV_USAGE}`);
|
${DEV_USAGE}`);
|
||||||
if (area && area !== '--help' && area !== '-h') process.exitCode = 1;
|
if (area && area !== '--help' && area !== '-h') process.exitCode = 1;
|
||||||
|
|
|
||||||
|
|
@ -18,7 +18,7 @@
|
||||||
// waybetter build-env -- tauri build run a command with that environment
|
// waybetter build-env -- tauri build run a command with that environment
|
||||||
// waybetter build-env [--json] show it
|
// waybetter build-env [--json] show it
|
||||||
//
|
//
|
||||||
// `waybetter release windows` builds with it. It is not a proof: `waybetter privacy-scan` is.
|
// `waybetter release windows` builds with it.
|
||||||
import { spawnSync } from 'node:child_process';
|
import { spawnSync } from 'node:child_process';
|
||||||
import { existsSync } from 'node:fs';
|
import { existsSync } from 'node:fs';
|
||||||
import { homedir } from 'node:os';
|
import { homedir } from 'node:os';
|
||||||
|
|
|
||||||
|
|
@ -1,768 +0,0 @@
|
||||||
// `waybetter privacy-scan`: the release gate that keeps personal names and account names out of
|
|
||||||
// everything an app publishes. It reads a list of forbidden terms (one per line, kept OUTSIDE every
|
|
||||||
// repository), always adds this machine's account name and home folder name, and searches files,
|
|
||||||
// folders, archives, git-tracked files and (with --history) a whole repository for them.
|
|
||||||
//
|
|
||||||
// - Text and binary alike: every file is searched as bytes, in UTF-8, UTF-16LE and UTF-16BE.
|
|
||||||
// - Containers are opened: zip (and .nupkg/.jar), tar, gzip, zstd, ar/.deb, and with an external
|
|
||||||
// tool xz/bzip2 (xz, bzip2 or 7-Zip), NSIS installers, AppImage/squashfs (7-Zip or unsquashfs)
|
|
||||||
// and DMG (zlib images natively, others with 7-Zip). A container that could not be opened is an
|
|
||||||
// error, not a pass.
|
|
||||||
// - A term is never printed. A hit shows the file, line:column or byte offset, and the term as its
|
|
||||||
// first letter plus `***`; the context shown has every term masked the same way.
|
|
||||||
//
|
|
||||||
// Matching: a term is a hit as a whole word, case-insensitively (letters with diacritics count as
|
|
||||||
// their base letter). A term of three characters or fewer is usually also an ordinary word fragment
|
|
||||||
// (`max(`, `max-width`), so it is a hit only in a path (`Users\<it>`, `/home/<it>`, `\<it>\`,
|
|
||||||
// `/<it>/`, `Users-<it>-`) or as the capitalized standalone word.
|
|
||||||
//
|
|
||||||
// Reviewed false positives go in an allowlist file inside the repository: lines of
|
|
||||||
// `<file glob>: <regular expression>`, where the expression is tested against the hit's context with
|
|
||||||
// the term replaced by `<term>` (another hit nearby reads `<other>`). The file never contains a term (the scan refuses one that does).
|
|
||||||
import { spawnSync } from 'node:child_process';
|
|
||||||
import { closeSync, existsSync, lstatSync, mkdtempSync, openSync, readFileSync, readSync, readdirSync, rmSync, writeFileSync } from 'node:fs';
|
|
||||||
import { homedir, tmpdir, userInfo } from 'node:os';
|
|
||||||
import { basename, dirname, join, relative, resolve } from 'node:path';
|
|
||||||
import * as zlib from 'node:zlib';
|
|
||||||
|
|
||||||
export const TERMS_ENV = 'WAYBETTER_FORBIDDEN_TERMS';
|
|
||||||
export const SEVEN_ZIP_ENV = 'WAYBETTER_7Z';
|
|
||||||
const CONFIG_FILE = 'waybetter.release.json';
|
|
||||||
|
|
||||||
export const PRIVACY_USAGE = `Usage:
|
|
||||||
waybetter privacy-scan <paths...> [--terms <file>] [--allow <file>] [--tracked] [--untracked]
|
|
||||||
[--history] [--all] [--json] [--if-terms] [--no-auto]
|
|
||||||
Searches files, folders and archives for forbidden terms (personal names, account names) and
|
|
||||||
exits 1 on a hit, 2 when something could not be opened. Terms: --terms, else the file named by
|
|
||||||
${TERMS_ENV}, else "forbiddenTerms" in ${CONFIG_FILE}; this machine's account name and
|
|
||||||
home folder name are always added (--no-auto: not). A term is never printed.
|
|
||||||
--tracked each path is inside a git work tree: only the files git tracks (and their names)
|
|
||||||
--untracked with --tracked: also files git neither tracks nor ignores
|
|
||||||
--history each path is a repository: every blob, path, commit message, tag and identity
|
|
||||||
--allow reviewed false positives: lines of "<file glob>: <regex over the context, the
|
|
||||||
term written <term>>" (default: "privacyAllowlist" in ${CONFIG_FILE})
|
|
||||||
--all list every hit (default: the first 5 of each file)
|
|
||||||
--if-terms do nothing, quietly, when no terms file exists (for a contributor's test run)`;
|
|
||||||
|
|
||||||
class PrivacyError extends Error {}
|
|
||||||
const fail = message => { throw new PrivacyError(message); };
|
|
||||||
|
|
||||||
// ---------------------------------------------------------------------------------------------
|
|
||||||
// Terms
|
|
||||||
|
|
||||||
/** Account names that say nothing about a person: never added automatically. */
|
|
||||||
const GENERIC_ACCOUNTS = new Set(['root', 'user', 'users', 'admin', 'administrator', 'runner', 'runneradmin', 'ubuntu', 'build',
|
|
||||||
'builder', 'ci', 'docker', 'node', 'vagrant', 'default', 'nobody', 'home', 'dev', 'developer', 'owner', 'guest', 'test',
|
|
||||||
'app', 'work', 'vscode', 'codespace', 'gitpod', 'circleci', 'jenkins', 'buildkite', 'ec2-user', 'azureuser', 'pi', 'public']);
|
|
||||||
|
|
||||||
/** The lines of a terms file: one term each, `#` comments and blank lines skipped. */
|
|
||||||
export function parseTerms(text) {
|
|
||||||
return text.split(/\r?\n/).map(line => line.trim()).filter(line => line && !line.startsWith('#'));
|
|
||||||
}
|
|
||||||
|
|
||||||
const expandHome = value => value.replace(/^~(?=$|[\\/])/, homedir());
|
|
||||||
|
|
||||||
/** The nearest waybetter.release.json's privacy settings (paths made absolute), or {}. */
|
|
||||||
export function privacyConfig(cwd) {
|
|
||||||
for (let dir = resolve(cwd); ; dir = dirname(dir)) {
|
|
||||||
const file = join(dir, CONFIG_FILE);
|
|
||||||
if (existsSync(file)) {
|
|
||||||
let raw;
|
|
||||||
try { raw = JSON.parse(readFileSync(file, 'utf8')); } catch { return {}; }
|
|
||||||
const at = value => (typeof value === 'string' && value.trim() ? resolve(dir, expandHome(value.trim())) : undefined);
|
|
||||||
return { root: dir, forbiddenTerms: at(raw.forbiddenTerms), privacyAllowlist: at(raw.privacyAllowlist) };
|
|
||||||
}
|
|
||||||
if (dirname(dir) === dir) return {};
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* The terms to search for: the file's lines plus (unless `auto` is false) this machine's account
|
|
||||||
* name and home folder name. `file` may be missing (then only the automatic ones).
|
|
||||||
* Returns `{ terms: [{ text, rule }], fromFile }`; rule is "word" or "account" (see the header).
|
|
||||||
*/
|
|
||||||
export function collectTerms({ file, auto = true, extra = [] } = {}) {
|
|
||||||
const seen = new Map();
|
|
||||||
const add = (text, automatic) => {
|
|
||||||
const value = text.trim();
|
|
||||||
const key = value.toLowerCase();
|
|
||||||
if (value.length < 2 || seen.has(key)) return;
|
|
||||||
if (automatic && GENERIC_ACCOUNTS.has(key)) return;
|
|
||||||
seen.set(key, { text: value, rule: [...value].length <= 3 ? 'account' : 'word' });
|
|
||||||
};
|
|
||||||
let fromFile = 0;
|
|
||||||
if (file) {
|
|
||||||
for (const term of parseTerms(readFileSync(file, 'utf8'))) add(term, false);
|
|
||||||
fromFile = seen.size;
|
|
||||||
}
|
|
||||||
for (const term of extra) add(term, false);
|
|
||||||
if (auto) {
|
|
||||||
let user = '';
|
|
||||||
try { user = userInfo().username; } catch { /* no passwd entry */ }
|
|
||||||
add(user, true);
|
|
||||||
add(basename(homedir()), true);
|
|
||||||
}
|
|
||||||
return { terms: [...seen.values()], fromFile };
|
|
||||||
}
|
|
||||||
|
|
||||||
// ---------------------------------------------------------------------------------------------
|
|
||||||
// Matching (on bytes, read as a latin1 string so one regular expression sees every encoding)
|
|
||||||
|
|
||||||
const ENCODINGS = ['utf8', 'utf16le', 'utf16be'];
|
|
||||||
const FOLDS = { a: 'àáâãäåā', c: 'çćč', e: 'èéêëēęě', g: 'ğ', i: 'ìíîïıī', l: 'ł', n: 'ñńň', o: 'òóôõöøō', r: 'ř', s: 'şšśș', t: 'țť', u: 'ùúûüūů', y: 'ýÿ', z: 'žźż' };
|
|
||||||
|
|
||||||
function bytesOf(text, encoding) {
|
|
||||||
if (encoding === 'utf8') return Buffer.from(text, 'utf8');
|
|
||||||
const bytes = Buffer.from(text, 'utf16le');
|
|
||||||
return encoding === 'utf16be' ? bytes.swap16() : bytes;
|
|
||||||
}
|
|
||||||
const hex = bytes => [...bytes].map(byte => `\\x${byte.toString(16).padStart(2, '0')}`).join('');
|
|
||||||
|
|
||||||
/** One character in one encoding; `fold`: either case, and the letter's diacritic forms. */
|
|
||||||
function unit(char, encoding, fold) {
|
|
||||||
const forms = new Set([char]);
|
|
||||||
if (fold) {
|
|
||||||
const lower = char.toLowerCase();
|
|
||||||
for (const form of [lower, char.toUpperCase(), ...(FOLDS[lower] ?? '')]) {
|
|
||||||
forms.add(form);
|
|
||||||
forms.add(form.toUpperCase());
|
|
||||||
}
|
|
||||||
if (lower === 'i') forms.add('İ');
|
|
||||||
}
|
|
||||||
const encoded = [...new Set([...forms].filter(form => [...form].length === 1).map(form => hex(bytesOf(form, encoding))))];
|
|
||||||
return encoded.length === 1 ? encoded[0] : `(?:${encoded.join('|')})`;
|
|
||||||
}
|
|
||||||
const literal = (text, encoding, fold) => [...text].map(char => unit(char, encoding, fold)).join('');
|
|
||||||
const oneOf = (texts, encoding, fold) => `(?:${texts.map(text => literal(text, encoding, fold)).join('|')})`;
|
|
||||||
|
|
||||||
const ALNUM = 'A-Za-z0-9';
|
|
||||||
const BEFORE = { utf8: `(?<![${ALNUM}])`, utf16le: `(?<![${ALNUM}]\\x00)`, utf16be: `(?<!\\x00[${ALNUM}])` };
|
|
||||||
const AFTER = { utf8: `(?![${ALNUM}])`, utf16le: `(?![${ALNUM}]\\x00)`, utf16be: `(?!\\x00[${ALNUM}])` };
|
|
||||||
const SEPARATORS = ['\\', '/', '%5C', '%2F'];
|
|
||||||
|
|
||||||
/**
|
|
||||||
* The alternatives (regular expression sources) that make `term` a hit. `firm` leaves out the one
|
|
||||||
* that three random bytes satisfy (the capitalized word in UTF-8), see `stringLike`.
|
|
||||||
*/
|
|
||||||
function alternatives(term, firm = false) {
|
|
||||||
const out = [];
|
|
||||||
for (const encoding of ENCODINGS) {
|
|
||||||
const word = literal(term.text, encoding, true);
|
|
||||||
if (term.rule === 'word') {
|
|
||||||
out.push(`${BEFORE[encoding]}${word}${AFTER[encoding]}`);
|
|
||||||
continue;
|
|
||||||
}
|
|
||||||
const sep = oneOf(SEPARATORS, encoding, true);
|
|
||||||
// In a path: between separators, right after a Users/home folder, or in a dash-flattened path.
|
|
||||||
out.push(`(?<=${sep})${word}(?=${sep})`);
|
|
||||||
out.push(`(?<=${oneOf(['Users', 'home'], encoding, true)}${sep}+)${word}${AFTER[encoding]}`);
|
|
||||||
out.push(`(?<=${literal('Users-', encoding, true)})${word}(?=${literal('-', encoding, false)})`);
|
|
||||||
// The capitalized standalone word.
|
|
||||||
const [first, ...rest] = [...term.text];
|
|
||||||
const capitalized = literal(first.toUpperCase(), encoding, false) + literal(rest.join('').toLowerCase(), encoding, false);
|
|
||||||
if (!firm || encoding !== 'utf8') out.push(`${BEFORE[encoding]}${capitalized}${AFTER[encoding]}`);
|
|
||||||
}
|
|
||||||
return out;
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Whether the bytes at [start, end) sit in text rather than in compressed or random data: part of a
|
|
||||||
* run of 12 or more printable characters, or a field between two NUL bytes. Megabytes of media hold
|
|
||||||
* every three-letter sequence by chance; they do not hold it inside a sentence.
|
|
||||||
*/
|
|
||||||
function stringLike(text, start, end) {
|
|
||||||
const printable = code => (code >= 0x20 && code <= 0x7e) || code === 9 || code === 10 || code === 13;
|
|
||||||
let run = end - start;
|
|
||||||
for (let at = start - 1; at >= 0 && run < 12 && printable(text.charCodeAt(at)); at -= 1) run += 1;
|
|
||||||
for (let at = end; at < text.length && run < 12 && printable(text.charCodeAt(at)); at += 1) run += 1;
|
|
||||||
if (run >= 12) return true;
|
|
||||||
return (start === 0 || text.charCodeAt(start - 1) === 0) && (end === text.length || text.charCodeAt(end) === 0);
|
|
||||||
}
|
|
||||||
|
|
||||||
/** First letter + `***`: the only form of a term that is ever printed. */
|
|
||||||
export const maskTerm = text => `${[...text][0]}***`;
|
|
||||||
|
|
||||||
/** A matcher over bytes for a term list. */
|
|
||||||
export function createMatcher(terms) {
|
|
||||||
const each = terms.map(term => ({ term, mask: maskTerm(term.text), source: alternatives(term).join('|') }));
|
|
||||||
const all = each.length ? new RegExp(each.map(entry => `(?:${entry.source})`).join('|'), 'g') : undefined;
|
|
||||||
const single = each.map(entry => new RegExp(entry.source, 'g'));
|
|
||||||
const firm = each.map(entry => (entry.term.rule === 'account' ? new RegExp(alternatives(entry.term, true).join('|'), 'g') : undefined));
|
|
||||||
const startsAt = (expression, window, at) => {
|
|
||||||
expression.lastIndex = at;
|
|
||||||
return expression.exec(window)?.index === at;
|
|
||||||
};
|
|
||||||
|
|
||||||
/** Every hit in `buffer`: `{ offset, length, mask, encoding }`, in order. `binary`: not a text file. */
|
|
||||||
function find(buffer, binary = false) {
|
|
||||||
if (!all || !buffer.length) return [];
|
|
||||||
const text = buffer.latin1Slice(0, buffer.length);
|
|
||||||
const hits = [];
|
|
||||||
all.lastIndex = 0;
|
|
||||||
let end = -1;
|
|
||||||
for (let match = all.exec(text); match; match = all.exec(text)) {
|
|
||||||
if (match.index >= end) {
|
|
||||||
end = match.index + match[0].length;
|
|
||||||
// Which term: the one whose own expression matches at this offset (seen with its context).
|
|
||||||
const from = Math.max(0, match.index - 64);
|
|
||||||
const window = text.slice(from, end + 8);
|
|
||||||
const which = single.findIndex(expression => startsAt(expression, window, match.index - from));
|
|
||||||
const mask = each[which]?.mask ?? '?***';
|
|
||||||
if (binary && firm[which] && !startsAt(firm[which], window, match.index - from) && !stringLike(text, match.index, end)) continue;
|
|
||||||
// (UTF-16 text matches in one byte order or, a byte earlier, in the other: the label names neither.)
|
|
||||||
hits.push({ offset: match.index, length: match[0].length, mask, encoding: match[0].includes('\x00') ? 'utf16' : 'utf8' });
|
|
||||||
}
|
|
||||||
if (match[0].length === 0) all.lastIndex += 1;
|
|
||||||
}
|
|
||||||
return hits;
|
|
||||||
}
|
|
||||||
|
|
||||||
/** `text` (a latin1 view of bytes) with every hit replaced by `mark` (default: the term's mask). */
|
|
||||||
function redact(text, mark) {
|
|
||||||
let out = text;
|
|
||||||
for (let i = 0; i < single.length; i += 1) {
|
|
||||||
single[i].lastIndex = 0;
|
|
||||||
out = out.replace(single[i], mark ?? each[i].mask);
|
|
||||||
}
|
|
||||||
return out;
|
|
||||||
}
|
|
||||||
|
|
||||||
return { find, redact, size: each.length };
|
|
||||||
}
|
|
||||||
|
|
||||||
// ---------------------------------------------------------------------------------------------
|
|
||||||
// Allowlist
|
|
||||||
|
|
||||||
const globToRegExp = glob => new RegExp(`^${glob.split('**').map(part => part.split('*').map(piece =>
|
|
||||||
piece.replace(/[.+^${}()|[\]\\]/g, '\\$&').replaceAll('?', '[^/]')).join('[^/]*')).join('.*')}$`);
|
|
||||||
|
|
||||||
/** Reads an allowlist file: `<file glob>: <regex>` lines. It must not contain a term itself. */
|
|
||||||
export function loadAllowlist(file, matcher) {
|
|
||||||
if (!file || !existsSync(file)) return [];
|
|
||||||
const raw = readFileSync(file);
|
|
||||||
if (matcher.find(raw).length) fail(`${file}: the allowlist must not contain a term (write <term> in its place)`);
|
|
||||||
const rules = [];
|
|
||||||
raw.toString('utf8').split(/\r?\n/).forEach((line, index) => {
|
|
||||||
const text = line.trim();
|
|
||||||
if (!text || text.startsWith('#')) return;
|
|
||||||
const cut = text.indexOf(': ');
|
|
||||||
if (cut <= 0) fail(`${file}:${index + 1}: expected "<file glob>: <regex>"`);
|
|
||||||
let pattern;
|
|
||||||
try { pattern = new RegExp(text.slice(cut + 2).trim()); } catch (error) { fail(`${file}:${index + 1}: ${error.message}`); }
|
|
||||||
rules.push({ glob: globToRegExp(text.slice(0, cut).trim()), pattern, line: index + 1, used: 0 });
|
|
||||||
});
|
|
||||||
return rules;
|
|
||||||
}
|
|
||||||
|
|
||||||
// ---------------------------------------------------------------------------------------------
|
|
||||||
// Containers
|
|
||||||
|
|
||||||
const startsWith = (buffer, bytes, at = 0) => buffer.length >= at + bytes.length && bytes.every((byte, i) => buffer[at + i] === byte);
|
|
||||||
const ascii = (buffer, from, length) => buffer.latin1Slice(Math.min(from, buffer.length), Math.min(from + length, buffer.length)).replace(/\0.*$/s, '');
|
|
||||||
|
|
||||||
function* tarEntries(buffer) {
|
|
||||||
let at = 0;
|
|
||||||
let longName;
|
|
||||||
while (at + 512 <= buffer.length) {
|
|
||||||
const header = buffer.subarray(at, at + 512);
|
|
||||||
if (header.every(byte => byte === 0)) break;
|
|
||||||
const sizeField = header.subarray(124, 136);
|
|
||||||
const size = sizeField[0] & 0x80
|
|
||||||
? Number(sizeField.subarray(4).readBigUInt64BE(0))
|
|
||||||
: parseInt(ascii(header, 124, 12).trim() || '0', 8) || 0;
|
|
||||||
const type = String.fromCharCode(header[156] || 48);
|
|
||||||
const data = buffer.subarray(at + 512, at + 512 + size);
|
|
||||||
at += 512 + Math.ceil(size / 512) * 512;
|
|
||||||
if (type === 'L') { longName = data.toString('utf8').replace(/\0+$/, ''); continue; }
|
|
||||||
if (type === 'x' || type === 'g') {
|
|
||||||
const path = /\d+ path=([^\n]*)\n/.exec(data.toString('utf8'));
|
|
||||||
if (type === 'x' && path) longName = path[1];
|
|
||||||
yield { name: `${longName ?? 'pax'} (pax header)`, data };
|
|
||||||
continue;
|
|
||||||
}
|
|
||||||
const prefix = ascii(header, 345, 155);
|
|
||||||
const name = longName ?? (prefix ? `${prefix}/${ascii(header, 0, 100)}` : ascii(header, 0, 100));
|
|
||||||
longName = undefined;
|
|
||||||
// The header also carries a link target and the owner's user and group names.
|
|
||||||
yield { name, data: type === '0' || type === '7' ? data : undefined, meta: [ascii(header, 157, 100), ascii(header, 265, 32), ascii(header, 297, 32)] };
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
function* zipEntries(buffer, note) {
|
|
||||||
const end = buffer.lastIndexOf(Buffer.from([0x50, 0x4b, 0x05, 0x06]));
|
|
||||||
if (end < 0) { note('zip: no central directory'); return; }
|
|
||||||
let count = buffer.readUInt16LE(end + 10);
|
|
||||||
let at = buffer.readUInt32LE(end + 16);
|
|
||||||
if (at === 0xffffffff || count === 0xffff) {
|
|
||||||
const locator = end - 20;
|
|
||||||
if (locator < 0 || !startsWith(buffer, [0x50, 0x4b, 0x06, 0x07], locator)) { note('zip64: no locator'); return; }
|
|
||||||
const record = Number(buffer.readBigUInt64LE(locator + 8));
|
|
||||||
count = Number(buffer.readBigUInt64LE(record + 32));
|
|
||||||
at = Number(buffer.readBigUInt64LE(record + 48));
|
|
||||||
}
|
|
||||||
yield { name: '(zip directory)', data: buffer.subarray(at, end + 22 + buffer.readUInt16LE(end + 20)), leaf: true };
|
|
||||||
for (let i = 0; i < count; i += 1) {
|
|
||||||
if (!startsWith(buffer, [0x50, 0x4b, 0x01, 0x02], at)) { note('zip: broken central directory'); return; }
|
|
||||||
const method = buffer.readUInt16LE(at + 10);
|
|
||||||
let size = buffer.readUInt32LE(at + 20);
|
|
||||||
let full = buffer.readUInt32LE(at + 24);
|
|
||||||
const nameLength = buffer.readUInt16LE(at + 28);
|
|
||||||
const extraLength = buffer.readUInt16LE(at + 30);
|
|
||||||
const commentLength = buffer.readUInt16LE(at + 32);
|
|
||||||
let local = buffer.readUInt32LE(at + 42);
|
|
||||||
const name = buffer.toString('utf8', at + 46, at + 46 + nameLength);
|
|
||||||
for (let extra = at + 46 + nameLength; extra + 4 <= at + 46 + nameLength + extraLength;) {
|
|
||||||
const id = buffer.readUInt16LE(extra);
|
|
||||||
const length = buffer.readUInt16LE(extra + 2);
|
|
||||||
if (id === 1) {
|
|
||||||
let field = extra + 4;
|
|
||||||
if (full === 0xffffffff) { full = Number(buffer.readBigUInt64LE(field)); field += 8; }
|
|
||||||
if (size === 0xffffffff) { size = Number(buffer.readBigUInt64LE(field)); field += 8; }
|
|
||||||
if (local === 0xffffffff) local = Number(buffer.readBigUInt64LE(field));
|
|
||||||
}
|
|
||||||
extra += 4 + length;
|
|
||||||
}
|
|
||||||
at += 46 + nameLength + extraLength + commentLength;
|
|
||||||
if (name.endsWith('/')) continue;
|
|
||||||
const start = local + 30 + buffer.readUInt16LE(local + 26) + buffer.readUInt16LE(local + 28);
|
|
||||||
const packed = buffer.subarray(start, start + size);
|
|
||||||
if (method === 0) yield { name, data: packed };
|
|
||||||
else if (method === 8) {
|
|
||||||
try { yield { name, data: zlib.inflateRawSync(packed) }; } catch (error) { note(`zip: ${name}: ${error.message}`); }
|
|
||||||
} else note(`zip: ${name}: compression method ${method}`);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
function* arEntries(buffer) {
|
|
||||||
let at = 8;
|
|
||||||
while (at + 60 <= buffer.length) {
|
|
||||||
const name = ascii(buffer, at, 16).trim().replace(/\/$/, '');
|
|
||||||
const size = parseInt(ascii(buffer, at + 48, 10).trim(), 10) || 0;
|
|
||||||
yield { name: `${name} (ar header)`, data: buffer.subarray(at, at + 60), leaf: true };
|
|
||||||
yield { name, data: buffer.subarray(at + 60, at + 60 + size) };
|
|
||||||
at += 60 + size + (size % 2);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/** The decompressed partitions of a DMG whose chunks are raw or zlib; undefined when it has others. */
|
|
||||||
function dmgPartitions(buffer) {
|
|
||||||
const trailer = buffer.subarray(buffer.length - 512);
|
|
||||||
const dataFork = Number(trailer.readBigUInt64BE(24));
|
|
||||||
const xmlAt = Number(trailer.readBigUInt64BE(216));
|
|
||||||
const xml = buffer.toString('latin1', xmlAt, xmlAt + Number(trailer.readBigUInt64BE(224)));
|
|
||||||
const blocks = /<key>blkx<\/key>([\s\S]*?)<\/array>/.exec(xml)?.[1] ?? '';
|
|
||||||
const partitions = [];
|
|
||||||
for (const [, encoded] of blocks.matchAll(/<data>([\s\S]*?)<\/data>/g)) {
|
|
||||||
const table = Buffer.from(encoded.replace(/\s+/g, ''), 'base64');
|
|
||||||
if (ascii(table, 0, 4) !== 'mish') continue;
|
|
||||||
const parts = [];
|
|
||||||
for (let i = 0; i < table.readUInt32BE(200); i += 1) {
|
|
||||||
const entry = 204 + i * 40;
|
|
||||||
const type = table.readUInt32BE(entry);
|
|
||||||
const from = dataFork + Number(table.readBigUInt64BE(entry + 24));
|
|
||||||
const packed = buffer.subarray(from, from + Number(table.readBigUInt64BE(entry + 32)));
|
|
||||||
if (type === 1) parts.push(packed);
|
|
||||||
else if (type === 0x80000005) parts.push(zlib.inflateSync(packed));
|
|
||||||
else if (type === 0x80000006 || type === 0x80000007 || type === 0x80000008) return undefined;
|
|
||||||
}
|
|
||||||
partitions.push(Buffer.concat(parts));
|
|
||||||
}
|
|
||||||
return partitions;
|
|
||||||
}
|
|
||||||
|
|
||||||
/** The offset where an ELF file ends (an AppImage's squashfs payload starts there). */
|
|
||||||
function elfEnd(buffer) {
|
|
||||||
if (buffer[4] !== 2 || buffer[5] !== 1) return undefined;
|
|
||||||
return Number(buffer.readBigUInt64LE(0x28)) + buffer.readUInt16LE(0x3a) * buffer.readUInt16LE(0x3c);
|
|
||||||
}
|
|
||||||
|
|
||||||
/** Whether bytes start like a container this scan would open. */
|
|
||||||
const isContainer = head => startsWith(head, [0x1f, 0x8b]) || startsWith(head, [0x50, 0x4b, 0x03, 0x04]) || ascii(head, 257, 5) === 'ustar'
|
|
||||||
|| ascii(head, 0, 8) === '!<arch>\n' || startsWith(head, [0x28, 0xb5, 0x2f, 0xfd]) || startsWith(head, [0xfd, 0x37, 0x7a, 0x58, 0x5a, 0x00])
|
|
||||||
|| ascii(head, 0, 4) === 'hsqs' || startsWith(head, [0x4d, 0x5a]) || startsWith(head, [0x7f, 0x45, 0x4c, 0x46]);
|
|
||||||
|
|
||||||
let sevenZipPath;
|
|
||||||
function sevenZip(env) {
|
|
||||||
if (sevenZipPath !== undefined) return sevenZipPath;
|
|
||||||
const candidates = [env[SEVEN_ZIP_ENV], '7z', '7zz', '7za'];
|
|
||||||
if (process.platform === 'win32') {
|
|
||||||
for (const base of [env.ProgramFiles, env['ProgramFiles(x86)'], env.ProgramW6432]) if (base) candidates.push(join(base, '7-Zip', '7z.exe'));
|
|
||||||
}
|
|
||||||
sevenZipPath = candidates.filter(Boolean).find(file => spawnSync(file, ['i'], { stdio: 'ignore', windowsHide: true }).status === 0) ?? '';
|
|
||||||
return sevenZipPath;
|
|
||||||
}
|
|
||||||
|
|
||||||
/** Runs a filter (`xz -dc`, `bzip2 -dc`) over bytes; undefined when the tool is missing or fails. */
|
|
||||||
function pipe(tool, args, input) {
|
|
||||||
const result = spawnSync(tool, args, { input, maxBuffer: 2 ** 31 - 1, windowsHide: true });
|
|
||||||
return result.status === 0 ? result.stdout : undefined;
|
|
||||||
}
|
|
||||||
|
|
||||||
// ---------------------------------------------------------------------------------------------
|
|
||||||
// The scan
|
|
||||||
|
|
||||||
const CHUNK = 64 * 1024 * 1024;
|
|
||||||
const OVERLAP = 4096;
|
|
||||||
const utf8 = latin1 => Buffer.from(latin1, 'latin1').toString('utf8');
|
|
||||||
const printable = text => text.replace(/\0/g, '').replace(/[\x01-\x1f\x7f-\x9f]+/g, ' ').trim();
|
|
||||||
|
|
||||||
/**
|
|
||||||
* A scan over any number of inputs. `report()` gives
|
|
||||||
* `{ hits, files: [{ file, count, masks, hits: [{ where, mask, encoding, context }] }], unopened, scanned, bytes }`.
|
|
||||||
*/
|
|
||||||
export function createScan({ terms, allow = [], env = process.env } = {}) {
|
|
||||||
const matcher = createMatcher(terms);
|
|
||||||
const files = new Map();
|
|
||||||
const unopened = [];
|
|
||||||
let scanned = 0;
|
|
||||||
let bytes = 0;
|
|
||||||
|
|
||||||
function record(name, hit, where, context, allowContext) {
|
|
||||||
const file = safeName(name);
|
|
||||||
for (const rule of allow) {
|
|
||||||
if (rule.glob.test(file) && rule.pattern.test(allowContext)) { rule.used += 1; return; }
|
|
||||||
}
|
|
||||||
let entry = files.get(file);
|
|
||||||
if (!entry) files.set(file, entry = { file, count: 0, masks: {}, hits: [] });
|
|
||||||
entry.count += 1;
|
|
||||||
entry.masks[hit.mask] = (entry.masks[hit.mask] ?? 0) + 1;
|
|
||||||
entry.hits.push({ where, mask: hit.mask, encoding: hit.encoding, context });
|
|
||||||
}
|
|
||||||
|
|
||||||
const safeName = name => printable(utf8(matcher.redact(Buffer.from(name, 'utf8').latin1Slice())));
|
|
||||||
|
|
||||||
/** The allowlist's view of a hit: its surroundings with this hit written <term>, any other <other>. */
|
|
||||||
const marked = (buffer, from, to, hit) => printable(`${matcher.redact(buffer.latin1Slice(from, hit.offset), '<other>')}<term>${matcher.redact(buffer.latin1Slice(hit.offset + hit.length, to), '<other>')}`);
|
|
||||||
|
|
||||||
/** Searches bytes; `base`: their offset inside the file, `skip`: hits below this offset were seen. */
|
|
||||||
function bytesOfFile(file, buffer, { base = 0, skip = 0, textual } = {}) {
|
|
||||||
const isText = textual ?? !buffer.subarray(0, 8000).includes(0);
|
|
||||||
const hits = matcher.find(buffer, !isText);
|
|
||||||
if (!hits.length) return;
|
|
||||||
let line = 1;
|
|
||||||
let lineStart = 0;
|
|
||||||
let cursor = 0;
|
|
||||||
for (const hit of hits) {
|
|
||||||
if (hit.offset < skip) continue;
|
|
||||||
let where;
|
|
||||||
let from;
|
|
||||||
let to;
|
|
||||||
if (isText && hit.encoding === 'utf8') {
|
|
||||||
for (let next = buffer.indexOf(10, cursor); next >= 0 && next < hit.offset; next = buffer.indexOf(10, cursor)) {
|
|
||||||
line += 1;
|
|
||||||
lineStart = next + 1;
|
|
||||||
cursor = next + 1;
|
|
||||||
}
|
|
||||||
where = `${line}:${hit.offset - lineStart + 1}`;
|
|
||||||
const lineEnd = buffer.indexOf(10, hit.offset);
|
|
||||||
from = Math.max(lineStart, hit.offset - 100);
|
|
||||||
to = Math.min(lineEnd < 0 ? buffer.length : lineEnd, hit.offset + hit.length + 100);
|
|
||||||
} else {
|
|
||||||
where = `@0x${(base + hit.offset).toString(16)}`;
|
|
||||||
from = Math.max(0, hit.offset - 160);
|
|
||||||
to = Math.min(buffer.length, hit.offset + hit.length + 160);
|
|
||||||
}
|
|
||||||
const near = buffer.latin1Slice(Math.max(from, hit.offset - 60), Math.min(to, hit.offset + hit.length + 60));
|
|
||||||
record(file, hit, where, printable(matcher.redact(near)), marked(buffer, from, to, hit));
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/** A name (a path inside an archive, a git path, an identity) searched as text. */
|
|
||||||
function text(file, value, label = 'name') {
|
|
||||||
const buffer = Buffer.from(value, 'utf8');
|
|
||||||
for (const hit of matcher.find(buffer)) {
|
|
||||||
const shown = printable(utf8(matcher.redact(buffer.latin1Slice(0, buffer.length))));
|
|
||||||
record(file, hit, label, shown.slice(0, 200), marked(buffer, 0, buffer.length, hit));
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/** Extracts with 7-Zip (or unsquashfs) into a temporary folder, scans it, removes it. */
|
|
||||||
function extracted(file, buffer, depth, { type, tool = 'container' } = {}) {
|
|
||||||
const dir = mkdtempSync(join(tmpdir(), 'wb-privacy-'));
|
|
||||||
try {
|
|
||||||
const source = join(dir, 'in');
|
|
||||||
const out = join(dir, 'out');
|
|
||||||
writeFileSync(source, buffer);
|
|
||||||
const seven = sevenZip(env);
|
|
||||||
let done = false;
|
|
||||||
if (seven) {
|
|
||||||
done = spawnSync(seven, ['x', '-y', '-bd', `-o${out}`, ...(type ? [`-t${type}`] : []), source], { stdio: 'ignore', windowsHide: true }).status === 0;
|
|
||||||
}
|
|
||||||
if (!done && type === 'SquashFS') {
|
|
||||||
done = spawnSync('unsquashfs', ['-no-progress', '-d', out, source], { stdio: 'ignore' }).status === 0;
|
|
||||||
}
|
|
||||||
if (!done || !existsSync(out)) {
|
|
||||||
unopened.push(`${file}: ${tool} not opened (install 7-Zip, or set ${SEVEN_ZIP_ENV})`);
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
folder(out, `${file}!`, depth + 1);
|
|
||||||
} finally {
|
|
||||||
rmSync(dir, { recursive: true, force: true });
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/** One file's bytes: searched, and opened when they are a container. */
|
|
||||||
function content(file, buffer, depth = 0) {
|
|
||||||
const note = message => unopened.push(`${file}: ${message}`);
|
|
||||||
const inner = (entries) => {
|
|
||||||
for (const entry of entries) {
|
|
||||||
const name = `${file}!/${safeName(entry.name)}`;
|
|
||||||
text(name, entry.name);
|
|
||||||
for (const value of entry.meta ?? []) text(`${name} (tar header)`, value, 'owner');
|
|
||||||
if (entry.leaf) bytesOfFile(name, entry.data, { textual: false });
|
|
||||||
else if (entry.data) content(name, entry.data, depth + 1);
|
|
||||||
}
|
|
||||||
};
|
|
||||||
const filtered = (data, what) => {
|
|
||||||
if (data) content(file, data, depth + 1);
|
|
||||||
else note(`${what} not opened`);
|
|
||||||
};
|
|
||||||
if (depth === 0) { scanned += 1; bytes += buffer.length; }
|
|
||||||
if (depth > 8) { bytesOfFile(file, buffer); return; }
|
|
||||||
try {
|
|
||||||
if (startsWith(buffer, [0x1f, 0x8b])) {
|
|
||||||
bytesOfFile(`${file} (gzip header)`, buffer.subarray(0, 512), { textual: false });
|
|
||||||
let data;
|
|
||||||
try { data = zlib.gunzipSync(buffer); } catch (error) { note(`gzip: ${error.message}`); }
|
|
||||||
if (data) content(file, data, depth + 1);
|
|
||||||
} else if (startsWith(buffer, [0x50, 0x4b, 0x03, 0x04])) {
|
|
||||||
inner(zipEntries(buffer, note));
|
|
||||||
} else if (buffer.length > 512 && ascii(buffer, 257, 5) === 'ustar') {
|
|
||||||
inner(tarEntries(buffer));
|
|
||||||
} else if (ascii(buffer, 0, 8) === '!<arch>\n') {
|
|
||||||
inner(arEntries(buffer));
|
|
||||||
} else if (startsWith(buffer, [0x28, 0xb5, 0x2f, 0xfd])) {
|
|
||||||
let data;
|
|
||||||
try { data = zlib.zstdDecompressSync?.(buffer); } catch { /* reported below */ }
|
|
||||||
filtered(data ?? pipe('zstd', ['-dc'], buffer), 'zstd');
|
|
||||||
} else if (startsWith(buffer, [0xfd, 0x37, 0x7a, 0x58, 0x5a, 0x00])) {
|
|
||||||
filtered(pipe('xz', ['-dc'], buffer), 'xz (install xz)');
|
|
||||||
} else if (startsWith(buffer, [0x42, 0x5a, 0x68]) && buffer[3] >= 0x31 && buffer[3] <= 0x39 && startsWith(buffer, [0x31, 0x41, 0x59, 0x26, 0x53, 0x59], 4)) {
|
|
||||||
filtered(pipe('bzip2', ['-dc'], buffer), 'bzip2 (install bzip2)');
|
|
||||||
} else if (ascii(buffer, 0, 4) === 'hsqs') {
|
|
||||||
extracted(file, buffer, depth, { type: 'SquashFS', tool: 'squashfs' });
|
|
||||||
} else if (buffer.length > 512 && ascii(buffer, buffer.length - 512, 4) === 'koly') {
|
|
||||||
const partitions = dmgPartitions(buffer);
|
|
||||||
if (partitions) partitions.forEach((data, i) => bytesOfFile(`${file}!/partition ${i}`, data, { textual: false }));
|
|
||||||
else extracted(file, buffer, depth, { tool: 'dmg' });
|
|
||||||
} else if (startsWith(buffer, [0x7f, 0x45, 0x4c, 0x46]) && ascii(buffer, 8, 2) === 'AI' && buffer[10] === 2) {
|
|
||||||
bytesOfFile(file, buffer);
|
|
||||||
const end = elfEnd(buffer);
|
|
||||||
if (end && ascii(buffer, end, 4) === 'hsqs') extracted(file, buffer.subarray(end), depth, { type: 'SquashFS', tool: 'AppImage' });
|
|
||||||
else note('AppImage: payload not found');
|
|
||||||
} else if (startsWith(buffer, [0x4d, 0x5a]) && buffer.includes('NullsoftInst')) {
|
|
||||||
bytesOfFile(file, buffer);
|
|
||||||
extracted(file, buffer, depth, { type: 'Nsis', tool: 'NSIS installer' });
|
|
||||||
} else {
|
|
||||||
bytesOfFile(file, buffer);
|
|
||||||
}
|
|
||||||
} catch (error) {
|
|
||||||
if (error instanceof PrivacyError) throw error;
|
|
||||||
note(`not opened (${error.message})`);
|
|
||||||
bytesOfFile(file, buffer);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/** A file on disk under the name `file`. Large plain files are read in overlapping chunks. */
|
|
||||||
function diskFile(path, file, depth = 0) {
|
|
||||||
const size = lstatSync(path).size;
|
|
||||||
if (size <= CHUNK * 4) { content(file, readFileSync(path), depth); return; }
|
|
||||||
scanned += 1;
|
|
||||||
bytes += size;
|
|
||||||
const handle = openSync(path, 'r');
|
|
||||||
const buffer = Buffer.alloc(CHUNK + OVERLAP);
|
|
||||||
let container = false;
|
|
||||||
try {
|
|
||||||
container = isContainer(buffer.subarray(0, readSync(handle, buffer, 0, 512, 0)));
|
|
||||||
for (let at = 0; at < size; at += CHUNK) {
|
|
||||||
const from = Math.max(0, at - OVERLAP);
|
|
||||||
const read = readSync(handle, buffer, 0, Math.min(CHUNK + (at - from), size - from), from);
|
|
||||||
bytesOfFile(file, buffer.subarray(0, read), { base: from, skip: at - from, textual: false });
|
|
||||||
}
|
|
||||||
} finally {
|
|
||||||
closeSync(handle);
|
|
||||||
}
|
|
||||||
if (container) unopened.push(`${file}: a container larger than ${(CHUNK * 4) / 2 ** 20} MB, searched as plain bytes only`);
|
|
||||||
}
|
|
||||||
|
|
||||||
function folder(path, prefix, depth = 0) {
|
|
||||||
for (const entry of readdirSync(path, { withFileTypes: true }).sort((a, b) => (a.name < b.name ? -1 : 1))) {
|
|
||||||
const full = join(path, entry.name);
|
|
||||||
const file = `${prefix}/${safeName(entry.name)}`;
|
|
||||||
text(file, entry.name);
|
|
||||||
if (entry.isDirectory()) { if (entry.name !== '.git') folder(full, file, depth); }
|
|
||||||
else if (entry.isFile()) diskFile(full, file, depth);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/** A file, folder or archive, shown relative to `cwd`. */
|
|
||||||
function path(target, cwd = process.cwd()) {
|
|
||||||
const full = resolve(cwd, target);
|
|
||||||
if (!existsSync(full)) fail(`not found: ${safeName(target)}`);
|
|
||||||
// Outside the working directory only the name is shown (the folders above it are not published).
|
|
||||||
const inside = relative(cwd, full);
|
|
||||||
const outside = inside.startsWith('..') || resolve(cwd, inside) !== full;
|
|
||||||
const shown = safeName(outside ? basename(full) : inside.replaceAll('\\', '/') || '.');
|
|
||||||
text(shown, outside ? basename(full) : inside);
|
|
||||||
if (lstatSync(full).isDirectory()) folder(full, shown === '.' ? '' : shown);
|
|
||||||
else diskFile(full, shown);
|
|
||||||
}
|
|
||||||
|
|
||||||
const git = (repo, args, options = {}) => {
|
|
||||||
const result = spawnSync('git', ['-C', repo, ...args], { maxBuffer: 2 ** 31 - 1, windowsHide: true, ...options });
|
|
||||||
if (result.status !== 0) fail(`git ${args[0]} failed in ${safeName(repo)}: ${String(result.stderr ?? '').trim().split('\n')[0]}`);
|
|
||||||
return result.stdout;
|
|
||||||
};
|
|
||||||
|
|
||||||
/** The files git tracks under `target` (and, with `untracked`, the ones it neither tracks nor ignores). */
|
|
||||||
function tracked(target, { cwd = process.cwd(), untracked = false } = {}) {
|
|
||||||
const dir = resolve(cwd, target);
|
|
||||||
const list = git(dir, ['ls-files', '-z', ...(untracked ? ['--cached', '--others', '--exclude-standard'] : [])]).toString('utf8').split('\0').filter(Boolean);
|
|
||||||
for (const name of [...new Set(list)]) {
|
|
||||||
const full = join(dir, name);
|
|
||||||
const shown = safeName(relative(cwd, full).replaceAll('\\', '/'));
|
|
||||||
text(shown, name);
|
|
||||||
let stat;
|
|
||||||
try { stat = lstatSync(full); } catch { continue; } // deleted in the work tree
|
|
||||||
if (stat.isFile()) diskFile(full, shown);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/** Everything a repository would publish: blobs, paths, commit and tag messages, identities, ref names. */
|
|
||||||
function history(target, { cwd = process.cwd() } = {}) {
|
|
||||||
const dir = resolve(cwd, target);
|
|
||||||
const label = safeName(relative(cwd, dir).replaceAll('\\', '/') || '.');
|
|
||||||
const names = new Map();
|
|
||||||
for (const line of git(dir, ['rev-list', '--all', '--objects']).toString('utf8').split('\n')) {
|
|
||||||
const space = line.indexOf(' ');
|
|
||||||
if (space > 0 && line.length > space + 1 && !names.has(line.slice(0, space))) names.set(line.slice(0, space), line.slice(space + 1));
|
|
||||||
}
|
|
||||||
for (const [, name] of names) text(`${label} history: ${safeName(name)}`, name, 'path');
|
|
||||||
const ids = [...names.keys()];
|
|
||||||
const kinds = git(dir, ['cat-file', '--batch-check=%(objectname) %(objecttype) %(objectsize)'], { input: `${ids.join('\n')}\n` }).toString('utf8').split('\n');
|
|
||||||
const blobs = kinds.map(line => line.split(' ')).filter(parts => parts[1] === 'blob');
|
|
||||||
// In batches, so a repository with large media never sits in memory whole.
|
|
||||||
for (let i = 0; i < blobs.length;) {
|
|
||||||
const batch = [];
|
|
||||||
for (let size = 0; i < blobs.length && (size < CHUNK * 4 || !batch.length); i += 1) {
|
|
||||||
batch.push(blobs[i]);
|
|
||||||
size += Number(blobs[i][2]);
|
|
||||||
}
|
|
||||||
const out = git(dir, ['cat-file', '--batch'], { input: `${batch.map(parts => parts[0]).join('\n')}\n` });
|
|
||||||
let at = 0;
|
|
||||||
for (const [id, , size] of batch) {
|
|
||||||
at = out.indexOf(10, at) + 1;
|
|
||||||
content(`${label} history: ${safeName(names.get(id))} (blob ${id.slice(0, 10)})`, out.subarray(at, at + Number(size)));
|
|
||||||
at += Number(size) + 1;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
const log = git(dir, ['log', '--all', '--format=%H%x00%an%x00%ae%x00%cn%x00%ce%x00%B%x01']).toString('utf8');
|
|
||||||
for (const entry of log.split('\x01')) {
|
|
||||||
const [id, author, authorMail, committer, committerMail, message] = entry.replace(/^\n/, '').split('\0');
|
|
||||||
if (!id) continue;
|
|
||||||
const commit = `${label} history: commit ${id.slice(0, 10)}`;
|
|
||||||
text(commit, `${author} <${authorMail}>`, 'author');
|
|
||||||
text(commit, `${committer} <${committerMail}>`, 'committer');
|
|
||||||
text(commit, message ?? '', 'message');
|
|
||||||
}
|
|
||||||
const refs = git(dir, ['for-each-ref', '--format=%(refname)%00%(taggername) %(taggeremail)%00%(contents)%01']).toString('utf8');
|
|
||||||
for (const entry of refs.split('\x01')) {
|
|
||||||
const [ref, tagger, message] = entry.replace(/^\n/, '').split('\0');
|
|
||||||
if (!ref) continue;
|
|
||||||
text(`${label} history: ${safeName(ref)}`, ref, 'ref');
|
|
||||||
if (tagger?.trim()) text(`${label} history: ${safeName(ref)}`, tagger, 'tagger');
|
|
||||||
if (ref.startsWith('refs/tags/') && message) text(`${label} history: ${safeName(ref)}`, message, 'tag message');
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
function report() {
|
|
||||||
const list = [...files.values()];
|
|
||||||
return { hits: list.reduce((sum, entry) => sum + entry.count, 0), files: list, unopened: [...unopened], scanned, bytes, terms: matcher.size };
|
|
||||||
}
|
|
||||||
|
|
||||||
return { path, tracked, history, content, text, report };
|
|
||||||
}
|
|
||||||
|
|
||||||
/** The report as lines (hits masked). `all`: every hit instead of the first five of each file. */
|
|
||||||
export function formatReport(report, { all = false, label = 'privacy-scan' } = {}) {
|
|
||||||
const lines = [];
|
|
||||||
for (const entry of report.files) {
|
|
||||||
const masks = Object.entries(entry.masks).map(([mask, count]) => `${mask} x${count}`).join(', ');
|
|
||||||
lines.push(`${entry.file}: ${entry.count} ${entry.count === 1 ? 'hit' : 'hits'} (${masks})`);
|
|
||||||
for (const hit of all ? entry.hits : entry.hits.slice(0, 5)) {
|
|
||||||
lines.push(` ${hit.where} ${hit.mask}${hit.encoding === 'utf8' ? '' : ` [${hit.encoding}]`} ${hit.context.slice(0, 160)}`);
|
|
||||||
}
|
|
||||||
if (!all && entry.hits.length > 5) lines.push(` ... ${entry.hits.length - 5} more (--all)`);
|
|
||||||
}
|
|
||||||
for (const item of report.unopened) lines.push(`not opened: ${item}`);
|
|
||||||
const size = report.bytes >= 2 ** 20 ? `${(report.bytes / 2 ** 20).toFixed(1)} MB` : `${report.bytes} bytes`;
|
|
||||||
lines.push(`${label}: ${report.hits} ${report.hits === 1 ? 'hit' : 'hits'} in ${report.files.length} ${report.files.length === 1 ? 'file' : 'files'}`
|
|
||||||
+ `${report.unopened.length ? `, ${report.unopened.length} not opened` : ''} (${report.scanned} files, ${size}, ${report.terms} terms)`);
|
|
||||||
return lines;
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Scans artifacts and throws when one holds a term or could not be opened (the release gate).
|
|
||||||
* `paths`: files or folders; `terms`/`allowlist`: files (both optional).
|
|
||||||
*/
|
|
||||||
export function assertClean(paths, { terms, allowlist, cwd = process.cwd(), env = process.env, log = console.log, label = 'privacy-scan' } = {}) {
|
|
||||||
const file = env[TERMS_ENV] || terms;
|
|
||||||
if (file && !existsSync(file)) fail(`terms file not found (${TERMS_ENV} or "forbiddenTerms"): fix the path, it is the release gate`);
|
|
||||||
const collected = collectTerms({ file });
|
|
||||||
const matcher = createMatcher(collected.terms);
|
|
||||||
const scan = createScan({ terms: collected.terms, allow: loadAllowlist(allowlist, matcher), env });
|
|
||||||
for (const target of paths) scan.path(target, cwd);
|
|
||||||
const report = scan.report();
|
|
||||||
for (const line of formatReport(report, { label })) log(line);
|
|
||||||
if (report.hits) fail(`${report.hits} forbidden ${report.hits === 1 ? 'term' : 'terms'} in the artifacts: nothing was released`);
|
|
||||||
if (report.unopened.length) fail(`${report.unopened.length} ${report.unopened.length === 1 ? 'container' : 'containers'} could not be opened, so the artifacts are not proven clean`);
|
|
||||||
return report;
|
|
||||||
}
|
|
||||||
|
|
||||||
function flag(args, name) {
|
|
||||||
const i = args.indexOf(name);
|
|
||||||
if (i < 0) return undefined;
|
|
||||||
const [, value] = args.splice(i, 2);
|
|
||||||
if (value === undefined) fail(`${name} needs a value`);
|
|
||||||
return value;
|
|
||||||
}
|
|
||||||
function toggle(args, name) {
|
|
||||||
const i = args.indexOf(name);
|
|
||||||
if (i >= 0) args.splice(i, 1);
|
|
||||||
return i >= 0;
|
|
||||||
}
|
|
||||||
|
|
||||||
/** `waybetter privacy-scan ...`; returns the exit code (0 clean, 1 hits, 2 something not opened). */
|
|
||||||
export function privacyCommand(argv, { cwd = process.cwd(), env = process.env, log = console.log } = {}) {
|
|
||||||
const args = [...argv];
|
|
||||||
if (toggle(args, '--help') || toggle(args, '-h')) { log(PRIVACY_USAGE); return 0; }
|
|
||||||
const config = privacyConfig(cwd);
|
|
||||||
const explicit = flag(args, '--terms');
|
|
||||||
const allowFile = flag(args, '--allow');
|
|
||||||
const options = Object.fromEntries(['--tracked', '--untracked', '--history', '--all', '--json', '--if-terms', '--no-auto'].map(name => [name, toggle(args, name)]));
|
|
||||||
const unknown = args.find(arg => arg.startsWith('--'));
|
|
||||||
if (unknown) fail(`unknown option ${unknown}`);
|
|
||||||
|
|
||||||
const candidate = explicit ? resolve(cwd, expandHome(explicit)) : env[TERMS_ENV] ? resolve(cwd, expandHome(env[TERMS_ENV])) : config.forbiddenTerms;
|
|
||||||
const file = candidate && existsSync(candidate) ? candidate : undefined;
|
|
||||||
if (!file && options['--if-terms']) return 0;
|
|
||||||
if (explicit && !file) fail('--terms: file not found');
|
|
||||||
const collected = collectTerms({ file, auto: !options['--no-auto'] });
|
|
||||||
if (!collected.terms.length) fail('no terms: pass --terms <file> or set ' + TERMS_ENV);
|
|
||||||
const matcher = createMatcher(collected.terms);
|
|
||||||
const allow = loadAllowlist(allowFile ? resolve(cwd, allowFile) : config.privacyAllowlist, matcher);
|
|
||||||
const scan = createScan({ terms: collected.terms, allow, env });
|
|
||||||
const targets = args.length ? args : ['.'];
|
|
||||||
for (const target of targets) {
|
|
||||||
if (options['--history']) scan.history(target, { cwd });
|
|
||||||
if (options['--tracked']) scan.tracked(target, { cwd, untracked: options['--untracked'] });
|
|
||||||
if (!options['--history'] && !options['--tracked']) scan.path(target, cwd);
|
|
||||||
}
|
|
||||||
const report = scan.report();
|
|
||||||
if (options['--json']) log(JSON.stringify(report, null, 2));
|
|
||||||
else {
|
|
||||||
for (const line of formatReport(report, { all: options['--all'] })) log(line);
|
|
||||||
if (!file) log(`privacy-scan: no terms file, only this machine's account name was searched (--terms, ${TERMS_ENV})`);
|
|
||||||
}
|
|
||||||
return report.hits ? 1 : report.unopened.length ? 2 : 0;
|
|
||||||
}
|
|
||||||
|
|
@ -17,17 +17,13 @@
|
||||||
// "prepareCommand": "bun install --frozen-lockfile", // remote/WSL builds: runs first in the unpacked sources
|
// "prepareCommand": "bun install --frozen-lockfile", // remote/WSL builds: runs first in the unpacked sources
|
||||||
// "distribution": { "publisher": "CRT", "id": "CRT.Name", "homepage": "https://...", "license": "...", "shortDescription": "...", "downloadBase": "https://..." },
|
// "distribution": { "publisher": "CRT", "id": "CRT.Name", "homepage": "https://...", "license": "...", "shortDescription": "...", "downloadBase": "https://..." },
|
||||||
// // manifests written after each release step (see cli/manifests.mjs)
|
// // manifests written after each release step (see cli/manifests.mjs)
|
||||||
// "exportExclude": ["verification"], // remote/WSL builds: app-repo paths left out of the upload
|
// "exportExclude": ["verification"] // remote/WSL builds: app-repo paths left out of the upload
|
||||||
// "forbiddenTerms": "~/.better-twttr-keys/forbidden-names.txt", // optional, outside the repo
|
|
||||||
// "privacyAllowlist": "privacy-allowlist.txt" // optional, reviewed false positives
|
|
||||||
// }
|
// }
|
||||||
//
|
//
|
||||||
// Privacy: the Windows build runs with the path-free build environment (cli/buildenv.mjs), and
|
// The Windows build runs with the path-free build environment (cli/buildenv.mjs). Every artifact
|
||||||
// every artifact (the installer and what it installs, the main binary, the front end, the dmg and
|
// (the installer, the main binary, the front end, the dmg and the updater bundle) goes through the
|
||||||
// the updater bundle) goes through `waybetter privacy-scan` (cli/privacy.mjs) before anything is
|
// optional release check (cli/releasecheck.mjs, WAYBETTER_RELEASE_CHECK) before anything is
|
||||||
// copied, signed or written: a forbidden term, or a container that cannot be opened (the NSIS
|
// copied, signed or written.
|
||||||
// installer needs 7-Zip), stops the release. The terms are the file "forbiddenTerms" names (or
|
|
||||||
// WAYBETTER_FORBIDDEN_TERMS) plus this machine's account name and home folder name.
|
|
||||||
//
|
//
|
||||||
// The private key is read only into the build's environment (Windows) or handed to the Tauri
|
// The private key is read only into the build's environment (Windows) or handed to the Tauri
|
||||||
// signer by path (macOS). Nothing here prints, copies or moves it.
|
// signer by path (macOS). Nothing here prints, copies or moves it.
|
||||||
|
|
@ -37,7 +33,7 @@ import { homedir } from 'node:os';
|
||||||
import { dirname, isAbsolute, join, relative, resolve } from 'node:path';
|
import { dirname, isAbsolute, join, relative, resolve } from 'node:path';
|
||||||
import { pathToFileURL } from 'node:url';
|
import { pathToFileURL } from 'node:url';
|
||||||
import { pathFreeBuildEnv } from './buildenv.mjs';
|
import { pathFreeBuildEnv } from './buildenv.mjs';
|
||||||
import { assertClean } from './privacy.mjs';
|
import { runReleaseCheck } from './releasecheck.mjs';
|
||||||
import { CHANNELS, resolveSpec, writeManifests } from './manifests.mjs';
|
import { CHANNELS, resolveSpec, writeManifests } from './manifests.mjs';
|
||||||
import { buildLinuxWsl, buildMacRemote, scanBuilt } from './remote.mjs';
|
import { buildLinuxWsl, buildMacRemote, scanBuilt } from './remote.mjs';
|
||||||
|
|
||||||
|
|
@ -65,10 +61,9 @@ export const RELEASE_USAGE = `Usage:
|
||||||
linux-x86_64 to latest.json (run the Windows step first).
|
linux-x86_64 to latest.json (run the Windows step first).
|
||||||
|
|
||||||
Config: ${CONFIG_FILE} in the app's root (product, artifact, desktopDir, keysDir, keysDirEnv,
|
Config: ${CONFIG_FILE} in the app's root (product, artifact, desktopDir, keysDir, keysDirEnv,
|
||||||
cdnBase, apiBase, outDir, optional manifestValidator, forbiddenTerms, privacyAllowlist). Env:
|
cdnBase, apiBase, outDir, optional manifestValidator). Env: CARGO_TARGET_DIR, CDN_BASE,
|
||||||
CARGO_TARGET_DIR, CDN_BASE, DEPLOY_TARGET, CDN_DIR, RELEASES_DIR, WAYBETTER_FORBIDDEN_TERMS,
|
DEPLOY_TARGET, CDN_DIR, RELEASES_DIR, WAYBETTER_RELEASE_CHECK (an optional command run on every
|
||||||
WAYBETTER_7Z. The build is path-free (waybetter build-env) and every artifact must pass
|
artifact first). The build is path-free (waybetter build-env). Nothing is ever uploaded.`;
|
||||||
waybetter privacy-scan. Nothing is ever uploaded.`;
|
|
||||||
|
|
||||||
class ReleaseError extends Error {}
|
class ReleaseError extends Error {}
|
||||||
const fail = message => { throw new ReleaseError(message); };
|
const fail = message => { throw new ReleaseError(message); };
|
||||||
|
|
@ -127,8 +122,6 @@ export function normalizeReleaseConfig(raw, root) {
|
||||||
distribution: raw.distribution,
|
distribution: raw.distribution,
|
||||||
exportExclude: Array.isArray(raw.exportExclude) ? raw.exportExclude.map(String) : [],
|
exportExclude: Array.isArray(raw.exportExclude) ? raw.exportExclude.map(String) : [],
|
||||||
manifestValidator: validator && { module: at(validator.module), export: validator.export, hint: validator.hint },
|
manifestValidator: validator && { module: at(validator.module), export: validator.export, hint: validator.hint },
|
||||||
forbiddenTerms: raw.forbiddenTerms === undefined ? undefined : at(home(text('forbiddenTerms'))),
|
|
||||||
privacyAllowlist: raw.privacyAllowlist === undefined ? undefined : at(text('privacyAllowlist')),
|
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -179,9 +172,9 @@ function readVersion(config) {
|
||||||
return { version, productName: conf.productName, binary, dist };
|
return { version, productName: conf.productName, binary, dist };
|
||||||
}
|
}
|
||||||
|
|
||||||
/** The release gate: no artifact may hold a forbidden term (see cli/privacy.mjs). */
|
/** The optional release check (cli/releasecheck.mjs) over every artifact that exists. */
|
||||||
function gate(config, paths, env, log) {
|
function gate(config, paths, env, log) {
|
||||||
assertClean(paths.filter(path => path && existsSync(path)), { terms: config.forbiddenTerms, allowlist: config.privacyAllowlist, cwd: config.root, env, log, label: 'release: privacy-scan' });
|
runReleaseCheck(paths.filter(path => path && existsSync(path)), { env, log, label: 'release: check' });
|
||||||
}
|
}
|
||||||
|
|
||||||
function option(args, name) {
|
function option(args, name) {
|
||||||
|
|
|
||||||
15
packages/waybetter/cli/releasecheck.mjs
Normal file
15
packages/waybetter/cli/releasecheck.mjs
Normal file
|
|
@ -0,0 +1,15 @@
|
||||||
|
// An optional, owner-supplied check run on every artifact before a release copies, signs or writes
|
||||||
|
// anything: WAYBETTER_RELEASE_CHECK names a command (run through the shell) that gets the artifact
|
||||||
|
// paths as arguments. A non-zero exit stops the release. Unset, nothing runs.
|
||||||
|
import { spawnSync } from 'node:child_process';
|
||||||
|
|
||||||
|
export const RELEASE_CHECK_ENV = 'WAYBETTER_RELEASE_CHECK';
|
||||||
|
|
||||||
|
export function runReleaseCheck(paths, { env = process.env, log = console.log, label = 'release check' } = {}) {
|
||||||
|
const command = env[RELEASE_CHECK_ENV];
|
||||||
|
if (!command || !paths.length) return;
|
||||||
|
const quoted = paths.map(path => `'${String(path).replaceAll("'", "'\\''")}'`).join(' ');
|
||||||
|
log(`${label}: ${paths.length} file(s)`);
|
||||||
|
const result = spawnSync(`${command} ${quoted}`, { shell: true, stdio: 'inherit', env });
|
||||||
|
if (result.status !== 0) throw new Error(`${label} failed (exit ${result.status ?? result.signal})`);
|
||||||
|
}
|
||||||
|
|
@ -1,7 +1,7 @@
|
||||||
// Remote and WSL builds for `waybetter release macos|linux`: the committed state of an app (its
|
// Remote and WSL builds for `waybetter release macos|linux`: the committed state of an app (its
|
||||||
// git HEAD plus every submodule at the commit the app records) is exported, sent to the machine
|
// git HEAD plus every submodule at the commit the app records) is exported, sent to the machine
|
||||||
// that builds it, built there through `waybetter build-env` (no build path in the binary) and the
|
// that builds it, built there through `waybetter build-env` (no build path in the binary) and the
|
||||||
// artifacts come back to be scanned (`waybetter privacy-scan`) and, later, signed on this machine.
|
// artifacts come back to be checked (the optional release check) and, later, signed on this machine.
|
||||||
// Nothing here touches the working tree the developer has open, and no key ever leaves this
|
// Nothing here touches the working tree the developer has open, and no key ever leaves this
|
||||||
// machine: the remote build is unsigned (updater artifacts off), signing is `waybetter release`.
|
// machine: the remote build is unsigned (updater artifacts off), signing is `waybetter release`.
|
||||||
//
|
//
|
||||||
|
|
@ -18,7 +18,7 @@ import { spawnSync } from 'node:child_process';
|
||||||
import { existsSync, mkdirSync, mkdtempSync, readdirSync, readFileSync, rmSync, statSync, writeFileSync } from 'node:fs';
|
import { existsSync, mkdirSync, mkdtempSync, readdirSync, readFileSync, rmSync, statSync, writeFileSync } from 'node:fs';
|
||||||
import { tmpdir } from 'node:os';
|
import { tmpdir } from 'node:os';
|
||||||
import { basename, join, relative, resolve } from 'node:path';
|
import { basename, join, relative, resolve } from 'node:path';
|
||||||
import { assertClean } from './privacy.mjs';
|
import { runReleaseCheck } from './releasecheck.mjs';
|
||||||
|
|
||||||
class RemoteError extends Error {}
|
class RemoteError extends Error {}
|
||||||
const fail = message => { throw new RemoteError(message); };
|
const fail = message => { throw new RemoteError(message); };
|
||||||
|
|
@ -250,11 +250,11 @@ export function buildLinuxWsl({ config, distro = 'Ubuntu-22.04', remoteDir, ref
|
||||||
|
|
||||||
const relTauri = (config, name) => `${relative(config.root, config.tauriDir).replaceAll('\\', '/')}/${name}`;
|
const relTauri = (config, name) => `${relative(config.root, config.tauriDir).replaceAll('\\', '/')}/${name}`;
|
||||||
|
|
||||||
/** Scans built artifacts here (the terms never leave this machine) and prints their sizes. */
|
/** Prints the built artifacts' sizes and runs the optional release check on them here. */
|
||||||
export function scanBuilt(config, files, { env = process.env, log = console.log } = {}) {
|
export function scanBuilt(config, files, { env = process.env, log = console.log } = {}) {
|
||||||
for (const file of files) {
|
for (const file of files) {
|
||||||
if (!existsSync(file)) fail(`expected build output missing: ${file}`);
|
if (!existsSync(file)) fail(`expected build output missing: ${file}`);
|
||||||
log(`built: ${basename(file)} ${(statSync(file).size / 1048576).toFixed(1)} MB`);
|
log(`built: ${basename(file)} ${(statSync(file).size / 1048576).toFixed(1)} MB`);
|
||||||
}
|
}
|
||||||
assertClean(files, { terms: config.forbiddenTerms, allowlist: config.privacyAllowlist, cwd: config.root, env, log, label: 'release: privacy-scan' });
|
runReleaseCheck(files, { env, log, label: 'release: check' });
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -3,7 +3,7 @@
|
||||||
"version": "0.7.0",
|
"version": "0.7.0",
|
||||||
"license": "GPL-3.0-or-later",
|
"license": "GPL-3.0-or-later",
|
||||||
"type": "module",
|
"type": "module",
|
||||||
"description": "wayBetter: the framework for apps that run the apps you already use in a light native (Tauri) window and extend them in place. Page runtime (DOM/request hubs, plugins, plugin UI kit with themed dialogs and toasts, theme API and theme JSON Schema, theme color model, link routing, account reporting, deck columns, i18n: ICU-lite messages, locale fallback, plugin strings, catalog checks and JSON Schemas), shell models (address bar, panes, decks, profiles, palette, updates), the shell frame (createShell: tabs, splits, address bar, titlebar, palette, account switcher, decks, workspaces, overlays, the language state; React + dockview peers), the panel (createPanel: an app's own tab with Appearance and the Theme Builder, plugins and custom CSS, the Store, Make, accounts, settings, backup and sync, developer mode, updates; the app adds its sections) and the Store: install screen and Store view React components (optional React peer), their state and catalog models, the store catalog JSON Schema and the `waybetter store build|sign|verify` command, `waybetter release windows|macos` (signed native releases and the update manifest, configured by waybetter.release.json; path-free builds and a privacy gate), `waybetter privacy-scan` (forbidden names in files, binaries, archives and git history), `waybetter build-env` (a build without the machine's paths) and `waybetter/build` (the page runtime bundle, with an optional esbuild peer). No dependencies.",
|
"description": "wayBetter: the framework for apps that run the apps you already use in a light native (Tauri) window and extend them in place. Page runtime (DOM/request hubs, plugins, plugin UI kit with themed dialogs and toasts, theme API and theme JSON Schema, theme color model, link routing, account reporting, deck columns, i18n: ICU-lite messages, locale fallback, plugin strings, catalog checks and JSON Schemas), shell models (address bar, panes, decks, profiles, palette, updates), the shell frame (createShell: tabs, splits, address bar, titlebar, palette, account switcher, decks, workspaces, overlays, the language state; React + dockview peers), the panel (createPanel: an app's own tab with Appearance and the Theme Builder, plugins and custom CSS, the Store, Make, accounts, settings, backup and sync, developer mode, updates; the app adds its sections) and the Store: install screen and Store view React components (optional React peer), their state and catalog models, the store catalog JSON Schema and the `waybetter store build|sign|verify` command, `waybetter release windows|macos` (signed native releases and the update manifest, configured by waybetter.release.json; path-free builds), `waybetter build-env` (a build without the machine's paths) and `waybetter/build` (the page runtime bundle, with an optional esbuild peer). No dependencies.",
|
||||||
"keywords": [
|
"keywords": [
|
||||||
"waybetter",
|
"waybetter",
|
||||||
"tauri",
|
"tauri",
|
||||||
|
|
@ -190,7 +190,6 @@
|
||||||
"./cli/store": "./cli/store.mjs",
|
"./cli/store": "./cli/store.mjs",
|
||||||
"./cli/release": "./cli/release.mjs",
|
"./cli/release": "./cli/release.mjs",
|
||||||
"./cli/i18n": "./cli/i18n.mjs",
|
"./cli/i18n": "./cli/i18n.mjs",
|
||||||
"./cli/privacy": "./cli/privacy.mjs",
|
|
||||||
"./cli/buildenv": "./cli/buildenv.mjs",
|
"./cli/buildenv": "./cli/buildenv.mjs",
|
||||||
"./build": "./build/runtime.mjs",
|
"./build": "./build/runtime.mjs",
|
||||||
"./package.json": "./package.json"
|
"./package.json": "./package.json"
|
||||||
|
|
|
||||||
|
|
@ -1,288 +0,0 @@
|
||||||
// `waybetter privacy-scan` and `waybetter build-env`: matching rules, masking, containers, git,
|
|
||||||
// the allowlist, exit codes, the path-free build flags and the release gate. The terms here are
|
|
||||||
// invented ("zorblat", "qix"); no real name belongs in a test.
|
|
||||||
import assert from 'node:assert/strict';
|
|
||||||
import { execFileSync } from 'node:child_process';
|
|
||||||
import { existsSync, mkdirSync, mkdtempSync, writeFileSync } from 'node:fs';
|
|
||||||
import { tmpdir } from 'node:os';
|
|
||||||
import { join } from 'node:path';
|
|
||||||
import test from 'node:test';
|
|
||||||
import { fileURLToPath } from 'node:url';
|
|
||||||
import { deflateRawSync, gzipSync } from 'node:zlib';
|
|
||||||
import { pathFreeBuildEnv } from '../cli/buildenv.mjs';
|
|
||||||
import { collectTerms, createMatcher, createScan, formatReport, loadAllowlist, privacyCommand } from '../cli/privacy.mjs';
|
|
||||||
import { releaseCommand } from '../cli/release.mjs';
|
|
||||||
|
|
||||||
const BIN = fileURLToPath(new URL('../bin/waybetter.mjs', import.meta.url));
|
|
||||||
const TERMS = [{ text: 'zorblat', rule: 'word' }, { text: 'qix', rule: 'account' }];
|
|
||||||
const dir = () => mkdtempSync(join(tmpdir(), 'wb-privacy-test-'));
|
|
||||||
const hitsIn = text => createMatcher(TERMS).find(Buffer.isBuffer(text) ? text : Buffer.from(text)).length;
|
|
||||||
|
|
||||||
/** A tar archive of `{ name, data, user }` entries (ustar headers, like npm pack writes). */
|
|
||||||
function tar(entries) {
|
|
||||||
const blocks = [];
|
|
||||||
for (const { name, data, user = '' } of entries) {
|
|
||||||
const header = Buffer.alloc(512);
|
|
||||||
header.write(name, 0);
|
|
||||||
header.write('0000644\0', 100);
|
|
||||||
header.write(`${data.length.toString(8).padStart(11, '0')}\0`, 124);
|
|
||||||
header.write('0', 156);
|
|
||||||
header.write('ustar\0' + '00', 257);
|
|
||||||
header.write(user, 265);
|
|
||||||
blocks.push(header, data, Buffer.alloc((512 - (data.length % 512)) % 512));
|
|
||||||
}
|
|
||||||
return Buffer.concat([...blocks, Buffer.alloc(1024)]);
|
|
||||||
}
|
|
||||||
|
|
||||||
/** A zip archive of `{ name, data, stored }` entries. */
|
|
||||||
function zip(entries) {
|
|
||||||
const locals = [];
|
|
||||||
const central = [];
|
|
||||||
let offset = 0;
|
|
||||||
for (const { name, data, stored } of entries) {
|
|
||||||
const packed = stored ? data : deflateRawSync(data);
|
|
||||||
const local = Buffer.alloc(30 + name.length);
|
|
||||||
local.writeUInt32LE(0x04034b50, 0);
|
|
||||||
local.writeUInt16LE(stored ? 0 : 8, 8);
|
|
||||||
local.writeUInt32LE(packed.length, 18);
|
|
||||||
local.writeUInt32LE(data.length, 22);
|
|
||||||
local.writeUInt16LE(name.length, 26);
|
|
||||||
local.write(name, 30);
|
|
||||||
const entry = Buffer.alloc(46 + name.length);
|
|
||||||
entry.writeUInt32LE(0x02014b50, 0);
|
|
||||||
entry.writeUInt16LE(stored ? 0 : 8, 10);
|
|
||||||
entry.writeUInt32LE(packed.length, 20);
|
|
||||||
entry.writeUInt32LE(data.length, 24);
|
|
||||||
entry.writeUInt16LE(name.length, 28);
|
|
||||||
entry.writeUInt32LE(offset, 42);
|
|
||||||
entry.write(name, 46);
|
|
||||||
locals.push(local, packed);
|
|
||||||
central.push(entry);
|
|
||||||
offset += local.length + packed.length;
|
|
||||||
}
|
|
||||||
const directory = Buffer.concat(central);
|
|
||||||
const end = Buffer.alloc(22);
|
|
||||||
end.writeUInt32LE(0x06054b50, 0);
|
|
||||||
end.writeUInt16LE(entries.length, 8);
|
|
||||||
end.writeUInt16LE(entries.length, 10);
|
|
||||||
end.writeUInt32LE(directory.length, 12);
|
|
||||||
end.writeUInt32LE(offset, 16);
|
|
||||||
return Buffer.concat([...locals, directory, end]);
|
|
||||||
}
|
|
||||||
|
|
||||||
test('a name is a hit as a whole word, in any case, with diacritics; never inside another word', () => {
|
|
||||||
for (const text of ['by zorblat', 'ZORBLAT', 'Zorblat!', 'x_zorblat_y', 'a-zorblat-b', '@zorblat', 'zórblat', 'ZÖRBLAT', 'dir/zorblat/file']) {
|
|
||||||
assert.equal(hitsIn(text), 1, text);
|
|
||||||
}
|
|
||||||
for (const text of ['zorblats', 'azorblat', 'zorblat2', 'zorbla t']) assert.equal(hitsIn(text), 0, text);
|
|
||||||
});
|
|
||||||
|
|
||||||
test('a short account name is a hit only in a path or as the capitalized word', () => {
|
|
||||||
for (const text of ['qix(a, b)', 'qix-width: 3px', 'Math.qix', 'const qix = 1', 'QIX_SIZE', 'a qix b', 'qixed', 'src/qixes/a']) {
|
|
||||||
assert.equal(hitsIn(text), 0, text);
|
|
||||||
}
|
|
||||||
for (const text of ['C:\\Users\\qix\\Desktop', 'c:\\users\\QIX\\x', 'C:\\\\Users\\\\qix\\\\x', '/Users/qix', '/home/qix/.cargo', 'a\\qix\\b', 'a/qix/b',
|
|
||||||
'C--Users-qix-Desktop', 'file:///C:/Users/qix/x', 'C%3A%5CUsers%5Cqix%5Cx', 'by Qix.', 'Qix']) {
|
|
||||||
assert.equal(hitsIn(text), 1, text);
|
|
||||||
}
|
|
||||||
});
|
|
||||||
|
|
||||||
test('binary content: UTF-16 in either byte order, with the offset', () => {
|
|
||||||
const wide = Buffer.concat([Buffer.from([1, 2, 0, 0, 3, 0]), Buffer.from('C:\\Users\\qix\\AppData zorblat', 'utf16le'), Buffer.from([0, 0, 9])]);
|
|
||||||
assert.equal(hitsIn(wide), 2);
|
|
||||||
assert.equal(hitsIn(Buffer.from('\u0000:/zorblat/x', 'utf16le').swap16()), 1);
|
|
||||||
const scan = createScan({ terms: TERMS });
|
|
||||||
scan.content('app.exe', wide);
|
|
||||||
const [file] = scan.report().files;
|
|
||||||
assert.deepEqual(file.hits.map(hit => hit.encoding), ['utf16', 'utf16']);
|
|
||||||
assert.match(file.hits[0].where, /^@0x[0-9a-f]+$/);
|
|
||||||
});
|
|
||||||
|
|
||||||
test('binary content: three chance bytes are not a name; a field or a sentence is', () => {
|
|
||||||
const noise = Buffer.from([0, 0x91, 0xfe, 0x07, 0xc4]);
|
|
||||||
const count = (...parts) => {
|
|
||||||
const scan = createScan({ terms: TERMS });
|
|
||||||
scan.content('image.png', Buffer.concat([noise, ...parts.map(part => (typeof part === 'string' ? Buffer.from(part) : part)), noise]));
|
|
||||||
return scan.report().hits;
|
|
||||||
};
|
|
||||||
assert.equal(count('Qix'), 0);
|
|
||||||
assert.equal(count('k;Qix_'), 0);
|
|
||||||
assert.equal(count(Buffer.from([0]), 'Qix', Buffer.from([0])), 1);
|
|
||||||
assert.equal(count('Author: Qix, 2026'), 1);
|
|
||||||
assert.equal(count('C:\\Users\\qix\\x'), 1);
|
|
||||||
assert.equal(count('zorblat'), 1);
|
|
||||||
});
|
|
||||||
|
|
||||||
test('the report masks every term: first letter and ***, never the word', () => {
|
|
||||||
const scan = createScan({ terms: TERMS });
|
|
||||||
scan.content('notes/zorblat.txt', Buffer.from('first line\nwritten by Zorblat on C:\\Users\\qix\\notes, zorblat again\n'));
|
|
||||||
scan.text('notes/zorblat.txt', 'notes/zorblat.txt');
|
|
||||||
const report = scan.report();
|
|
||||||
assert.equal(report.hits, 4);
|
|
||||||
assert.equal(report.files[0].hits[0].where, '2:12');
|
|
||||||
const text = formatReport(report, { all: true }).join('\n');
|
|
||||||
assert.doesNotMatch(text, /zorblat|qix/i);
|
|
||||||
assert.match(text, /^notes\/z\*\*\*\.txt: 4 hits \(z\*\*\* x3, q\*\*\* x1\)/);
|
|
||||||
assert.match(text, /written by z\*\*\* on C:\\Users\\q\*\*\*\\notes/);
|
|
||||||
});
|
|
||||||
|
|
||||||
test('containers are opened: tar.gz (and its owner names), zip stored and deflated, nested', () => {
|
|
||||||
const inner = tar([{ name: 'package/readme.md', data: Buffer.from('made by zorblat') }, { name: 'package/clean.js', data: Buffer.from('ok'), user: 'zorblat' }]);
|
|
||||||
const archive = zip([
|
|
||||||
{ name: 'app/a.txt', data: Buffer.from('/home/qix/project'), stored: true },
|
|
||||||
{ name: 'app/b.bin', data: Buffer.concat([Buffer.alloc(40), Buffer.from('zorblat')]) },
|
|
||||||
{ name: 'app/zorblat.cfg', data: Buffer.from('nothing') },
|
|
||||||
{ name: 'app/pkg.tgz', data: gzipSync(inner) },
|
|
||||||
]);
|
|
||||||
const scan = createScan({ terms: TERMS });
|
|
||||||
scan.content('bundle.zip', archive);
|
|
||||||
const found = Object.fromEntries(scan.report().files.map(entry => [entry.file, entry.count]));
|
|
||||||
assert.deepEqual(found, {
|
|
||||||
'bundle.zip!/(zip directory)': 1,
|
|
||||||
'bundle.zip!/app/a.txt': 1,
|
|
||||||
'bundle.zip!/app/b.bin': 1,
|
|
||||||
'bundle.zip!/app/z***.cfg': 1,
|
|
||||||
'bundle.zip!/app/pkg.tgz!/package/readme.md': 1,
|
|
||||||
'bundle.zip!/app/pkg.tgz!/package/clean.js (tar header)': 1,
|
|
||||||
});
|
|
||||||
assert.equal(scan.report().files.at(-1).hits[0].where, 'owner');
|
|
||||||
assert.deepEqual(scan.report().unopened, []);
|
|
||||||
const clean = createScan({ terms: TERMS });
|
|
||||||
clean.content('ok.tgz', gzipSync(tar([{ name: 'package/index.js', data: Buffer.from('Math.max(qix, 1)') }])));
|
|
||||||
assert.equal(clean.report().hits, 0);
|
|
||||||
});
|
|
||||||
|
|
||||||
test('a container that cannot be opened is reported, not passed', () => {
|
|
||||||
const scan = createScan({ terms: TERMS });
|
|
||||||
const broken = gzipSync(Buffer.from('zorblat')).subarray(0, 14);
|
|
||||||
scan.content('broken.gz', broken);
|
|
||||||
assert.equal(scan.report().hits, 0);
|
|
||||||
assert.equal(scan.report().unopened.length, 1);
|
|
||||||
});
|
|
||||||
|
|
||||||
test('terms file: comments skipped, short terms get the account rule, generic accounts never added', () => {
|
|
||||||
const root = dir();
|
|
||||||
writeFileSync(join(root, 'terms.txt'), '# names\nzorblat\n\n Qix \nzorblat\n');
|
|
||||||
const { terms, fromFile } = collectTerms({ file: join(root, 'terms.txt'), auto: false });
|
|
||||||
assert.deepEqual(terms, [{ text: 'zorblat', rule: 'word' }, { text: 'Qix', rule: 'account' }]);
|
|
||||||
assert.equal(fromFile, 2);
|
|
||||||
assert.ok(!collectTerms({}).terms.some(term => ['root', 'runner', 'user'].includes(term.text.toLowerCase())));
|
|
||||||
});
|
|
||||||
|
|
||||||
test('allowlist: a reviewed context is skipped; the file may not hold a term', () => {
|
|
||||||
const root = dir();
|
|
||||||
writeFileSync(join(root, 'allow.txt'), '# a generated identifier\nsrc/**/*.rs: <term>-Id\n');
|
|
||||||
const matcher = createMatcher(TERMS);
|
|
||||||
const allow = loadAllowlist(join(root, 'allow.txt'), matcher);
|
|
||||||
const scan = createScan({ terms: TERMS, allow });
|
|
||||||
scan.content('src/net/cookie.rs', Buffer.from('header("Qix-Id=3600")\n// Qix wrote this\n'));
|
|
||||||
scan.content('docs/cookie.md', Buffer.from('Qix-Id\n'));
|
|
||||||
const report = scan.report();
|
|
||||||
assert.deepEqual(report.files.map(entry => [entry.file, entry.hits.map(hit => hit.where)]), [['src/net/cookie.rs', ['2:4']], ['docs/cookie.md', ['1:1']]]);
|
|
||||||
writeFileSync(join(root, 'bad.txt'), 'src/**: by zorblat\n');
|
|
||||||
assert.throws(() => loadAllowlist(join(root, 'bad.txt'), matcher), /must not contain a term/);
|
|
||||||
});
|
|
||||||
|
|
||||||
test('git: tracked files, then the history (blobs, paths, messages, identities)', () => {
|
|
||||||
const root = dir();
|
|
||||||
const git = (...args) => execFileSync('git', ['-C', root, '-c', 'user.name=Zorblat Dev', '-c', 'user.email=dev@example.test', '-c', 'core.autocrlf=false', '-c', `core.hooksPath=${join(root, '.no-hooks')}`, ...args], { stdio: 'pipe' });
|
|
||||||
git('init', '-q');
|
|
||||||
writeFileSync(join(root, 'notes.txt'), 'C:\\Users\\qix\\secret\n');
|
|
||||||
writeFileSync(join(root, 'ignored.log'), 'zorblat\n');
|
|
||||||
writeFileSync(join(root, '.gitignore'), '*.log\n');
|
|
||||||
git('add', '.');
|
|
||||||
git('commit', '-q', '-m', 'first, thanks zorblat');
|
|
||||||
writeFileSync(join(root, 'notes.txt'), 'C:\\Users\\<user>\\secret\n');
|
|
||||||
git('commit', '-q', '-am', 'paths made neutral');
|
|
||||||
|
|
||||||
const tracked = createScan({ terms: TERMS });
|
|
||||||
tracked.tracked('.', { cwd: root });
|
|
||||||
assert.equal(tracked.report().hits, 0);
|
|
||||||
|
|
||||||
const history = createScan({ terms: TERMS });
|
|
||||||
history.history('.', { cwd: root });
|
|
||||||
const where = history.report().files.flatMap(entry => entry.hits.map(hit => `${entry.file.replace(/[0-9a-f]{10}/, 'ID')} ${hit.where}`)).sort();
|
|
||||||
assert.deepEqual(where, [
|
|
||||||
'. history: commit ID author', '. history: commit ID author',
|
|
||||||
'. history: commit ID committer', '. history: commit ID committer',
|
|
||||||
'. history: commit ID message',
|
|
||||||
'. history: notes.txt (blob ID) 1:10',
|
|
||||||
]);
|
|
||||||
});
|
|
||||||
|
|
||||||
test('the command: exit 1 on a hit, 0 when clean, quiet with --if-terms and no terms file', () => {
|
|
||||||
const root = dir();
|
|
||||||
mkdirSync(join(root, 'dist'));
|
|
||||||
writeFileSync(join(root, 'terms.txt'), 'zorblat\n');
|
|
||||||
writeFileSync(join(root, 'dist/app.js'), 'console.log("clean")\n');
|
|
||||||
const lines = [];
|
|
||||||
const run = args => privacyCommand(args, { cwd: root, env: {}, log: line => lines.push(line) });
|
|
||||||
assert.equal(run(['dist', '--terms', 'terms.txt', '--no-auto']), 0);
|
|
||||||
assert.match(lines.at(-1), /^privacy-scan: 0 hits in 0 files \(1 files, /);
|
|
||||||
writeFileSync(join(root, 'dist/app.js.map'), '{"sources":["/home/zorblat/app/src/main.ts"]}');
|
|
||||||
assert.equal(run(['dist', '--terms', 'terms.txt', '--no-auto']), 1);
|
|
||||||
assert.doesNotMatch(lines.join('\n'), /zorblat/);
|
|
||||||
lines.length = 0;
|
|
||||||
assert.equal(run(['dist', '--if-terms']), 0);
|
|
||||||
assert.deepEqual(lines, []);
|
|
||||||
assert.throws(() => run(['dist', '--terms', 'missing.txt']), /not found/);
|
|
||||||
assert.equal(privacyCommand(['dist'], { cwd: root, env: { WAYBETTER_FORBIDDEN_TERMS: join(root, 'terms.txt') }, log: () => {} }), 1);
|
|
||||||
const out = execFileSync(process.execPath, [BIN, '--help'], { encoding: 'utf8' });
|
|
||||||
assert.match(out, /waybetter privacy-scan/);
|
|
||||||
assert.match(out, /waybetter build-env/);
|
|
||||||
});
|
|
||||||
|
|
||||||
test('build-env: roots remapped general to specific, existing flags kept, per platform', () => {
|
|
||||||
const windows = pathFreeBuildEnv({ root: 'C:\\Users\\dev\\app', home: 'C:\\Users\\dev', platform: 'win32', env: { RUSTFLAGS: '-Ctarget-cpu=native --cfg foo' } });
|
|
||||||
const flags = windows.env.CARGO_ENCODED_RUSTFLAGS.split('\x1f');
|
|
||||||
assert.deepEqual(flags.slice(0, 2), ['-Ctarget-cpu=native', '--cfg']);
|
|
||||||
assert.ok(flags.indexOf('--remap-path-prefix=C:\\Users\\dev=/home') < flags.indexOf('--remap-path-prefix=C:\\Users\\dev\\app=/src'));
|
|
||||||
assert.ok(flags.indexOf('--remap-path-prefix=C:\\Users\\dev\\app=/src') < flags.indexOf('--remap-path-prefix=C:\\Users\\dev\\.cargo=/cargo'));
|
|
||||||
assert.ok(flags.includes('--remap-path-prefix=\\\\?\\C:\\Users\\dev\\.rustup=/rustup'));
|
|
||||||
assert.equal(flags.at(-1), '-Clink-arg=/PDBALTPATH:%_PDB%');
|
|
||||||
assert.equal(windows.env.CFLAGS, undefined);
|
|
||||||
// MSVC C and C++: cl.exe takes the first /d1trimfile that matches, so the longest root leads.
|
|
||||||
assert.deepEqual(windows.env.CL.split(' '), ['C:/Users/dev/.rustup', 'C:/Users/dev/.cargo', 'C:/Users/dev/app', 'C:/Users/dev'].map(dir => `"/d1trimfile:${dir.replaceAll('/', '\\')}/"`));
|
|
||||||
assert.equal(windows.env.LINK, '/PDBALTPATH:%_PDB%');
|
|
||||||
const kept = pathFreeBuildEnv({ root: 'D:\\my apps\\app', home: 'C:\\Users\\dev', platform: 'win32', env: { CL: '/MP', LINK: '/DEBUG:NONE', CARGO_TARGET_DIR: 'D:\\t' } });
|
|
||||||
assert.ok(kept.env.CL.startsWith('/MP "/d1trimfile:C:\\Users\\dev\\.rustup/" '));
|
|
||||||
assert.ok(kept.env.CL.includes(' "/d1trimfile:D:\\my apps\\app/" ') && kept.env.CL.endsWith(' "/d1trimfile:D:\\t/"'));
|
|
||||||
assert.equal(kept.env.LINK, '/DEBUG:NONE /PDBALTPATH:%_PDB%');
|
|
||||||
const gnu = pathFreeBuildEnv({ root: 'C:\\Users\\dev\\app', home: 'C:\\Users\\dev', platform: 'win32', env: { CARGO_BUILD_TARGET: 'x86_64-pc-windows-gnu' } });
|
|
||||||
assert.equal(gnu.env.CL, undefined);
|
|
||||||
assert.equal(gnu.env.LINK, undefined);
|
|
||||||
|
|
||||||
const encoded = pathFreeBuildEnv({ root: '/home/dev/app', home: '/home/dev', platform: 'linux', env: { CARGO_ENCODED_RUSTFLAGS: '-Cfoo\x1f--remap-path-prefix=/home/dev=/home', CARGO_HOME: '/opt/cargo', CARGO_TARGET_DIR: '/tmp/t', CFLAGS: '-O2' } });
|
|
||||||
assert.deepEqual(encoded.env.CARGO_ENCODED_RUSTFLAGS.split('\x1f'), ['-Cfoo', '--remap-path-prefix=/home/dev=/home', '--remap-path-prefix=/home/dev/app=/src',
|
|
||||||
'--remap-path-prefix=/tmp/t=/target', '--remap-path-prefix=/opt/cargo=/cargo', '--remap-path-prefix=/home/dev/.rustup=/rustup']);
|
|
||||||
assert.match(encoded.env.CFLAGS, /^-O2 -ffile-prefix-map=\/home\/dev=\/home .*-ffile-prefix-map=\/opt\/cargo=\/cargo/);
|
|
||||||
assert.equal(encoded.env.CXXFLAGS.split(' ').length, 5);
|
|
||||||
assert.equal(encoded.env.CL, undefined);
|
|
||||||
});
|
|
||||||
|
|
||||||
test('release: an artifact holding a term stops the release before anything is written', async () => {
|
|
||||||
const root = dir();
|
|
||||||
const tauri = join(root, 'apps/desktop/src-tauri');
|
|
||||||
const nsis = join(root, 'target/release/bundle/nsis');
|
|
||||||
mkdirSync(tauri, { recursive: true });
|
|
||||||
mkdirSync(nsis, { recursive: true });
|
|
||||||
writeFileSync(join(root, 'terms.txt'), 'zorblat\n');
|
|
||||||
writeFileSync(join(root, 'waybetter.release.json'), JSON.stringify({ product: 'Test', artifact: 'Test', desktopDir: 'apps/desktop', keysDir: 'keys',
|
|
||||||
cdnBase: 'https://cdn.example.test', apiBase: 'https://api.example.test', forbiddenTerms: 'terms.txt' }));
|
|
||||||
for (const file of ['package.json', 'apps/desktop/package.json']) writeFileSync(join(root, file), JSON.stringify({ version: '1.0.0' }));
|
|
||||||
writeFileSync(join(tauri, 'tauri.conf.json'), JSON.stringify({ version: '1.0.0', productName: 'Test App' }));
|
|
||||||
writeFileSync(join(tauri, 'Cargo.toml'), '[package]\nname = "t"\nversion = "1.0.0"\n');
|
|
||||||
writeFileSync(join(nsis, 'Test App_1.0.0_x64-setup.exe.sig'), 'SIG\n');
|
|
||||||
const lines = [];
|
|
||||||
const release = () => releaseCommand(['windows', '--skip-build'], { cwd: root, env: { CARGO_TARGET_DIR: join(root, 'target') }, log: line => lines.push(line) });
|
|
||||||
|
|
||||||
writeFileSync(join(nsis, 'Test App_1.0.0_x64-setup.exe'), 'MZ... C:\\Users\\zorblat\\.cargo\\registry\\src\\lib.rs ...');
|
|
||||||
await assert.rejects(release(), /1 forbidden term in the artifacts: nothing was released/);
|
|
||||||
assert.ok(!existsSync(join(root, 'releases')));
|
|
||||||
assert.doesNotMatch(lines.join('\n'), /zorblat/);
|
|
||||||
assert.match(lines.join('\n'), /Test App_1\.0\.0_x64-setup\.exe: 1 hit \(z\*\*\* x1\)/);
|
|
||||||
|
|
||||||
writeFileSync(join(nsis, 'Test App_1.0.0_x64-setup.exe'), 'MZ... /cargo/registry/src/lib.rs ...');
|
|
||||||
await release();
|
|
||||||
assert.ok(existsSync(join(root, 'releases/native/latest.json')));
|
|
||||||
});
|
|
||||||
Loading…
Add table
Add a link
Reference in a new issue