fast ci: task graph runner, sharded tests balanced by recorded durations, ci:quick (typecheck + related tests + scrub), Rust only when src-tauri / sidecar build changed, event waits instead of sleeps
This commit is contained in:
parent
e38f68fa33
commit
5e1357116d
13 changed files with 472 additions and 45 deletions
|
|
@ -1,5 +1,7 @@
|
||||||
# bun run ci on every push to main and every pull request into main: adapters, desktop version, tests, typecheck,
|
# bun run ci on every push to main and every pull request into main: adapters, desktop version, tests, typecheck,
|
||||||
# desktop i18n / typecheck / build, desktop boot check (headless Chromium), desktop Rust tests and clippy, scrub. Runs on the VDS runner (docs/desktop-release.md).
|
# desktop i18n / typecheck / build, desktop boot check (headless Chromium), desktop Rust tests and clippy, scrub. Runs on the VDS runner (docs/desktop-release.md).
|
||||||
|
# tools/ci.ts runs the steps as a task graph and the tests as parallel shards (CPU cores - 1, max 8) on the one
|
||||||
|
# self-hosted runner, so a job matrix would only queue; CI=full makes it run the Rust steps on every push.
|
||||||
# Publishes nothing: site/ is still published by Jenkins.
|
# Publishes nothing: site/ is still published by Jenkins.
|
||||||
name: ci
|
name: ci
|
||||||
|
|
||||||
|
|
@ -14,6 +16,7 @@ env:
|
||||||
BUN_VERSION: '1.4.0'
|
BUN_VERSION: '1.4.0'
|
||||||
CARGO_TERM_COLOR: never
|
CARGO_TERM_COLOR: never
|
||||||
CARGO_INCREMENTAL: '0'
|
CARGO_INCREMENTAL: '0'
|
||||||
|
CI: full
|
||||||
|
|
||||||
jobs:
|
jobs:
|
||||||
ci:
|
ci:
|
||||||
|
|
@ -40,6 +43,12 @@ jobs:
|
||||||
apps/desktop/src-tauri/target
|
apps/desktop/src-tauri/target
|
||||||
key: ci-${{ runner.os }}-${{ hashFiles('bun.lock', 'apps/desktop/src-tauri/Cargo.lock') }}
|
key: ci-${{ runner.os }}-${{ hashFiles('bun.lock', 'apps/desktop/src-tauri/Cargo.lock') }}
|
||||||
restore-keys: ci-${{ runner.os }}-
|
restore-keys: ci-${{ runner.os }}-
|
||||||
|
# recorded test durations (shard balance) and tsc's incremental build info
|
||||||
|
- uses: actions/cache@v4
|
||||||
|
with:
|
||||||
|
path: node_modules/.cache/storebound-ci
|
||||||
|
key: ci-timings-${{ runner.os }}-${{ github.sha }}
|
||||||
|
restore-keys: ci-timings-${{ runner.os }}-
|
||||||
- name: Bun and Rust
|
- name: Bun and Rust
|
||||||
run: |
|
run: |
|
||||||
P="${FORGEJO_PATH:-$GITHUB_PATH}"
|
P="${FORGEJO_PATH:-$GITHUB_PATH}"
|
||||||
|
|
|
||||||
|
|
@ -86,5 +86,7 @@ These rules apply to every AI tool that uses storebound (skills, MCP server or t
|
||||||
## Working on storebound itself
|
## Working on storebound itself
|
||||||
|
|
||||||
- Bun + TypeScript strict, zero runtime dependencies. `bun run ci` must pass (adapters fresh, tests, typecheck, scrub).
|
- Bun + TypeScript strict, zero runtime dependencies. `bun run ci` must pass (adapters fresh, tests, typecheck, scrub).
|
||||||
|
- Inner loop: `bun run ci:quick` (typecheck, the tests related to your change against origin/main, scrub; seconds).
|
||||||
|
Before every push: the full `bun run ci` (every check and test, in parallel shards), and check its exit code.
|
||||||
- Skills are the single source: edit `skills/*/SKILL.md` or `docs/agents-core.md`, then `bun tools/gen-adapters.ts`.
|
- Skills are the single source: edit `skills/*/SKILL.md` or `docs/agents-core.md`, then `bun tools/gen-adapters.ts`.
|
||||||
- Never commit credentials, account / team / issuer / key / app ids or personal data; placeholders use example.com.
|
- Never commit credentials, account / team / issuer / key / app ids or personal data; placeholders use example.com.
|
||||||
|
|
|
||||||
|
|
@ -2,6 +2,11 @@
|
||||||
|
|
||||||
## Unreleased
|
## Unreleased
|
||||||
|
|
||||||
|
- Faster CI: `bun run ci` runs as a task graph (independent steps at once, `[step]`-prefixed output, fail fast, time
|
||||||
|
per step) with the tests split into parallel shards (CPU cores - 1, max 8) balanced by recorded durations, browser
|
||||||
|
suites in shards of their own. New `bun run ci:quick` for the inner loop: typecheck plus the tests related to the
|
||||||
|
change against origin/main (through imports) plus scrub. The desktop Rust host runs only when `src-tauri` or the
|
||||||
|
sidecar build changed (`CI=full` always; Forgejo CI sets it). Fixed sleeps in tests became event waits.
|
||||||
- Local font files in `shots.fonts`: `title`, `body` and `fallback[]` take, besides a Google Fonts family, a path
|
- Local font files in `shots.fonts`: `title`, `body` and `fallback[]` take, besides a Google Fonts family, a path
|
||||||
(relative to the config) or `{ family, file, weight?, style?, license? }` (TTF / OTF / WOFF / WOFF2, validated by
|
(relative to the config) or `{ family, file, weight?, style?, license? }` (TTF / OTF / WOFF / WOFF2, validated by
|
||||||
extension and signature; the family comes from the name table when not given). `shots fonts` checks the file's cmap
|
extension and signature; the family comes from the name table when not given). `shots fonts` checks the file's cmap
|
||||||
|
|
|
||||||
|
|
@ -7,9 +7,25 @@ but fixes, store rule updates and new engines / AI tools are very welcome.
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
bun install # Bun 1.1+; playwright is a dev dependency (renders, web capture)
|
bun install # Bun 1.1+; playwright is a dev dependency (renders, web capture)
|
||||||
bun run ci # adapters fresh, tests, typecheck, secrets / personal data scrub
|
bun run ci:quick # inner loop: typecheck + the tests your change touches + scrub (seconds)
|
||||||
|
bun run ci # full gate before pushing: every check and test (adapters, API tables, typecheck,
|
||||||
|
# desktop build and boot check, Rust host when it changed, scrub)
|
||||||
```
|
```
|
||||||
|
|
||||||
|
**Which one when.** Run `bun run ci:quick` while you work: it diffs against `origin/main` (committed, staged,
|
||||||
|
unstaged and new files), maps the changed files to the tests that import them (directly, through other modules, or
|
||||||
|
by naming a fixture or page) and runs those, plus `tsc` (incremental) and the scrub. A change to `package.json`,
|
||||||
|
`bun.lock`, `bunfig.toml`, `tsconfig.json` or the test preload runs every test. Run the full `bun run ci` before every
|
||||||
|
push and check its exit code: it is the gate, `ci:quick` is not.
|
||||||
|
|
||||||
|
`tools/ci.ts` runs the steps as a task graph (independent steps at the same time, output prefixed with the step
|
||||||
|
name, the first failure stops the rest, a time-per-step summary at the end). The tests run as parallel shards
|
||||||
|
(CPU cores - 1, at most 8; `STOREBOUND_CI_SHARDS=n` overrides) balanced by the durations of the last run, kept in
|
||||||
|
`node_modules/.cache/storebound-ci/`; the browser suites get a shard each. So tests must be shard-safe: temp folders
|
||||||
|
from `mkdtempSync`, servers on port 0, no fixed sleeps (wait for the event), no state shared through files in the
|
||||||
|
checkout. The desktop Rust host (`cargo test`, `clippy`) runs only when `apps/desktop/src-tauri/` or
|
||||||
|
`tools/build-sidecars.ts` changed against `origin/main`; `CI=full bun run ci` always runs it (Forgejo CI does).
|
||||||
|
|
||||||
Try the screenshot and video pipeline on the demo game without any store account: see
|
Try the screenshot and video pipeline on the demo game without any store account: see
|
||||||
[examples/demo-game/README.md](examples/demo-game/README.md).
|
[examples/demo-game/README.md](examples/demo-game/README.md).
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -48,7 +48,8 @@ describe('webview rules', () => {
|
||||||
});
|
});
|
||||||
test('built bundle (when present) has no fixture mode and no dev switches', () => {
|
test('built bundle (when present) has no fixture mode and no dev switches', () => {
|
||||||
const dist = join(ROOT, 'dist', 'assets');
|
const dist = join(ROOT, 'dist', 'assets');
|
||||||
if (!existsSync(dist)) return;
|
// tools/ci.ts runs this file again as "desktop bundle check" after the build; its test shards skip it here
|
||||||
|
if (!existsSync(dist) || process.env.STOREBOUND_SKIP_DIST_CHECK === '1') return;
|
||||||
for (const n of readdirSync(dist)) {
|
for (const n of readdirSync(dist)) {
|
||||||
const s = readFileSync(join(dist, n), 'utf8');
|
const s = readFileSync(join(dist, n), 'utf8');
|
||||||
expect(s, n).not.toContain('storebound-dev-fixtures');
|
expect(s, n).not.toContain('storebound-dev-fixtures');
|
||||||
|
|
|
||||||
|
|
@ -59,6 +59,7 @@
|
||||||
"adapters": "bun tools/gen-adapters.ts",
|
"adapters": "bun tools/gen-adapters.ts",
|
||||||
"scrub": "bun tools/scrub.ts",
|
"scrub": "bun tools/scrub.ts",
|
||||||
"ci": "bun tools/ci.ts",
|
"ci": "bun tools/ci.ts",
|
||||||
|
"ci:quick": "bun tools/ci.ts --quick",
|
||||||
"check": "bun run typecheck && bun test"
|
"check": "bun run typecheck && bun test"
|
||||||
},
|
},
|
||||||
"peerDependencies": {
|
"peerDependencies": {
|
||||||
|
|
|
||||||
|
|
@ -79,7 +79,9 @@ const server = Bun.serve({
|
||||||
});
|
});
|
||||||
afterAll(() => server.stop(true));
|
afterAll(() => server.stop(true));
|
||||||
|
|
||||||
const canLaunch = spawnSync('bun', ['-e', `const { launch } = await import(${JSON.stringify(join(import.meta.dir, '..', 'cli', 'shots', 'browser.ts'))}); await (await launch(${JSON.stringify(join(import.meta.dir, '..'))}, { headless: true })).close();`], { timeout: 60_000 }).status === 0;
|
// launch only: exiting closes the browser's pipe and it quits on its own (a graceful close of Google Chrome, the
|
||||||
|
// fallback without playwright's Chromium, can take 30 s)
|
||||||
|
const canLaunch = spawnSync('bun', ['-e', `const { launch } = await import(${JSON.stringify(join(import.meta.dir, '..', 'cli', 'shots', 'browser.ts'))}); await launch(${JSON.stringify(join(import.meta.dir, '..'))}, { headless: true }); process.exit(0);`], { timeout: 60_000 }).status === 0;
|
||||||
if (!canLaunch) console.warn('capture-web browser tests skipped: no Chromium could launch (bunx playwright install chromium)');
|
if (!canLaunch) console.warn('capture-web browser tests skipped: no Chromium could launch (bunx playwright install chromium)');
|
||||||
|
|
||||||
/** A temp project with this config; cli() runs `storebound shots capture <args>` in it. */
|
/** A temp project with this config; cli() runs `storebound shots capture <args>` in it. */
|
||||||
|
|
|
||||||
62
tests/ci-plan.test.ts
Normal file
62
tests/ci-plan.test.ts
Normal file
|
|
@ -0,0 +1,62 @@
|
||||||
|
import { describe, expect, test } from 'bun:test';
|
||||||
|
import { balance, depsOf, GLOBAL, HEAVY, junitTimes, listTests, relatedTests, rustNeeded, shardCount } from '../tools/ci-plan';
|
||||||
|
|
||||||
|
describe('tools/ci-plan: test shards', () => {
|
||||||
|
test('shard count: cores - 1, between 1 and 8, env override', () => {
|
||||||
|
expect(shardCount({}, 10)).toBe(8);
|
||||||
|
expect(shardCount({}, 4)).toBe(3);
|
||||||
|
expect(shardCount({}, 1)).toBe(1);
|
||||||
|
expect(shardCount({ STOREBOUND_CI_SHARDS: '3' }, 10)).toBe(3);
|
||||||
|
expect(shardCount({ STOREBOUND_CI_SHARDS: 'x' }, 10)).toBe(8);
|
||||||
|
});
|
||||||
|
test('balance: every file once, longest first, heavy suites never share a shard', () => {
|
||||||
|
const cost: Record<string, number> = { h1: 10, h2: 9, a: 5, b: 4, c: 3, d: 2, e: 1 };
|
||||||
|
const shards = balance(Object.keys(cost), 3, f => cost[f]!, ['h1', 'h2']);
|
||||||
|
expect(shards.flat().sort()).toEqual(Object.keys(cost).sort());
|
||||||
|
for (const s of shards) expect(s.filter(f => f.startsWith('h')).length).toBeLessThanOrEqual(1);
|
||||||
|
const loads = shards.map(s => s.reduce((n, f) => n + cost[f]!, 0));
|
||||||
|
expect(Math.max(...loads) - Math.min(...loads)).toBeLessThanOrEqual(cost.a!);
|
||||||
|
});
|
||||||
|
test('balance: more heavy suites than shards still places each once; empty shards dropped', () => {
|
||||||
|
expect(balance(['h1', 'h2', 'h3'], 2, () => 1, ['h1', 'h2', 'h3']).flat().sort()).toEqual(['h1', 'h2', 'h3']);
|
||||||
|
expect(balance(['a'], 8, () => 1)).toEqual([['a']]);
|
||||||
|
});
|
||||||
|
test('JUnit report: per-file seconds from the top-level suites only', () => {
|
||||||
|
const xml = `<testsuites><testsuite name="tests/a.test.ts" file="tests/a.test.ts" tests="2" time="1.5" hostname="h">
|
||||||
|
<testsuite name="group" file="tests/a.test.ts" time="0.2"></testsuite></testsuite>
|
||||||
|
<testsuite name="tests/b.test.ts" file="tests/b.test.ts" time="0.25"></testsuite></testsuites>`;
|
||||||
|
expect(junitTimes(xml)).toEqual({ 'tests/a.test.ts': 1.5, 'tests/b.test.ts': 0.25 });
|
||||||
|
});
|
||||||
|
test('every heavy suite exists and every test file is found', () => {
|
||||||
|
const all = listTests();
|
||||||
|
for (const h of HEAVY) expect(all).toContain(h);
|
||||||
|
expect(all).toContain('tests/ci-plan.test.ts');
|
||||||
|
expect(all).toContain('apps/desktop/tests/webview.test.ts');
|
||||||
|
expect(all.some(f => f.includes('node_modules') || f.includes('/fixtures/'))).toBe(false);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describe('tools/ci-plan: related tests (ci:quick)', () => {
|
||||||
|
const all = listTests();
|
||||||
|
test('imports, transitive imports and spawned CLI paths', () => {
|
||||||
|
expect(depsOf('tests/ci-plan.test.ts')).toEqual(['tools/ci-plan.ts']);
|
||||||
|
expect(relatedTests(['tools/ci-plan.ts'], all)).toEqual(['tests/ci-plan.test.ts']);
|
||||||
|
// a CLI library change reaches the tests that import it directly and through other modules
|
||||||
|
const upsert = relatedTests(['cli/lib/upsert.ts'], all);
|
||||||
|
expect(upsert).toContain('tests/idempotency.test.ts');
|
||||||
|
expect(upsert.length).toBeGreaterThan(3);
|
||||||
|
});
|
||||||
|
test('a changed test runs itself; a global file runs everything; docs run nothing', () => {
|
||||||
|
expect(relatedTests(['tests/pure.test.ts'], all)).toEqual(['tests/pure.test.ts']);
|
||||||
|
expect(relatedTests(['package.json'], all)).toEqual(all);
|
||||||
|
expect(GLOBAL.test('bunfig.toml')).toBe(true);
|
||||||
|
expect(relatedTests([['docs/nobody', 'reads.md'].join('-')], all)).toEqual([]);
|
||||||
|
});
|
||||||
|
test('Rust steps only for the Tauri crate or the sidecar build, or with CI=full / unknown diff', () => {
|
||||||
|
expect(rustNeeded(['cli/index.ts'], {})).toBe(false);
|
||||||
|
expect(rustNeeded(['apps/desktop/src-tauri/src/main.rs'], {})).toBe(true);
|
||||||
|
expect(rustNeeded(['tools/build-sidecars.ts'], {})).toBe(true);
|
||||||
|
expect(rustNeeded([], { CI: 'full' })).toBe(true);
|
||||||
|
expect(rustNeeded(null, {})).toBe(true);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
@ -34,6 +34,12 @@ function fakeRunner(reply: (argv: string[]) => { code?: number; out?: string; da
|
||||||
return { calls, run };
|
return { calls, run };
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/** Resolves once `cond` holds (the child was started), instead of a fixed sleep. */
|
||||||
|
async function until(cond: () => boolean, ms = 5000) {
|
||||||
|
const end = Date.now() + ms;
|
||||||
|
while (!cond()) { if (Date.now() > end) throw new Error('until: timed out'); await new Promise(r => setImmediate(r)); }
|
||||||
|
}
|
||||||
|
|
||||||
async function session(reply?: Parameters<typeof fakeRunner>[0], clock: () => number = Date.now) {
|
async function session(reply?: Parameters<typeof fakeRunner>[0], clock: () => number = Date.now) {
|
||||||
const sent: J[] = [];
|
const sent: J[] = [];
|
||||||
const f = fakeRunner(reply);
|
const f = fakeRunner(reply);
|
||||||
|
|
@ -195,10 +201,10 @@ describe('serve --stdio: session', () => {
|
||||||
} finally { rmSync(dir, { recursive: true, force: true }); }
|
} finally { rmSync(dir, { recursive: true, force: true }); }
|
||||||
});
|
});
|
||||||
test('cancellation answers -32800 and kills the child', async () => {
|
test('cancellation answers -32800 and kills the child', async () => {
|
||||||
const { s, call, nextId } = await session(() => ({ wait: new Promise(() => {}) }));
|
const { s, call, calls, nextId } = await session(() => ({ wait: new Promise(() => {}) }));
|
||||||
const id = nextId();
|
const id = nextId();
|
||||||
const pending = call('validate', { project: DEMO });
|
const pending = call('validate', { project: DEMO });
|
||||||
await Bun.sleep(5);
|
await until(() => calls.length > 0);
|
||||||
await s.handle({ jsonrpc: '2.0', method: '$/cancelRequest', params: { id } });
|
await s.handle({ jsonrpc: '2.0', method: '$/cancelRequest', params: { id } });
|
||||||
expect(errOf(await pending).code).toBe(ERR.cancelled);
|
expect(errOf(await pending).code).toBe(ERR.cancelled);
|
||||||
});
|
});
|
||||||
|
|
@ -223,7 +229,7 @@ describe('serve --stdio: session', () => {
|
||||||
// real run: waits for the owner (no timeout), the token answers the command's own question, a cancel stops it
|
// real run: waits for the owner (no timeout), the token answers the command's own question, a cancel stops it
|
||||||
const id = nextId();
|
const id = nextId();
|
||||||
const pending = call('console.fill', { ...p, dryRun: false, confirmToken: mint(SECRET, 'console.fill', p, { now }) });
|
const pending = call('console.fill', { ...p, dryRun: false, confirmToken: mint(SECRET, 'console.fill', p, { now }) });
|
||||||
await Bun.sleep(5);
|
await until(() => calls.length > n);
|
||||||
expect(calls.length).toBe(n + 1);
|
expect(calls.length).toBe(n + 1);
|
||||||
expect(calls.at(-1)!.argv).toEqual(['console', 'fill', 'apple-age-rating', '--json']);
|
expect(calls.at(-1)!.argv).toEqual(['console', 'fill', 'apple-age-rating', '--json']);
|
||||||
expect(calls.at(-1)!.o.env?.STOREBOUND_OWNER_CONFIRMED).toBe('1');
|
expect(calls.at(-1)!.o.env?.STOREBOUND_OWNER_CONFIRMED).toBe('1');
|
||||||
|
|
|
||||||
|
|
@ -16,7 +16,8 @@ try {
|
||||||
} catch (e) { skipWhy = (e as Error).message.split('\n')[0]!; }
|
} catch (e) { skipWhy = (e as Error).message.split('\n')[0]!; }
|
||||||
if (!browser) console.warn(`site overflow test skipped: no Chromium could launch (${skipWhy}). Run \`bunx playwright install chromium\`.`);
|
if (!browser) console.warn(`site overflow test skipped: no Chromium could launch (${skipWhy}). Run \`bunx playwright install chromium\`.`);
|
||||||
|
|
||||||
afterAll(async () => { await browser?.close(); });
|
// Google Chrome (the fallback when playwright's own Chromium is missing) can take 30 s to shut down
|
||||||
|
afterAll(async () => { await browser?.close(); }, 60_000);
|
||||||
|
|
||||||
describe.skipIf(!browser)('site: no horizontal overflow', () => {
|
describe.skipIf(!browser)('site: no horizontal overflow', () => {
|
||||||
for (const scheme of ['light', 'dark'] as const) {
|
for (const scheme of ['light', 'dark'] as const) {
|
||||||
|
|
|
||||||
193
tools/ci-plan.ts
Normal file
193
tools/ci-plan.ts
Normal file
|
|
@ -0,0 +1,193 @@
|
||||||
|
// The planning half of tools/ci.ts: which test files exist, how they are split into shards, which tests a change
|
||||||
|
// touches (ci:quick) and whether the desktop Rust host needs to run. Pure functions plus small file / git reads;
|
||||||
|
// tests/ci-plan.test.ts covers them.
|
||||||
|
import { mkdirSync, readdirSync, readFileSync, statSync, writeFileSync } from 'node:fs';
|
||||||
|
import { cpus } from 'node:os';
|
||||||
|
import { dirname, join, normalize, relative } from 'node:path';
|
||||||
|
|
||||||
|
export const ROOT = join(import.meta.dir, '..');
|
||||||
|
/** Recorded per-file test durations (seconds), written after every run. Gitignored (node_modules/). */
|
||||||
|
export const TIMINGS = join(ROOT, 'node_modules/.cache/storebound-ci/test-timings.json');
|
||||||
|
|
||||||
|
/** Suites that drive a real Chromium (or a heavy child process): each gets a shard of its own, never another heavy
|
||||||
|
* suite next to it, so they don't fight for CPU and time out. */
|
||||||
|
export const HEAVY = [
|
||||||
|
'tests/console-fill.test.ts', 'tests/site-overflow.test.ts', 'tests/local-fonts.test.ts', 'tests/glyphs.test.ts',
|
||||||
|
'tests/capture-web.test.ts', 'tests/console-session.test.ts',
|
||||||
|
];
|
||||||
|
|
||||||
|
/** Changes that can affect every test: run the whole suite. */
|
||||||
|
export const GLOBAL = /^(package\.json|bun\.lock|bunfig\.toml|tsconfig\.json|tests\/helpers\/preload\.ts)$/;
|
||||||
|
|
||||||
|
const SKIP_DIRS = new Set(['node_modules', '.git', '.scratch', 'dist', 'target', 'fixtures', '.claude']);
|
||||||
|
|
||||||
|
/** Every *.test.ts / *.test.tsx `bun test` would pick up, relative to the root, sorted. */
|
||||||
|
export function listTests(root = ROOT): string[] {
|
||||||
|
const out: string[] = [];
|
||||||
|
const walk = (dir: string) => {
|
||||||
|
for (const e of readdirSync(join(root, dir), { withFileTypes: true })) {
|
||||||
|
const p = dir ? `${dir}/${e.name}` : e.name;
|
||||||
|
if (e.isDirectory()) { if (!SKIP_DIRS.has(e.name) && !e.name.startsWith('.')) walk(p); }
|
||||||
|
else if (/\.test\.tsx?$/.test(e.name)) out.push(p);
|
||||||
|
}
|
||||||
|
};
|
||||||
|
walk('');
|
||||||
|
return out.sort();
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Shard count: CPU cores - 1, at least 1, at most 8 (STOREBOUND_CI_SHARDS overrides). */
|
||||||
|
export function shardCount(env = process.env, cores = cpus().length): number {
|
||||||
|
const n = Number(env.STOREBOUND_CI_SHARDS);
|
||||||
|
if (Number.isInteger(n) && n > 0) return n;
|
||||||
|
return Math.max(1, Math.min(8, cores - 1));
|
||||||
|
}
|
||||||
|
|
||||||
|
export function readTimings(file = TIMINGS): Record<string, number> {
|
||||||
|
try { return JSON.parse(readFileSync(file, 'utf8')) as Record<string, number>; } catch { return {}; }
|
||||||
|
}
|
||||||
|
|
||||||
|
export function writeTimings(t: Record<string, number>, file = TIMINGS): void {
|
||||||
|
try { mkdirSync(dirname(file), { recursive: true }); writeFileSync(file, `${JSON.stringify(t, null, 1)}\n`); } catch { /* cache only */ }
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Per-file seconds from bun's JUnit report (the top-level <testsuite> of each file). */
|
||||||
|
export function junitTimes(xml: string): Record<string, number> {
|
||||||
|
const out: Record<string, number> = {};
|
||||||
|
for (const m of xml.matchAll(/<testsuite name="([^"]+)" file="([^"]+)"[^>]*?time="([\d.]+)"/g)) {
|
||||||
|
if (m[1] === m[2]) out[m[2]!] = Number(m[3]);
|
||||||
|
}
|
||||||
|
return out;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Estimated cost of a file: its recorded time, else its size (about 1 s per 40 KB, 0.2 s minimum). */
|
||||||
|
export function cost(file: string, timings: Record<string, number>, size: (f: string) => number): number {
|
||||||
|
return timings[file] ?? Math.max(0.2, size(file) / 40_000);
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Longest-processing-time-first split into at most n shards. Heavy files each open a shard of their own and never
|
||||||
|
* share it with another heavy file; light files fill the least loaded shard. Empty shards are dropped.
|
||||||
|
*/
|
||||||
|
export function balance(files: string[], n: number, costOf: (f: string) => number, heavy: string[] = HEAVY): string[][] {
|
||||||
|
const shards = Array.from({ length: Math.max(1, n) }, () => ({ files: [] as string[], load: 0, heavy: false }));
|
||||||
|
const sorted = [...files].sort((a, b) => costOf(b) - costOf(a) || a.localeCompare(b));
|
||||||
|
const isHeavy = (f: string) => heavy.includes(f);
|
||||||
|
for (const f of sorted.filter(isHeavy)) {
|
||||||
|
const free = shards.filter(s => !s.heavy);
|
||||||
|
const s = (free.length ? free : shards).reduce((a, b) => (b.load < a.load ? b : a));
|
||||||
|
s.files.push(f); s.load += costOf(f); s.heavy = true;
|
||||||
|
}
|
||||||
|
for (const f of sorted.filter(f => !isHeavy(f))) {
|
||||||
|
const s = shards.reduce((a, b) => (b.load < a.load ? b : a));
|
||||||
|
s.files.push(f); s.load += costOf(f);
|
||||||
|
}
|
||||||
|
return shards.filter(s => s.files.length).map(s => s.files);
|
||||||
|
}
|
||||||
|
|
||||||
|
export function planShards(files: string[], n = shardCount(), root = ROOT): string[][] {
|
||||||
|
const t = readTimings();
|
||||||
|
return balance(files, n, f => cost(f, t, p => { try { return statSync(join(root, p)).size; } catch { return 0; } }));
|
||||||
|
}
|
||||||
|
|
||||||
|
// ---- which tests a change touches ----
|
||||||
|
|
||||||
|
const IMPORT_RE = /(?:\bfrom\s*|\bimport\s*\(\s*|\brequire\s*\(\s*|^\s*import\s+)['"](\.{1,2}\/[^'"]+)['"]/gm;
|
||||||
|
const JOIN_RE = /join\(\s*(?:import\.meta\.dir|__dirname|ROOT|root|REPO)\s*,((?:\s*'[^']*'\s*,?)+)\)/g;
|
||||||
|
|
||||||
|
function resolveFile(p: string): string | null {
|
||||||
|
for (const c of [p, `${p}.ts`, `${p}.tsx`, join(p, 'index.ts'), join(p, 'index.tsx')]) {
|
||||||
|
try { if (statSync(c).isFile()) return c; } catch { /* next */ }
|
||||||
|
}
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Repo files a source file depends on: relative imports, plus `join(import.meta.dir, '..', 'cli', 'index.ts')`
|
||||||
|
* style paths (a test that spawns the CLI depends on it). */
|
||||||
|
export function depsOf(file: string, root = ROOT, src = readFileSync(join(root, file), 'utf8')): string[] {
|
||||||
|
const dir = dirname(join(root, file));
|
||||||
|
const out = new Set<string>();
|
||||||
|
for (const m of src.matchAll(IMPORT_RE)) {
|
||||||
|
const r = resolveFile(normalize(join(dir, m[1]!)));
|
||||||
|
if (r) out.add(relative(root, r));
|
||||||
|
}
|
||||||
|
for (const m of src.matchAll(JOIN_RE)) {
|
||||||
|
const parts = [...m[1]!.matchAll(/'([^']*)'/g)].map(x => x[1]!);
|
||||||
|
for (const base of [dir, root]) {
|
||||||
|
const r = resolveFile(normalize(join(base, ...parts)));
|
||||||
|
if (r && r.startsWith(root)) { out.add(relative(root, r)); break; }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return [...out];
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Tests related to the changed files: a changed test itself, every test that imports a changed file (transitively),
|
||||||
|
* and every test that names a changed non-code file (fixture, page, skill) by its file name or its fixture folder.
|
||||||
|
* A change to a GLOBAL file returns every test.
|
||||||
|
*/
|
||||||
|
export function relatedTests(changed: string[], tests: string[], root = ROOT): string[] {
|
||||||
|
if (changed.some(f => GLOBAL.test(f))) return tests;
|
||||||
|
const changedSet = new Set(changed);
|
||||||
|
const memo = new Map<string, Set<string>>();
|
||||||
|
const direct = new Map<string, string[]>();
|
||||||
|
const depsCached = (f: string) => {
|
||||||
|
let d = direct.get(f);
|
||||||
|
if (!d) { try { d = depsOf(f, root); } catch { d = []; } direct.set(f, d); }
|
||||||
|
return d;
|
||||||
|
};
|
||||||
|
const closure = (f: string): Set<string> => {
|
||||||
|
const hit = memo.get(f);
|
||||||
|
if (hit) return hit;
|
||||||
|
const seen = new Set<string>([f]);
|
||||||
|
memo.set(f, seen);
|
||||||
|
const stack = [f];
|
||||||
|
while (stack.length) {
|
||||||
|
const cur = stack.pop()!;
|
||||||
|
if (!/\.(ts|tsx)$/.test(cur)) continue;
|
||||||
|
for (const d of depsCached(cur)) if (!seen.has(d)) { seen.add(d); stack.push(d); }
|
||||||
|
}
|
||||||
|
return seen;
|
||||||
|
};
|
||||||
|
const names = changed.filter(f => !/\.(ts|tsx)$/.test(f) || f.includes('/fixtures/')).flatMap(f => {
|
||||||
|
const base = f.split('/').pop()!;
|
||||||
|
const fx = /(?:^|\/)fixtures\/([^/]+)/.exec(f)?.[1];
|
||||||
|
return [base, ...(fx ? [`fixtures/${fx}`, `'${fx}'`] : [])].filter(s => s.length > 3);
|
||||||
|
});
|
||||||
|
return tests.filter(t => {
|
||||||
|
if (changedSet.has(t)) return true;
|
||||||
|
const deps = closure(t);
|
||||||
|
if (changed.some(c => deps.has(c))) return true;
|
||||||
|
if (!names.length) return false;
|
||||||
|
const srcs = [...deps].filter(d => d.startsWith('tests/') || d.startsWith('apps/desktop/tests/')).map(d => {
|
||||||
|
try { return readFileSync(join(root, d), 'utf8'); } catch { return ''; }
|
||||||
|
}).join('\n');
|
||||||
|
return names.some(n => srcs.includes(n));
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
// ---- git ----
|
||||||
|
|
||||||
|
function git(args: string[], root = ROOT): string | null {
|
||||||
|
const r = Bun.spawnSync(['git', ...args], { cwd: root, stdout: 'pipe', stderr: 'ignore' });
|
||||||
|
return r.exitCode === 0 ? r.stdout.toString() : null;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Files changed against origin/main (committed since the merge base, staged, unstaged and untracked), or null
|
||||||
|
* when that cannot be told (no git, no origin/main). */
|
||||||
|
export function changedFiles(base = process.env.STOREBOUND_CI_BASE ?? 'origin/main', root = ROOT): string[] | null {
|
||||||
|
const mb = git(['merge-base', base, 'HEAD'], root)?.trim();
|
||||||
|
if (!mb) return null;
|
||||||
|
const lists = [git(['diff', '--name-only', mb], root), git(['ls-files', '--others', '--exclude-standard'], root)];
|
||||||
|
if (lists.some(l => l === null)) return null;
|
||||||
|
return [...new Set(lists.join('\n').split('\n').map(s => s.trim()).filter(Boolean))].sort();
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Desktop Rust host inputs: the crate, and the sidecar build its build script needs (cargo test / clippy never run
|
||||||
|
* the sidecar itself, so a CLI change does not need them; tools/ci.ts rebuilds the sidecar on CLI changes). */
|
||||||
|
export const RUST_INPUTS = /^(apps\/desktop\/src-tauri\/|tools\/build-sidecars\.ts$)/;
|
||||||
|
|
||||||
|
/** Run cargo test / clippy? Always with CI=full or when the change set is unknown; else only when a Rust input changed. */
|
||||||
|
export function rustNeeded(changed: string[] | null, env = process.env): boolean {
|
||||||
|
if (env.CI === 'full' || changed === null) return true;
|
||||||
|
return changed.some(f => RUST_INPUTS.test(f));
|
||||||
|
}
|
||||||
203
tools/ci.ts
203
tools/ci.ts
|
|
@ -1,46 +1,173 @@
|
||||||
#!/usr/bin/env bun
|
#!/usr/bin/env bun
|
||||||
// Everything CI runs, in order; stops at the first failure. bun run ci
|
// Everything CI runs, as a small task graph: independent steps run at the same time, output is streamed with a
|
||||||
// `bun test` covers the CLI and the desktop webview (apps/desktop/tests: model, i18n, theme contrast, webview rules),
|
// [step] prefix, the first failure stops everything, and a summary prints the time per step.
|
||||||
// the desktop boot check (tools/desktop-boot-check.ts: the built window in headless Chromium, needs playwright's
|
// bun run ci full gate before pushing (every check, every test)
|
||||||
// browser or Google Chrome), the site overflow check (tests/site-overflow.test.ts: no sideways scroll at 320-1440 px,
|
// bun run ci:quick inner loop: typecheck + the tests related to the change (git diff against origin/main,
|
||||||
// light and dark; skips without Chromium), and the no-hand-written-SVG guard (tests/no-handmade-svg.test.ts: visx charts, Phosphor icons, simple-icons store marks, Mermaid diagrams).
|
// mapped through imports) + scrub
|
||||||
// The desktop Rust host (cargo test, clippy
|
// The tests run as shards: N = CPU cores - 1 (max 8; STOREBOUND_CI_SHARDS=n overrides) parallel `bun test <files>`
|
||||||
// with warnings as errors) runs when cargo is installed; set STOREBOUND_CI_RUST=0 to skip it on a machine without Rust.
|
// processes, balanced by the durations recorded after the last run (node_modules/.cache/storebound-ci/, falling back
|
||||||
|
// to file size). Browser suites (tools/ci-plan.ts HEAVY) each get a shard of their own.
|
||||||
|
// `bun test` covers the CLI and the desktop webview (apps/desktop/tests), the site overflow check and the
|
||||||
|
// no-hand-written-SVG guard; the desktop boot check (tools/desktop-boot-check.ts: the built window in headless
|
||||||
|
// Chromium, needs playwright's browser or Google Chrome) runs after the desktop build.
|
||||||
|
// The desktop Rust host (cargo test, clippy with warnings as errors) runs when cargo is installed and
|
||||||
|
// apps/desktop/src-tauri or the sidecar build changed against origin/main, or always with CI=full;
|
||||||
|
// STOREBOUND_CI_RUST=0 skips it on a machine without Rust.
|
||||||
|
import { existsSync, mkdtempSync, readFileSync, rmSync } from 'node:fs';
|
||||||
|
import { homedir, tmpdir } from 'node:os';
|
||||||
import { join } from 'node:path';
|
import { join } from 'node:path';
|
||||||
|
import { hostTarget, OUT_DIR, outName } from './build-sidecars';
|
||||||
|
import { changedFiles, HEAVY, junitTimes, listTests, planShards, readTimings, relatedTests, ROOT, rustNeeded, writeTimings } from './ci-plan';
|
||||||
|
|
||||||
const root = new URL('..', import.meta.url).pathname;
|
interface Task { name: string; cmd: string[]; cwd?: string; env?: Record<string, string>; after?: string[]; junit?: string }
|
||||||
const tauri = join(root, 'apps/desktop/src-tauri');
|
|
||||||
const cargo = process.env.STOREBOUND_CI_RUST === '0' ? null : Bun.which('cargo');
|
const quick = process.argv.includes('--quick');
|
||||||
const steps: [string, string[], string?][] = [
|
const tauri = join(ROOT, 'apps/desktop/src-tauri');
|
||||||
['adapters', ['bun', 'tools/gen-adapters.ts', '--check']],
|
const cache = join(ROOT, 'node_modules/.cache/storebound-ci');
|
||||||
|
const changed = changedFiles();
|
||||||
|
const tests = listTests();
|
||||||
|
const touches = (re: RegExp) => changed === null || changed.some(f => re.test(f));
|
||||||
|
const tasks: Task[] = [];
|
||||||
|
const add = (t: Task) => tasks.push(t);
|
||||||
|
const tsc = (name: string, project: string) => add({ name, cmd: ['bunx', 'tsc', '--noEmit', '--incremental', '-p', project,
|
||||||
|
'--tsBuildInfoFile', join(cache, `${name.replace(/\W+/g, '-')}.tsbuildinfo`)] });
|
||||||
|
const junitDir = mkdtempSync(join(tmpdir(), 'sb-ci-'));
|
||||||
|
|
||||||
|
function addTests(files: string[]) {
|
||||||
|
planShards(files).forEach((shard, i, all) => {
|
||||||
|
const junit = join(junitDir, `shard-${i + 1}.xml`);
|
||||||
|
// the built-bundle check in apps/desktop/tests/webview.test.ts runs as "desktop bundle check" once the build is
|
||||||
|
// done; here it would read dist/ while vite rewrites it
|
||||||
|
// browser suites share the machine with the other steps: a browser start or close can pass bun's 5 s default
|
||||||
|
const timeout = shard.some(f => HEAVY.includes(f)) ? ['--timeout=30000'] : [];
|
||||||
|
add({ name: `tests ${i + 1}/${all.length}`, cmd: ['bun', 'test', ...timeout, '--reporter=junit', `--reporter-outfile=${junit}`, ...shard.map(f => `./${f}`)],
|
||||||
|
env: { STOREBOUND_SKIP_DIST_CHECK: '1' }, junit });
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
if (quick) {
|
||||||
|
const related = changed === null ? tests : relatedTests(changed, tests);
|
||||||
|
console.log(`ci:quick: ${changed === null ? 'no origin/main to diff against: every test' : `${changed.length} changed file(s), ${related.length} related test file(s)`}`);
|
||||||
|
add({ name: 'adapters', cmd: ['bun', 'tools/gen-adapters.ts', '--check'] });
|
||||||
|
tsc('typecheck', '.');
|
||||||
|
if (touches(/^apps\/desktop\//)) tsc('desktop typecheck', 'apps/desktop');
|
||||||
|
if (related.length) addTests(related);
|
||||||
|
add({ name: 'scrub', cmd: ['bun', 'tools/scrub.ts'] });
|
||||||
|
} else {
|
||||||
|
const cargo = process.env.STOREBOUND_CI_RUST === '0' ? null : Bun.which('cargo');
|
||||||
|
const rust = cargo && rustNeeded(changed);
|
||||||
|
if (!cargo) console.log('ci: cargo not found (or STOREBOUND_CI_RUST=0): desktop Rust tests and clippy skipped');
|
||||||
|
else if (!rust) console.log('ci: apps/desktop/src-tauri and the sidecar build unchanged against origin/main: desktop Rust tests and clippy skipped (CI=full runs them)');
|
||||||
|
add({ name: 'adapters', cmd: ['bun', 'tools/gen-adapters.ts', '--check'] });
|
||||||
// the App Store Connect / Google Play operation tables, the wrapped-by scan and docs/api-coverage.md match specs/ and cli/
|
// the App Store Connect / Google Play operation tables, the wrapped-by scan and docs/api-coverage.md match specs/ and cli/
|
||||||
['api tables', ['bun', 'tools/gen-api.ts', '--check']],
|
add({ name: 'api tables', cmd: ['bun', 'tools/gen-api.ts', '--check'] });
|
||||||
// one desktop version: package.json desktopVersion = apps/desktop/package.json = tauri.conf.json = Cargo.toml / .lock
|
// one desktop version: package.json desktopVersion = apps/desktop/package.json = tauri.conf.json = Cargo.toml / .lock
|
||||||
['desktop version', ['bun', 'tools/desktop-version.ts']],
|
add({ name: 'desktop version', cmd: ['bun', 'tools/desktop-version.ts'] });
|
||||||
['tests', ['bun', 'test']],
|
add({ name: 'desktop i18n', cmd: ['bun', 'tools/i18n-check.ts'] });
|
||||||
['typecheck', ['bunx', 'tsc', '--noEmit']],
|
tsc('typecheck', '.');
|
||||||
['desktop i18n', ['bun', 'tools/i18n-check.ts']],
|
tsc('desktop typecheck', 'apps/desktop');
|
||||||
['desktop typecheck', ['bunx', 'tsc', '--noEmit', '-p', 'apps/desktop']],
|
add({ name: 'scrub', cmd: ['bun', 'tools/scrub.ts'] });
|
||||||
['desktop build', ['bun', 'run', '--cwd', 'apps/desktop', 'build']],
|
addTests(tests);
|
||||||
// the release bundle must not carry the dev fixture mode (apps/desktop/tests/webview.test.ts checks dist/ when present)
|
// `vite build` only: the desktop package's own build script runs tsc first, which "desktop typecheck" already does
|
||||||
['desktop bundle check', ['bun', 'test', 'apps/desktop/tests/webview.test.ts']],
|
add({ name: 'desktop build', cmd: ['bunx', 'vite', 'build', '--logLevel', 'warn'], cwd: join(ROOT, 'apps/desktop') });
|
||||||
|
// the release bundle must not carry the dev fixture mode
|
||||||
|
add({ name: 'desktop bundle check', cmd: ['bun', 'test', './apps/desktop/tests/webview.test.ts'], after: ['desktop build'] });
|
||||||
// the built window boots (release CSP, Tauri's frozen Object.prototype, mocked host + real sidecar) on an empty, a
|
// the built window boots (release CSP, Tauri's frozen Object.prototype, mocked host + real sidecar) on an empty, a
|
||||||
// D1 (folders gone) and a full D3 saved state, through every screen: no uncaught error, never an empty window
|
// D1 (folders gone) and a full D3 saved state, through every screen: no uncaught error, never an empty window
|
||||||
['desktop boot check', ['bun', 'tools/desktop-boot-check.ts']],
|
add({ name: 'desktop boot check', cmd: ['bun', 'tools/desktop-boot-check.ts'], after: ['desktop build'] });
|
||||||
...(cargo ? [
|
// the CLI compiled into the desktop sidecar (bun build --compile, scanned for this machine's paths): rebuilt when
|
||||||
// tauri's build script needs the host sidecar binary (gitignored), so a clean checkout builds it first
|
// its inputs changed, with CI=full, or when missing; tauri's build script needs it for cargo test / clippy
|
||||||
['desktop sidecar (host)', ['bun', 'tools/build-sidecars.ts', '--host']],
|
const hostBin = join(OUT_DIR, outName(hostTarget()));
|
||||||
['desktop rust tests', [cargo, 'test', '--quiet'], tauri],
|
const sidecarBuild = process.env.CI === 'full' || !existsSync(hostBin) || touches(/^(cli\/|tools\/build-sidecars\.ts$|package\.json$|bun\.lock$)/);
|
||||||
['desktop clippy', [cargo, 'clippy', '--all-targets', '--quiet', '--', '-D', 'warnings'], tauri],
|
if (sidecarBuild) add({ name: 'desktop sidecar (host)', cmd: ['bun', 'tools/build-sidecars.ts', '--host'] });
|
||||||
] as [string, string[], string][] : []),
|
if (cargo && rust) {
|
||||||
['scrub', ['bun', 'tools/scrub.ts']],
|
const after = sidecarBuild ? ['desktop sidecar (host)'] : [];
|
||||||
];
|
// cargo locks the target dir, so test and clippy run one after the other
|
||||||
if (!cargo) console.log('ci: cargo not found (or STOREBOUND_CI_RUST=0): desktop Rust tests and clippy skipped');
|
add({ name: 'desktop rust tests', cmd: [cargo, 'test', '--quiet'], cwd: tauri, after });
|
||||||
for (const [name, cmd, cwd] of steps) {
|
add({ name: 'desktop clippy', cmd: [cargo, 'clippy', '--all-targets', '--quiet', '--', '-D', 'warnings'], cwd: tauri, after: ['desktop rust tests'] });
|
||||||
const t = Date.now();
|
}
|
||||||
console.log(`\n== ${name}: ${cmd.join(' ')}`);
|
|
||||||
const r = Bun.spawnSync(cmd, { cwd: cwd ?? root, stdout: 'inherit', stderr: 'inherit' });
|
|
||||||
if (r.exitCode !== 0) { console.error(`\nci: ${name} failed (exit ${r.exitCode})`); process.exit(1); }
|
|
||||||
console.log(`== ${name} ok (${((Date.now() - t) / 1000).toFixed(1)}s)`);
|
|
||||||
}
|
}
|
||||||
console.log('\nci: all green');
|
|
||||||
|
// The browser suites fall back to Google Chrome without playwright's own headless Chromium; Chrome's shutdown takes
|
||||||
|
// seconds to half a minute per browser, which makes those shards several times slower.
|
||||||
|
try {
|
||||||
|
const { registry } = await import('playwright-core/lib/server/registry/index' as string);
|
||||||
|
const exe = registry.findExecutable('chromium-headless-shell')?.executablePath('javascript') as string | undefined;
|
||||||
|
if (exe && !existsSync(exe)) console.log(`ci: playwright's headless Chromium is not installed (${exe.replace(homedir(), '~')}): the browser tests use Google Chrome, which is much slower to close. Install it once: bunx playwright install chromium-headless-shell`);
|
||||||
|
} catch { /* playwright internals moved: no hint */ }
|
||||||
|
|
||||||
|
// ---- runner ----
|
||||||
|
|
||||||
|
interface Run { task: Task; start: number; end?: number; code?: number; proc?: ReturnType<typeof Bun.spawn> }
|
||||||
|
const runs = new Map<string, Run>();
|
||||||
|
const t0 = Date.now();
|
||||||
|
const width = Math.max(...tasks.map(t => t.name.length));
|
||||||
|
const tag = (n: string) => `[${n.padEnd(width)}]`;
|
||||||
|
let failed: string | null = null;
|
||||||
|
|
||||||
|
async function pump(stream: ReadableStream<Uint8Array>, name: string, out: NodeJS.WriteStream) {
|
||||||
|
const dec = new TextDecoder();
|
||||||
|
let buf = '';
|
||||||
|
const reader = stream.getReader();
|
||||||
|
for (;;) {
|
||||||
|
const { done, value } = await reader.read();
|
||||||
|
if (done) break;
|
||||||
|
if (failed && failed !== name) continue; // a stopped step's last words are noise
|
||||||
|
buf += dec.decode(value, { stream: true });
|
||||||
|
const lines = buf.split('\n');
|
||||||
|
buf = lines.pop()!;
|
||||||
|
if (lines.length) out.write(lines.map(l => `${tag(name)} ${l}\n`).join(''));
|
||||||
|
}
|
||||||
|
if (buf && !(failed && failed !== name)) out.write(`${tag(name)} ${buf}\n`);
|
||||||
|
}
|
||||||
|
|
||||||
|
async function start(task: Task): Promise<void> {
|
||||||
|
const run: Run = { task, start: Date.now() };
|
||||||
|
runs.set(task.name, run);
|
||||||
|
console.log(`${tag(task.name)} $ ${task.cmd.join(' ').replace(junitDir, '$TMP')}`);
|
||||||
|
const proc = Bun.spawn(task.cmd, { cwd: task.cwd ?? ROOT, stdout: 'pipe', stderr: 'pipe', env: { ...process.env, FORCE_COLOR: '0', ...task.env } });
|
||||||
|
run.proc = proc;
|
||||||
|
await Promise.all([pump(proc.stdout, task.name, process.stdout), pump(proc.stderr, task.name, process.stderr), proc.exited]);
|
||||||
|
run.code = proc.exitCode ?? 1;
|
||||||
|
run.end = Date.now();
|
||||||
|
if (failed) return;
|
||||||
|
console.log(`${tag(task.name)} ${run.code === 0 ? 'ok' : `FAILED (exit ${run.code})`} in ${((run.end - run.start) / 1000).toFixed(1)}s`);
|
||||||
|
if (run.code !== 0) {
|
||||||
|
failed = task.name;
|
||||||
|
for (const r of runs.values()) if (r.end === undefined) r.proc?.kill('SIGKILL'); // bun test ignores SIGTERM until its files finish
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
const pending = new Set(tasks);
|
||||||
|
const active = new Set<Promise<void>>();
|
||||||
|
while ((pending.size || active.size) && !failed) {
|
||||||
|
for (const t of [...pending]) {
|
||||||
|
if ((t.after ?? []).every(a => runs.get(a)?.code === 0)) {
|
||||||
|
pending.delete(t);
|
||||||
|
const p = start(t).finally(() => active.delete(p));
|
||||||
|
active.add(p);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if (!active.size) { failed = 'scheduler (unmet dependency)'; break; }
|
||||||
|
await Promise.race(active);
|
||||||
|
}
|
||||||
|
if (failed) await Promise.allSettled([...active]);
|
||||||
|
|
||||||
|
// record per-file durations for the next split (only from shards that finished)
|
||||||
|
const timings = readTimings();
|
||||||
|
for (const r of runs.values()) {
|
||||||
|
if (!r.task.junit || r.end === undefined) continue;
|
||||||
|
try { Object.assign(timings, junitTimes(readFileSync(r.task.junit, 'utf8'))); } catch { /* no report */ }
|
||||||
|
}
|
||||||
|
writeTimings(timings);
|
||||||
|
rmSync(junitDir, { recursive: true, force: true });
|
||||||
|
|
||||||
|
console.log(`\n${quick ? 'ci:quick' : 'ci'} summary (wall ${((Date.now() - t0) / 1000).toFixed(1)}s)`);
|
||||||
|
for (const r of [...runs.values()].sort((a, b) => a.start - b.start)) {
|
||||||
|
const s = r.end === undefined ? 'killed' : r.code === 0 ? 'ok' : r.task.name === failed ? 'FAILED' : 'stopped';
|
||||||
|
const secs = (((r.end ?? Date.now()) - r.start) / 1000).toFixed(1).padStart(6);
|
||||||
|
console.log(` ${r.task.name.padEnd(width)} ${secs}s @${((r.start - t0) / 1000).toFixed(1).padStart(5)}s ${s}`);
|
||||||
|
}
|
||||||
|
for (const t of pending) console.log(` ${t.name.padEnd(width)} - not started`);
|
||||||
|
if (failed) { console.error(`\nci: ${failed} failed`); process.exit(1); }
|
||||||
|
console.log(`\n${quick ? 'ci:quick' : 'ci'}: all green`);
|
||||||
|
|
|
||||||
|
|
@ -84,6 +84,8 @@ export function generate(root: string): Map<string, string> {
|
||||||
'file below and follow it step by step.', '', coreBody.replace(/^/, ''), '', cli, '', '## Skills', '', indexPaths, '',
|
'file below and follow it step by step.', '', coreBody.replace(/^/, ''), '', cli, '', '## Skills', '', indexPaths, '',
|
||||||
'## Working on storebound itself', '',
|
'## Working on storebound itself', '',
|
||||||
'- Bun + TypeScript strict, zero runtime dependencies. `bun run ci` must pass (adapters fresh, tests, typecheck, scrub).',
|
'- Bun + TypeScript strict, zero runtime dependencies. `bun run ci` must pass (adapters fresh, tests, typecheck, scrub).',
|
||||||
|
'- Inner loop: `bun run ci:quick` (typecheck, the tests related to your change against origin/main, scrub; seconds).',
|
||||||
|
' Before every push: the full `bun run ci` (every check and test, in parallel shards), and check its exit code.',
|
||||||
'- Skills are the single source: edit `skills/*/SKILL.md` or `docs/agents-core.md`, then `bun tools/gen-adapters.ts`.',
|
'- Skills are the single source: edit `skills/*/SKILL.md` or `docs/agents-core.md`, then `bun tools/gen-adapters.ts`.',
|
||||||
'- Never commit credentials, account / team / issuer / key / app ids or personal data; placeholders use example.com.',
|
'- Never commit credentials, account / team / issuer / key / app ids or personal data; placeholders use example.com.',
|
||||||
''].join('\n'));
|
''].join('\n'));
|
||||||
|
|
|
||||||
Loading…
Add table
Add a link
Reference in a new issue