frameworkapp-review-framework · v41099:claude-chat-c1d32026-11-09All documents

JBNX app review framework (v1)

North star: Any JBNX web property gets a cheap, evidence-backed, ranked review that feeds a reusable context doc so the next agent re-measures nothing. Hard gates (never bend): claim before work; findings proven (positive and negative probes) before reported; probes rolled back; no secret values exposed; verify live before done; test-to-production promotion stays a chat approval Everything else is a default. Try a better or unusual way when it serves the north star; record what you changed and why in the change log. Change log: bottom of this page (who, when, why).

What this is: a repeatable, token-bounded procedure for reviewing any JBNX web property and returning ranked, evidence-backed improvements. Written 2026-08-12 from the NodeDough review. Follow it instead of improvising — improvised reviews re-derive the same facts every time and cost 5–10× the tokens. Default: follow it; use a cheaper or sharper method when it serves the review and log why.

Contract: every finding must carry a measured number or a quoted line. "Consider adding caching" is not a finding. "cache-control: no-cache on a 344 KB uncompressed app.js" is.

Phase 0 — Boot (default: do not skip, ~1.5k tokens)

GET  https://hi.jbnx.io/core                          # directive core (~4 KB); read once per version, skip /json (~69 KB)
#    first reply line before tool work: "Jev picked: skills …; frameworks …"
POST /api/1099/claim   {actor, slug, note}            # never work without a claim
GET  /api/1099/brief?actor=…&slug=…                   # BLUF + traps + last handover
GET  /framework/<slug>-context                        # the app's own context doc, if one exists

If a <slug>-context doc exists, read it and skip every probe it already answers. That doc is the entire point of this framework. If one does not exist, you are writing it at the end (Phase 5).

Phase 1 — Resolve identity before touching anything

Never guess which database belongs to which app. From the ops runbook or railway list-projects → list-services → get-service-config, pin down: repo, Railway project/service/environment IDs, Supabase ref, custom domains, and the source branch.

Check the source branch every time. A gated project whose production service still builds from main is not gated, whatever the directive says. This has been wrong before.

Phase 2 — Measure the front end (one bash block, cheap)

curl -sI https://<domain>                                  # headers: CSP, HSTS, XFO, nosniff
curl -sI -H 'Accept-Encoding: br,gzip' https://<domain>     # content-encoding present?
curl -sI https://<domain>/static/app.js | grep -i cache     # cache-control vs ?v= fingerprints
curl -s  https://<domain>/config.js                         # which key class ships to browser
for p in /robots.txt /sitemap.xml /healthz /legal/terms /admin; do
  curl -s -o /dev/null -w "$p %{http_code} %{size_download}\n" https://<domain>$p; done

Bundle path varies: find it first (curl -s https://<domain>/ | grep -o 'src="[^"]*\.js"') instead of assuming /static/app.js. Then download the bundle to disk and grep it — never into context:

curl -s https://<domain>/static/app.js -o /tmp/app.js; wc -c /tmp/app.js
for k in stripe checkout subscription trial export csv notification reminder \
         service.worker offline sentry analytics aria-live TODO; do
  printf '%-16s %s\n' "$k" "$(grep -ioc "$k" /tmp/app.js)"; done

A zero count is a finding. notification 0 + reminder 0 on a product whose value prop is timing is a retention hole, not a missing nice-to-have.

Phase 3 — Measure the backend

Advisors first, in a subagent (default). These payloads run to 140 K+ characters and will blow your context. Spawn a subagent with an explicit return contract: counts by rule and level, full affected-object lists, and verbatim detail for the top findings. Then diff against the app's recorded baseline rather than reading it whole again next time.

Then one reality-check SQL — scale and business state in a single round trip:

select (select count(*) from auth.users) users,
       (select count(*) from auth.users where last_sign_in_at > now()-interval '7 days') active_7d,
       (select count(*) from public.subscriptions) subs,
       (select count(*) from public.stripe_events) stripe_events,
       (select count(*) from public.analytics_events) analytics,
       (select count(*) from public.app_errors) errors;

Counts are context, not findings: zero users or subs on a pre-launch app is expected. The finding is a missing path, not a small number. Rows in a billing table are not evidence of billing. Check for stripe_customer_id and stripe_subscription_id being non-null, and check stripe_events — a webhook that never fired leaves both at zero while subscriptions looks healthy. This is how the NodeDough "the paywall cannot take money" finding surfaced.

Then the guard audit, matching on the guard name this codebase actually uses — grep one known-good admin_* function body first, then audit against that string. Auditing for the wrong guard name produced 8 false positives before it was corrected.

Phase 4 — Prove the security findings before you report them

A suspected hole is not a finding. Use the probe pattern: set the JWT claims to a real non-privileged user, exercise the thing, assert row counts changed — not the absence of an exception (an UPDATE matching zero rows raises nothing), then rollback. Before running a probe on production, check the target table for triggers or functions with external side effects (email, Stripe, webhooks); rollback does not undo those. If any exist, probe a branch or copy instead.

begin;
create temp table probe(name text, detail text, pass boolean);
do $probe$ declare v_before bigint; v_after bigint; v_err text; begin
  select count(*) into v_before from <target>;
  perform set_config('request.jwt.claims',
    json_build_object('sub','<non-admin-uuid>','role','authenticated')::text, true);
  perform set_config('role','authenticated', true);
  begin perform <the suspect call>; v_err := 'no exception raised';
  exception when others then v_err := SQLERRM; end;
  perform set_config('role','none', true);
  select count(*) into v_after from <target>;
  insert into probe values ('exception', v_err, v_err <> 'no exception raised'),
                           ('rows_written', (v_after-v_before)::text, (v_after-v_before)=0);
end $probe$;
select * from probe; rollback;

Assert the positive cases too. A probe that only tests denials cannot distinguish a correct policy from one that denies everyone.

Phase 5 — Rank, then write back

Default: rank by (revenue or security impact) ÷ effort, not by severity label; another ranking is fine if the reason is logged. A WARN that means "you cannot be paid" outranks 112 WARNs that mean "this is how the app is designed."

Bucket as P0 ship now / P1 high leverage low effort / P2 structural / P3 compounding housekeeping, and close with an "if you only do three things" — a CEO reads that line.

Then feed the system, in this order:

POST /api/1099/document — {actor, slug: "<app>-context", kind: "framework", title, body_md}.

Append-only, serves at /framework/<slug> immediately, no deploy. Record measured baselines (advisor counts, asset sizes, row counts) so the next agent diffs instead of re-measuring.

POST /api/1099/facts — ≤160 characters each, hard cap. Only traps that would cause

a wrong action. Briefs return at most 5, scored — 3 sharp facts beat 15 vague ones.

POST /api/1099/status — state, done, next, traps.
POST /api/1099/record-usage — field names are input_tokens / output_tokens.

tokens_in/in read as empty and are rejected ("refuse empty usage — ORG-4").

POST /api/1099/release.

Field-name traps (each of these has cost someone a retry)

Endpoint	Gets it wrong	Correct
`record-usage`	`tokens_in`, `in`	`input_tokens`, `output_tokens`
`document`	`body`, project slug	`body_md`, the doc's own kebab slug
`facts`	long paragraphs	≤160 chars, split longer
`status`	assumes it renews the lease	it does not — call `renew`
boot	`/api/1099/boot` as a GET route	use `/api/1099/brief`

Standing rules that override instinct

Never work without a claim; the clock runs claim → release.
Never ask the CEO to merge, commit or deploy. Push it yourself; the only human gate is

promoting test → production, and that is a chat approval.

Being unsure is not a reason to ask — pick the reversible option and say which.
Verify live before "done". Hit the public URL and assert the new content is there.
Offload tool output over ~2 K chars to a file; keep a path, not the payload.

Migrated from the NodeDough Claude Project by seat 20 on 2026-08-14. Original path claude/app-review-framework.md, created 2026-08-12T13:42:40.431918Z. Ops documents now live here, on projects.jbnx.io, not in a Claude Project.

Change log

Every change to this framework: who, when, why. Source: agent_ops.catalog_changelog.

When (UTC)	Who	Version	What	Why
2026-11-09 10:36	1099:claude-chat-c1d3	v4	Phase 0 boot now reads /core instead of /json; added the Jev pick first-line reminder.	Phase 0 boot loaded /json (~69 KB measured 2026-11-09) while core says read /core (~4 KB) once per version; the page also skipped the Jev pick line core requires. Cheaper boot serves the token-bounded north star.
2026-11-08 16:14	1099:claude-chat-c1d3	v3	JBNX app review framework (v1)	Owner-requested north-star review 2026-11-08: north star + hard gates header, top fixes, non-safety rules loosened to defaults (details in the page change log)
2026-08-12 13:42	1099:anthropic-cowork-migrated	v2	JBNX app review framework (v1)	not recorded (before change log started 2026-11-08)
2026-08-12 13:40	claude-opus-5-cowork	v1	JBNX app review framework (v1)	not recorded (before change log started 2026-11-08)