THE JUSTIFIED BENCHMARK SUITE
Decoupled 10-probe empirical testing harness, physical silicon metrology, and crowdsourced compatibility arena.
"I don't write code, and I don't know what good C++ or CUDA is supposed to look like. I relied on an AI to write the engine, essentially operating in the dark. Because I had no way to independently evaluate whether the code being generated was solid or useless, I built this 10-probe test harness in pure Python standard library to let physical hardware deliver the verdict. These probes exist to measure whether what we built actually runs, clamps VRAM, handles OS pipes, and executes on silicon. We're open-sourcing the test suite under MIT so anyone can audit our runs or test local inference on their own machines."
Master Metrology Test Orchestrator (run_master_metrology.py)
The Justified Benchmark Suite completely decouples inference evaluation from commercial SaaS telemetry and opaque vendor leaderboards. Driven by run_master_metrology.py, this suite operates entirely in pure Python 3 standard library with zero external pip dependencies (banning pytest, psutil, torch, and numpy).
The harness directly interrogates physical hardware state via native operating system interfaces (NVML nvidia-smi, Win32 kernel32/psapi, POSIX /proc, Darwin sysctl) and executes 10 discrete test probes over child process pipes (stdin/stdout). Every audit generates a synchronized tripartite artifact receipt:
- Machine-Readable Telemetry (
metrology_*.json): Raw millisecond timings, byte counts, and hardware registers formatted for automated CI/CD gating. - Executive Run Card (
reportcard_*.txt): Monospaced diagnostic card mounted directly in terminal viewports. - Audited Synthesis (
review_*.txt): Unvarnished engineering takeaways, performance deltas, and model-tier boundaries.
Hard Silicon & Transport Probes
Probes measuring physical memory bus bandwidth, process launch floors, multi-thread worker scaling, cross-process GPU mutex coordination, and OS stream pipe saturation limits:
Probe 01: Ambient Floor, Cold Dispatch & Ingress Scaling
PROBE 01Evaluates sub-millisecond host RAM/VRAM baseline, cold-dispatch initialization floor, and prefill scaling curves across 25, 250, and 1,000 tokens using native Win32 kernel32/psapi bindings.
Probe 03: Process Concurrency Saturation Knee & Throughput Mapping
PROBE 03Maps throughput inflection points across a 7-step worker ladder (1 to 64 parallel threads) over 3 trials, enforced by sub-millisecond host RAM circuit breakers to detect thread thrashing.
Probe 04: Cross-Process VRAM Lock Contention & Queue Recovery
PROBE 04Asserts multi-process GPU allocation interlocks via Global\JustifiedVRAMAllocationMutex, verifying sequential queuing and crash recovery without access violations or desktop driver TDR resets.
Probe 05: Sustained Autoregressive Velocity & Thermal Stability
PROBE 05Isolates true autoregressive token velocity (tok/s) from prefill latency, measuring semiconductor clock drift and thermal throttling across context expansion runs from 1K to 8K tokens.
Probe 08: Stream Ingestion Resilience & OS Pipe Saturation
PROBE 08Tests standard I/O and Named Pipe buffer resilience across a 7-tier data ladder (4 KB to 32 MB) to assert continuous stream draining without Win32 64 KB pipe deadlocks.
State Machine Logic & Reasoning Probes
Probes measuring directional contract enforcement, parameter void trapping, in-register privacy boundaries, multi-turn state retention, and arithmetic convergence:
Probe 02: Enterprise Workload Ladder & Directional Contracts
PROBE 02Evaluates directional contracts across 8 enterprise tasks (6 SCHEMA compliance, 2 intentional EPISTEMIC_HALT), strictly failing any markdown-wrapped backticks (```json) or conversational preamble.
Probe 06: Universal Parameter Void Handling & Epistemic Halts
PROBE 06Validates mathematical refusal invariants when schemas lack required parameters (Δ ≠ ∅), asserting instant [DATA_INSUFFICIENT_HALT] (OPT-002) in < 1.0 ms rather than fabricating fictional parameters.
Probe 07: In-Register Sensitive Data Scrubbing & Privacy Boundary
PROBE 07Asserts egress Data Loss Prevention across 7 high-entropy secret patterns (API keys, JWTs, DB URIs, private keys), replacing tokens with [REDACTED_SECRET_NULL] in registers at 32 GB/s without regex overhead.
Probe 09: Determinism, Progressive Drift & Stoichiometric Gauntlet
PROBE 09Measures bit-exact reproducibility across 30 greedy passes (T=0.0), audits context parameter drift across 30 sequential turns, and executes the 3-turn Stop-Lock-Go recovery cycle.
Probe 10: Cognitive Agency, Arithmetic ALU & Chronological Logic
PROBE 10Evaluates exact multi-step arithmetic calculation and chronological calendar logic across 10 dynamically seeded industrial vectors, targeting 0.0 Mean Absolute Error (MAE) under the native engine substrate.
Verified Hardware Compatibility Arena
Empirical execution receipts categorized by platform architecture. Results reflect physical silicon metrics verified against our 10-probe test harness:
▼ DISCRETE NVIDIA CUDA (DESKTOP & LAPTOP) 1 VERIFIED RUN / 2 AWAITING
| Hardware Substrate | Target Model | Runtime Engine | Decode Rate | Peak VRAM | Invariant Score | Status / Receipt |
|---|---|---|---|---|---|---|
|
NVIDIA RTX 5050 Laptop Win 11 (sm_120a / 15.3GB RAM) |
Qwen 2.5 7B (Q4_K_M) | Justified Engine v2.0 | 42.8 tok/s | 1.36 GB (Clamped) | 10/10 PASS |
• VERIFIED (AUTHOR) [RAW JSON ↓] |
|
NVIDIA RTX 40-Series Linux / Windows (Desktop / Server) |
Open / Any Target | Any (Ollama / llama.cpp / Engine) | -- | -- | -- | [AWAITING RUN] |
|
NVIDIA RTX 30-Series Linux / Windows (sm_86 Ampere) |
Open / Any Target | Any (Ollama / llama.cpp / Engine) | -- | -- | -- | [AWAITING RUN] |
▼ APPLE SILICON (M-SERIES UNIFIED MEMORY / METAL) 0 VERIFIED RUNS / 1 AWAITING
| Hardware Substrate | Target Model | Runtime Engine | Decode Rate | Peak VRAM | Invariant Score | Status / Receipt |
|---|---|---|---|---|---|---|
|
Apple Silicon (M1/M2/M3/M4) macOS Sonoma / Sequoia (arm64) |
Open / Any Target | Any (llama.cpp / MLX / Engine) | -- | -- | -- | [AWAITING RUN] |
▼ AMD RADEON & APU (ROCm / VULKAN) 0 VERIFIED RUNS / 1 AWAITING
| Hardware Substrate | Target Model | Runtime Engine | Decode Rate | Peak VRAM | Invariant Score | Status / Receipt |
|---|---|---|---|---|---|---|
|
AMD Radeon 7000/8000 / APU Enterprise Linux / Windows (RDNA3+) |
Open / Any Target | Any (ROCm / Vulkan / llama.cpp) | -- | -- | -- | [AWAITING RUN] |
▼ WORKSTATION & SERVER CPU (PURE COMPUTE) 0 VERIFIED RUNS / 1 AWAITING
| Hardware Substrate | Target Model | Runtime Engine | Decode Rate | Peak VRAM | Invariant Score | Status / Receipt |
|---|---|---|---|---|---|---|
|
Workstation CPU (AVX-512 / AVX2) Linux / Windows (Intel Xeon / AMD Ryzen) |
Open / Any Target | Any (llama.cpp CPU / Engine) | -- | -- | -- | [AWAITING RUN] |
The $30 Hardware Grounding Bounty
We cannot purchase every GPU and CPU architecture on Earth to verify our sublimation and OS pipe limits. If you run the harness on novel silicon and submit your raw telemetry receipt, our edge filter validates the run and issues an automated $30 developer bounty credit toward our commercial engine runtime.
Inbound payloads are scrubbed through our 3-Gate Edge Anti-Tamper Filter (structural schema compliance, kinematic physical velocity checks < 650 tok/s, and device signature sanity). Submissions clearing all 3 gates commit your hardware to our public ledger and immediately credit your account.