← BACK TO DEMOS & PRODUCTS
// STANDALONE VERIFICATION APPARATUS :: JUS-TEST-v2.0.0

THE JUSTIFIED BENCHMARK SUITE

Decoupled 10-probe empirical testing harness, physical silicon metrology, and crowdsourced compatibility arena.

// ARCHITECT'S FIELD NOTE :: OPERATOR CONTEXT

"I don't write code, and I don't know what good C++ or CUDA is supposed to look like. I relied on an AI to write the engine, essentially operating in the dark. Because I had no way to independently evaluate whether the code being generated was solid or useless, I built this 10-probe test harness in pure Python standard library to let physical hardware deliver the verdict. These probes exist to measure whether what we built actually runs, clamps VRAM, handles OS pipes, and executes on silicon. We're open-sourcing the test suite under MIT so anyone can audit our runs or test local inference on their own machines."

// ORCHESTRATION ARCHITECTURE :: ASSET 26

Master Metrology Test Orchestrator (run_master_metrology.py)

The Justified Benchmark Suite completely decouples inference evaluation from commercial SaaS telemetry and opaque vendor leaderboards. Driven by run_master_metrology.py, this suite operates entirely in pure Python 3 standard library with zero external pip dependencies (banning pytest, psutil, torch, and numpy).

The harness directly interrogates physical hardware state via native operating system interfaces (NVML nvidia-smi, Win32 kernel32/psapi, POSIX /proc, Darwin sysctl) and executes 10 discrete test probes over child process pipes (stdin/stdout). Every audit generates a synchronized tripartite artifact receipt:

# Windows (Discrete NVIDIA CUDA): python run_master_metrology.py --engine justified_engine.exe --model weights\qwen2.5-7b-instruct.gguf --export metrology_stock_7b.json # macOS (Apple Silicon Metal): python run_master_metrology.py --engine ./justified_engine --model ./weights/qwen2.5-7b-instruct.gguf --export metrology_stock_7b.json # Linux (POSIX CPU / CUDA): python run_master_metrology.py --engine ./justified_engine --model ./weights/qwen2.5-7b-instruct.gguf --export metrology_stock_7b.json # Calibrated Companion Vault (.jef_bin): python run_master_metrology.py --engine ./justified_engine --model ./vaults/j8b_refactor_pro.jef_bin --export metrology_j8b_calibrated.json
// SUITE A :: PHYSICAL SILICON & HARDWARE BOUNDARIES

Hard Silicon & Transport Probes

Probes measuring physical memory bus bandwidth, process launch floors, multi-thread worker scaling, cross-process GPU mutex coordination, and OS stream pipe saturation limits:

Probe 01: Ambient Floor, Cold Dispatch & Ingress Scaling

PROBE 01

Evaluates sub-millisecond host RAM/VRAM baseline, cold-dispatch initialization floor, and prefill scaling curves across 25, 250, and 1,000 tokens using native Win32 kernel32/psapi bindings.

Script: tests/probe_01_cold_boot_ram.py Pass Condition: Cold boot ≤ 1,200 ms | Working set ≤ model mass + 512 MB

Probe 03: Process Concurrency Saturation Knee & Throughput Mapping

PROBE 03

Maps throughput inflection points across a 7-step worker ladder (1 to 64 parallel threads) over 3 trials, enforced by sub-millisecond host RAM circuit breakers to detect thread thrashing.

Script: tests/probe_03_concurrency.py Pass Condition: Zero process deadlocks | Linear scaling up to GPU execution queues

Probe 04: Cross-Process VRAM Lock Contention & Queue Recovery

PROBE 04

Asserts multi-process GPU allocation interlocks via Global\JustifiedVRAMAllocationMutex, verifying sequential queuing and crash recovery without access violations or desktop driver TDR resets.

Script: tests/probe_04_vram_mutex.py Pass Condition: 100% mutex ownership handoff | Zero orphaned locks on SIGKILL

Probe 05: Sustained Autoregressive Velocity & Thermal Stability

PROBE 05

Isolates true autoregressive token velocity (tok/s) from prefill latency, measuring semiconductor clock drift and thermal throttling across context expansion runs from 1K to 8K tokens.

Script: tests/probe_05_sublogit_bypass.py Pass Condition: Clock drift ≤ 5% | Zero token loss across continuous decoding

Probe 08: Stream Ingestion Resilience & OS Pipe Saturation

PROBE 08

Tests standard I/O and Named Pipe buffer resilience across a 7-tier data ladder (4 KB to 32 MB) to assert continuous stream draining without Win32 64 KB pipe deadlocks.

Script: tests/probe_08_pipe_scaling.py Pass Condition: 32 MB continuous AST ingestion | Zero pipe backpressure buffer stalls
// SUITE B :: COGNITIVE AGENCY & STATE INVARIANTS

State Machine Logic & Reasoning Probes

Probes measuring directional contract enforcement, parameter void trapping, in-register privacy boundaries, multi-turn state retention, and arithmetic convergence:

Probe 02: Enterprise Workload Ladder & Directional Contracts

PROBE 02

Evaluates directional contracts across 8 enterprise tasks (6 SCHEMA compliance, 2 intentional EPISTEMIC_HALT), strictly failing any markdown-wrapped backticks (```json) or conversational preamble.

Script: tests/probe_02_workload_stream.py Pass Condition: 100% RFC 8259 bare JSON purity | First-token structural adherence

Probe 06: Universal Parameter Void Handling & Epistemic Halts

PROBE 06

Validates mathematical refusal invariants when schemas lack required parameters (Δ ≠ ∅), asserting instant [DATA_INSUFFICIENT_HALT] (OPT-002) in < 1.0 ms rather than fabricating fictional parameters.

Script: tests/probe_06_epistemic_void.py Pass Condition: 3/3 contract voids halted | Zero synthetic field hallucinations

Probe 07: In-Register Sensitive Data Scrubbing & Privacy Boundary

PROBE 07

Asserts egress Data Loss Prevention across 7 high-entropy secret patterns (API keys, JWTs, DB URIs, private keys), replacing tokens with [REDACTED_SECRET_NULL] in registers at 32 GB/s without regex overhead.

Script: tests/probe_07_swar_scrubbing.py Pass Condition: 7/7 injected secrets nullified | Zero plaintext leakage to stdout

Probe 09: Determinism, Progressive Drift & Stoichiometric Gauntlet

PROBE 09

Measures bit-exact reproducibility across 30 greedy passes (T=0.0), audits context parameter drift across 30 sequential turns, and executes the 3-turn Stop-Lock-Go recovery cycle.

Script: tests/probe_09_semantic_drift_hallucination.py Pass Condition: 100% bit-exact across N=30 passes | Invariant retention ≥ 21 turns

Probe 10: Cognitive Agency, Arithmetic ALU & Chronological Logic

PROBE 10

Evaluates exact multi-step arithmetic calculation and chronological calendar logic across 10 dynamically seeded industrial vectors, targeting 0.0 Mean Absolute Error (MAE) under the native engine substrate.

Script: tests/probe_10_cognitive_reasoning.py Pass Condition: MAE = 0.0 on Tier 2+ (J8B/J14B) | Explicit excision notice on Tier 1 (J1B)
// CROWDSOURCED ARENA LEDGER :: KV_ANALYTICS:HARDWARE

Verified Hardware Compatibility Arena

Empirical execution receipts categorized by platform architecture. Results reflect physical silicon metrics verified against our 10-probe test harness:

▼ DISCRETE NVIDIA CUDA (DESKTOP & LAPTOP) 1 VERIFIED RUN / 2 AWAITING
Hardware Substrate Target Model Runtime Engine Decode Rate Peak VRAM Invariant Score Status / Receipt
NVIDIA RTX 5050 Laptop
Win 11 (sm_120a / 15.3GB RAM)
Qwen 2.5 7B (Q4_K_M) Justified Engine v2.0 42.8 tok/s 1.36 GB (Clamped) 10/10 PASS • VERIFIED (AUTHOR)
[RAW JSON ↓]
NVIDIA RTX 40-Series
Linux / Windows (Desktop / Server)
Open / Any Target Any (Ollama / llama.cpp / Engine) -- -- -- [AWAITING RUN]
NVIDIA RTX 30-Series
Linux / Windows (sm_86 Ampere)
Open / Any Target Any (Ollama / llama.cpp / Engine) -- -- -- [AWAITING RUN]
▼ APPLE SILICON (M-SERIES UNIFIED MEMORY / METAL) 0 VERIFIED RUNS / 1 AWAITING
Hardware Substrate Target Model Runtime Engine Decode Rate Peak VRAM Invariant Score Status / Receipt
Apple Silicon (M1/M2/M3/M4)
macOS Sonoma / Sequoia (arm64)
Open / Any Target Any (llama.cpp / MLX / Engine) -- -- -- [AWAITING RUN]
▼ AMD RADEON & APU (ROCm / VULKAN) 0 VERIFIED RUNS / 1 AWAITING
Hardware Substrate Target Model Runtime Engine Decode Rate Peak VRAM Invariant Score Status / Receipt
AMD Radeon 7000/8000 / APU
Enterprise Linux / Windows (RDNA3+)
Open / Any Target Any (ROCm / Vulkan / llama.cpp) -- -- -- [AWAITING RUN]
▼ WORKSTATION & SERVER CPU (PURE COMPUTE) 0 VERIFIED RUNS / 1 AWAITING
Hardware Substrate Target Model Runtime Engine Decode Rate Peak VRAM Invariant Score Status / Receipt
Workstation CPU (AVX-512 / AVX2)
Linux / Windows (Intel Xeon / AMD Ryzen)
Open / Any Target Any (llama.cpp CPU / Engine) -- -- -- [AWAITING RUN]
// COMMUNITY SILICON TELEMETRY :: THE HARDWARE BOUNTY PROTOCOL

The $30 Hardware Grounding Bounty

We cannot purchase every GPU and CPU architecture on Earth to verify our sublimation and OS pipe limits. If you run the harness on novel silicon and submit your raw telemetry receipt, our edge filter validates the run and issues an automated $30 developer bounty credit toward our commercial engine runtime.

Inbound payloads are scrubbed through our 3-Gate Edge Anti-Tamper Filter (structural schema compliance, kinematic physical velocity checks < 650 tok/s, and device signature sanity). Submissions clearing all 3 gates commit your hardware to our public ledger and immediately credit your account.