orion-trt-llm-edge deployment artifacts

This is the artifact companion of the GitHub repo hous-lab/orion-trt-llm-edge: the GitHub repo is the source of truth and holds the full deployment toolchain, the same results tables, get-started and docs. This Hugging Face repo holds the self-produced ONNX exports and the distilled student checkpoint. The upstream ORION checkpoint and its sub-modules are NOT included here and are not redistributed - get them from the upstream releases (download table in the repo's ckpts/README.md).

Files

File	Size	What it is
orion_lite_vision_sim_mixed.onnx	485 MB	Vision tower (EVA02 backbone + heads), 6-layer distilled variant at 384 input, simplified + mixed-precision (backbone FP16-marked, heads FP32)
orion_lite_student_sim.onnx	1.6 GB	Distilled planning head (529M-param student), seq_len=529 contract, FP32
orion_lite_student_best.pt	2.0 GB	Student training checkpoint (weights + training metadata; optimizer state dropped)

Status: Active. Results last verified on Jetson AGX Orin 64 GB (TensorRT 10.4.0 / CUDA 12.6, MAXN with locked clocks, 2026-10) and RTX 4090 (TensorRT-LLM v0.13 chain, 2026-08); issues and PRs are welcome.

Results

7B Orion - Full-pipeline end-to-end latency on the 4090 24GB GPU, measured as input frame to trajectory output through vision engine + canbus/memory update + TRT-LLM LLM + ego/planning heads + output post-processing (per-module CUDA-event profiling, 30-frame mean, plan_L2 re-verified at 0.692 on the 173-frame set):

Configuration	Vision	LLM	Ego head	End-to-end
FP16-tier LLM	59.1 ms	57.5 ms	5.0 ms	~122 ms
W8A8 SmoothQuant LLM	59.1 ms	26.0 ms	5.0 ms	~90 ms

LLM-only latency on two RTX 4090-class GPUs (24 GB server / 16 GB laptop), plan_L2 on the 173-frame Bench2Drive val subset (official PyTorch reference: 0.690):

LLM tier	plan_L2 @1s	LLM latency, 4090 24 GB	LLM latency, 4090 Laptop 16 GB
PyTorch official (reference)	0.690	-	-
TRT-LLM FP16	0.697	64.6 ms	OOM (13.5 GB weights)
TRT-LLM INT8 (w8abf16)	0.692	65.6 ms	132.5 ms
TRT-LLM INT4 (w4a16)	0.750 (degrades)	64.9 ms	126.0 ms
TRT-LLM W8A8 SmoothQuant	0.696	26.0 ms	50.4 ms

The LLM stage reads 57.5 ms inside this pipeline vs 64.6 ms in the isolated prefill profile above - different harness and frame set, both reported as measured. The INT8/INT4 tiers were not end-to-end timed (their LLM-only latency sits within ~1 ms of FP16, so the end-to-end number is expected in the same ~122 ms band); the laptop was not end-to-end timed either. Reproduce with trtllm_7b/scripts/01_profile_modules.py (per-module CUDA events) and trtllm_7b/scripts/10_e2e_latency.py (full serial wall-clock pass), both driving the same patched training repo via ORION_PROJECT_ROOT.

Every row of this table is the same original upstream 7B model (Orion.pth - no distillation, no retraining anywhere in the 7B chain). Rows differ only in the LLM quantization tier; the vision engine and the planning heads are identical in every row. "LLM latency" is the LLM-only prefill (one forward pass of the 599-token prompt with max_new_tokens=1, mean over repeated post-warmup runs, f10 frame). plan_L2 is full-pipeline accuracy (vision engine + TRT-LLM LLM + planning heads) under the same eval protocol as the PyTorch reference; the INT8/INT4 tiers re-evaluated on the laptop scored 0.689/0.746, within 0.004 of the server values.

Orion-Lite on Jetson AGX Orin, MAXN power mode, plan_L2 on the same 173-frame subset:

Variant	Orin end-to-end	plan_L2 @1s
24L@640 + original student (undistilled starting point)	678 ms	0.814
24L@512 + original student (baseline)	404 ms	0.868
12L@512 (ViT distilled 24 to 12 layers)	236 ms	0.694
6L@448 (earlier 6-layer round)	121 ms (154 ms at 50W)	0.776
6L@384 + original student	81.2 ms (re-verified on the shipped artifacts)	0.704

Scheme notes: the 529M student is not a from-scratch small model - it is itself an LLM distillate, trained upstream to imitate the 7B LLM's planning token, and it stays frozen in every row. What this repository distills is the vision tower only: rows 1-2 are the undistilled 24L starting points, rows 3-5 replace the 24-layer EVA02 tower with 12/6-layer versions distilled against the 24L teacher at reduced input resolution. Orin end-to-end = vision tower + student + planning heads in one process at MAXN; plan_L2 = the fut-valid subset (133 frames) of the same 173-frame route. The Lite metric is not directly comparable to the 7B table above - different harness and frame count; see docs/model_variants.md.

About the 6L@384 latency numbers: a re-verification pass on the shipped artifacts (HF ONNX -> our TRT builder -> a zero-copy joint chain that aliases the vision output buffer as the student input) measures the end-to-end chain at 81.2 ms on an idle Orin at MAXN with locked clocks (per-frame wall clock, n=50). In the shipped step4 real-data harness the vision engine alone measures 84.3 ms including per-frame input copies and output copies, and the student engine 14.2 ms in an inputs-resident execute+sync loop. The earlier campaign numbers (85.2 ms = vision 69.7 + student 15.5) come from an inputs-resident execute+sync harness on an earlier engine build - the segment sums are not directly comparable to the harness numbers above, and the rebuilt engine is faster than the earlier build (88.9 ms vs 84.3 ms in the same shipped harness on the same test frame). Accuracy gates pass on the re-verification: vision-embedding cosine 0.864 (gate 0.809) and bbox cosine 0.9988 (matching the committed x86 evidence at 0.9985). Full numbers in lite/results/orin_reverify_6l384_20261024.json.

Measurement conditions. Lite (Orin): Jetson AGX Orin 64 GB, MAXN + locked clocks (jetson_clocks), TensorRT 10.4.0 / CUDA 12.6; the 6L@384 end-to-end figure is a zero-copy joint-chain per-frame wall clock (n=50, mean 81.2 / p50 81.3 / p95 81.4 ms), step4-harness figures include per-frame input/output copies, and variant rows 1-4 are from the earlier measurement campaign under its inputs-resident execute+sync convention (individually measured then, not individually re-verified on the shipped artifacts - only the final 6L@384 row was re-verified; full numbers in lite/results/orin_reverify_6l384_20261024.json). 7B (x86): RTX 4090 24 GB, TensorRT-LLM v0.13 with the hidden-states patch, end-to-end via per-module CUDA events (30-frame mean), the LLM-only table via isolated prefill runs (mean over repeated post-warmup runs, single frame); plan_L2 everywhere = the 173-frame Bench2Drive val route, fut-valid-only subset (133 frames), computed by lite/eval/eval_planning_local.py for the Lite chain and the upstream eval protocol for 7B; engine-level gates: vision-embedding cosine >= 0.809 and bbox cosine in the 0.999 tier.

Takeaways we would want every reader to leave with:

• The hidden-states export is the enabling piece for the 7B deployment: TRT-LLM v0.13 has no API for it (upstream issue #4414); the two-line patch turns any v0.x LLaMA-family build into a hidden-states exporter (docs/hidden_states_export.md also covers the v1.x backend).
• INT8/PTQ measured no latency win on the vision workload: calibrated INT8 ran within noise of FP16 on x86 (128.8 vs 129.7 ms at parity accuracy, avg cosine 0.9228 vs 0.9229 on the same real-frame benchmark) and no gain on Orin (Q/DQ overhead cancelled the savings); the shipped INT8 tier exists for completeness, FP16 is the recommended tier.
• On the 7B LLM the interesting result is the opposite: W8A16/INT4 weight-only quantization did not move prefill latency (65 ms band), while full W8A8 SmoothQuant halved activation traffic and dropped it 2.5x to 26 ms. Quantize activations, not just weights.
• Resolution reduction is the training-free lever (640 to 512 alone: 678 to 404 ms e2e), 384 only worked after the backbone-only 6-layer distillation (ve cos 0.926 vs 0.866 for the 24-layer tower at 384).
• Power mode matters as much as kernel work on Orin: 50W to MAXN alone cuts the vision module 118.96 to 84.76 ms (-29%), e2e 154 to 121 ms.

Usage

The artifacts are consumed by the pipeline in hous-lab/orion-trt-llm-edge - the GitHub repo is the source of truth: run_pipeline.sh builds TRT engines from the ONNX files (step 3) and runs the engine-vs-reference benchmark with cos gates (step 4). To load the student checkpoint, use the student definition in lite/distill_code/.

Provenance

• The vision ONNX was exported from the 6-layer distilled vision tower (Bench2Drive 384x384, Bench2Drive-based distillation), with flash_attn=False (plain attention path, ONNX-traceable) and the simplification step marks the backbone FP16 and the heads FP32.
• The student checkpoint is our from-scratch distillation training output (orion_lite_subset8 run, best epoch 20 / step 8520, best_val_loss 4.8959); the exported ONNX (seq 529) carries the deployed student. The student is trained from scratch - it does not embed upstream weights.
• Engine outputs were verified against the FP32 reference on real Bench2Drive frames: vision-embedding cosine 0.883 and bbox cosine 0.999 (FP16 engine, RTX 4090 Laptop, TRT 10.16; see the repo's lite/results/ for the committed benchmark JSONs).

Platform caveats

• Engines built and verified with TensorRT 10.4 / 10.16 on RTX 4090 (24GB server, 16GB laptop) and Jetson AGX Orin 64GB (TRT 10.4, JetPack). TRT engines are not portable across major TRT versions - rebuild from the ONNX files on your target.
• The vision ONNX expects the full 21-input stateful interface (memory/canbus streams); see the repo's data/README.md and docs/pipeline_guide.md.
• FP16 on the backbone only; the heads stay FP32 (FP16 everywhere collapsed accuracy in our runs).