RUN_GUIDE.md

PetitGPT native run guide

Author: Yang Qi. Documentation: CC BY 4.0. Download the loose model repository files together, preserving the src/ directory. The exact release file list and hashes are in SHA256SUMS. Model weights, tokenizer, config and executable code retain the accepted export bytes.

2. Requirements

A CUDA GPU is required by this CLI. The tested configuration is Python 3.10.12 with:

torch==2.11.0+cu126
numpy==2.2.6
tokenizers==0.22.2
safetensors==0.8.0

on an NVIDIA GeForce RTX 4090 (driver 580.178.04). Prepare the environment separately, from locally supplied wheels:

python -m pip install --no-index --find-links /path/to/local/wheelhouse \
    -r /path/to/bundle/requirements-inference-tested.txt

No packages were installed during the export itself. Matching versions do not guarantee bit-identical results on arbitrary hardware or untested software.

3. Command line

Replace /path/to/bundle with the real extracted location.

python /path/to/bundle/inference.py \
    --model-directory /path/to/bundle \
    --prompt "Say hello in one sentence." \
    --profile bf16_native \
    --max-new-tokens 32

python /path/to/bundle/inference.py \
    --model-directory /path/to/bundle \
    --messages-json /path/to/messages.json \
    --profile fp32_math \
    --max-new-tokens 32

A synthetic messages.json:

[{"role":"system","content":"Use short sentences."},
 {"role":"user","content":"My name is Lin."},
 {"role":"assistant","content":"Hello, Lin."},
 {"role":"user","content":"What name did I give you?"}]

Illustrative and unexecuted. The greeting above is an authored example showing command syntax only.

4. Python API

With the extracted bundle directory on sys.path:

from inference import load_bundle, generate

model, tokenizer = load_bundle("/path/to/bundle")
model = model.to("cuda").eval()
result = generate(
    model, tokenizer,
    [{"role": "user", "content": "Say hello."}],
    cap=32,
    profile="fp32_math",
)
print(result["output_text_raw_including_terminal_eos"])

5. Input contract

messages must be a JSON array of objects with exactly the fields role and content. A conversation is an optional initial system turn followed by alternating user/assistant turns, ending in user. A plain --prompt becomes one user message.

Rejected, by design, with an explicit error rather than a repair:

Case	Recorded rejection
Missing content	Each message must contain only role and content
Unsupported role (e.g. tool)	message 0 has invalid role 'tool'
Any extra message field	Each message must contain only role and content
Conversation ending in assistant	chat must end with a non-empty user turn; got 'assistant'
Prompt + budget over 2,048	Context overflow: prompt plus token budget exceeds 2048
max_new_tokens outside 1..384	max_new_tokens must be 1..384

default_system=None: a supplied system turn and full history are retained, with no injected default and no text normalization. Literal special-token spellings inside content — for example the literal spelling [EOS] — are encoded as ordinary text and cannot inject control IDs.

Native token structure:

[BOS] <|system|> system <|user|> user <|assistant|> assistant [EOS] … <|user|> user <|assistant|>

The system segment is omitted when absent. [BOS] occurs once; [EOS] closes completed assistant turns; no duplicate assistant prefix is added. IDs are fixed: [PAD]=0, [UNK]=1, [BOS]=2, [EOS]=3, <|system|>=4, <|user|>=5, <|assistant|>=6.

6. Precision profiles

Both profiles store FP32 parameters. They differ only in the forward numerical path, and they are not asserted to agree with each other.

	bf16_native	fp32_math
Forward	CUDA BF16 autocast	FP32, autocast off
matmul TF32	off	off
cuDNN TF32	on	off
float32_matmul_precision	highest	highest
SDPA backend	native backends enabled	MATH

7. Decoding and stopping

Greedy only: temperature=0, top_k=0, top_p=1, EOS=3. Default max_new_tokens is 384; an explicit lower integer in 1..384 is supported. Prompt length plus budget must fit 2,048 — there is no cropping.

Output JSON preserves the generated token IDs, the raw text including the terminal EOS, and a stop reason of eos or max_new_tokens. There is no answer cleanup, no fact fixing, no best-of-N, and no retry. A token cap can truncate an answer mid-sentence; that is the recorded behaviour, not a defect.

8. Format support and limits

Native PyTorch CUDA inference only. There is no implemented or tested Transformers AutoModel, GGUF, ONNX, vLLM or llama.cpp path. special_tokens_map.json is descriptive native metadata, not a loader contract.

Local import closure was demonstrated once, in a fresh process with a temporary working directory and an empty PYTHONPATH, under an audit hook that denied network access and out-of-bundle repository access; no denied access occurred. That establishes closure on the measured environment only. It is not a clean-machine test, not a CPU test, not a cross-hardware test, and not a fresh dependency-installation test.

The recorded export parity is likewise profile-specific: within each of the two profiles the export reproduced its source exactly, and no claim is made that the two profiles agree with each other, nor that any Transformers, GGUF, ONNX, vLLM or llama.cpp path exists.