Instructions to use tiiuae/Falcon-H1-1.5B-Instruct-GPTQ-Int8 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use tiiuae/Falcon-H1-1.5B-Instruct-GPTQ-Int8 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="tiiuae/Falcon-H1-1.5B-Instruct-GPTQ-Int8")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("tiiuae/Falcon-H1-1.5B-Instruct-GPTQ-Int8")
model = AutoModelForCausalLM.from_pretrained("tiiuae/Falcon-H1-1.5B-Instruct-GPTQ-Int8", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use tiiuae/Falcon-H1-1.5B-Instruct-GPTQ-Int8 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "tiiuae/Falcon-H1-1.5B-Instruct-GPTQ-Int8"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "tiiuae/Falcon-H1-1.5B-Instruct-GPTQ-Int8",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/tiiuae/Falcon-H1-1.5B-Instruct-GPTQ-Int8

SGLang

How to use tiiuae/Falcon-H1-1.5B-Instruct-GPTQ-Int8 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "tiiuae/Falcon-H1-1.5B-Instruct-GPTQ-Int8" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "tiiuae/Falcon-H1-1.5B-Instruct-GPTQ-Int8",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "tiiuae/Falcon-H1-1.5B-Instruct-GPTQ-Int8" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "tiiuae/Falcon-H1-1.5B-Instruct-GPTQ-Int8",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use tiiuae/Falcon-H1-1.5B-Instruct-GPTQ-Int8 with Docker Model Runner:
```
docker model run hf.co/tiiuae/Falcon-H1-1.5B-Instruct-GPTQ-Int8
```

Falcon-H1-1.5B-Instruct-GPTQ-Int8 / quant_log.csv

HamzaYous89

Upload folder using huggingface_hub

21387d8 verified over 1 year ago

Raw History Blame Contribute Delete

3.72 kB

	layer,module,loss,samples,damp,time
	0,feed_forward.gate_proj,1.30510879,0.01000,3.725
	0,feed_forward.up_proj,4.05003500,0.01000,0.510
	0,feed_forward.down_proj,8.25818729,0.01000,1.516
	1,feed_forward.gate_proj,1.29748273,0.01000,0.511
	1,feed_forward.up_proj,2.73292828,0.01000,0.511
	1,feed_forward.down_proj,5.14805508,0.01000,1.159
	2,feed_forward.gate_proj,1.23090601,0.01000,0.510
	2,feed_forward.up_proj,2.30279851,0.01000,0.511
	2,feed_forward.down_proj,4.61468744,0.01000,1.158
	3,feed_forward.gate_proj,1.10048580,0.01000,0.511
	3,feed_forward.up_proj,1.90120482,0.01000,0.511
	3,feed_forward.down_proj,11.11385250,0.01000,1.158
	4,feed_forward.gate_proj,0.91697407,0.01000,0.510
	4,feed_forward.up_proj,1.58906507,0.01000,0.510
	4,feed_forward.down_proj,3.70194459,0.01000,1.160
	5,feed_forward.gate_proj,1.04244685,0.01000,0.509
	5,feed_forward.up_proj,1.76986051,0.01000,0.511
	5,feed_forward.down_proj,3.52866268,0.01000,1.158
	6,feed_forward.gate_proj,0.96679544,0.01000,0.510
	6,feed_forward.up_proj,1.59820867,0.01000,0.510
	6,feed_forward.down_proj,3.57418442,0.01000,1.159
	7,feed_forward.gate_proj,0.93978858,0.01000,0.511
	7,feed_forward.up_proj,1.59313560,0.01000,0.511
	7,feed_forward.down_proj,3.60026312,0.01000,1.161
	8,feed_forward.gate_proj,1.01717091,0.01000,0.510
	8,feed_forward.up_proj,1.74156618,0.01000,0.511
	8,feed_forward.down_proj,4.19336700,0.01000,1.158
	9,feed_forward.gate_proj,1.10104227,0.01000,0.512
	9,feed_forward.up_proj,1.79372549,0.01000,0.511
	9,feed_forward.down_proj,4.49549913,0.01000,1.161
	10,feed_forward.gate_proj,1.24670470,0.01000,0.511
	10,feed_forward.up_proj,2.07013083,0.01000,0.512
	10,feed_forward.down_proj,5.10455418,0.01000,1.160
	11,feed_forward.gate_proj,1.36271381,0.01000,0.511
	11,feed_forward.up_proj,2.28341436,0.01000,0.511
	11,feed_forward.down_proj,7.72139645,0.01000,1.161
	12,feed_forward.gate_proj,1.60862815,0.01000,0.512
	12,feed_forward.up_proj,2.74706793,0.01000,0.513
	12,feed_forward.down_proj,9.77115631,0.01000,1.161
	13,feed_forward.gate_proj,2.02612519,0.01000,0.511
	13,feed_forward.up_proj,3.36923885,0.01000,0.513
	13,feed_forward.down_proj,15.91432858,0.01000,1.159
	14,feed_forward.gate_proj,2.29283261,0.01000,0.511
	14,feed_forward.up_proj,3.86089945,0.01000,0.511
	14,feed_forward.down_proj,18.98839760,0.01000,1.157
	15,feed_forward.gate_proj,2.88071609,0.01000,0.512
	15,feed_forward.up_proj,4.80165291,0.01000,0.512
	15,feed_forward.down_proj,24.14971542,0.01000,1.159
	16,feed_forward.gate_proj,3.48945332,0.01000,0.510
	16,feed_forward.up_proj,5.78722095,0.01000,0.509
	16,feed_forward.down_proj,32.97491074,0.01000,1.159
	17,feed_forward.gate_proj,4.11738968,0.01000,0.512
	17,feed_forward.up_proj,6.77789783,0.01000,0.511
	17,feed_forward.down_proj,44.23390961,0.01000,1.156
	18,feed_forward.gate_proj,5.10589027,0.01000,0.510
	18,feed_forward.up_proj,8.46793747,0.01000,0.510
	18,feed_forward.down_proj,69.04174805,0.01000,1.160
	19,feed_forward.gate_proj,5.87184620,0.01000,0.511
	19,feed_forward.up_proj,9.68615150,0.01000,0.512
	19,feed_forward.down_proj,98.21003723,0.01000,1.160
	20,feed_forward.gate_proj,6.33989811,0.01000,0.510
	20,feed_forward.up_proj,10.79653072,0.01000,0.511
	20,feed_forward.down_proj,133.90747070,0.01000,1.159
	21,feed_forward.gate_proj,7.71545410,0.01000,0.512
	21,feed_forward.up_proj,12.76975441,0.01000,0.512
	21,feed_forward.down_proj,161.60745239,0.01000,1.159
	22,feed_forward.gate_proj,9.65821552,0.01000,0.511
	22,feed_forward.up_proj,14.39665413,0.01000,0.511
	22,feed_forward.down_proj,238.05212402,0.01000,1.161
	23,feed_forward.gate_proj,16.67799759,0.01000,0.512
	23,feed_forward.up_proj,23.21625900,0.01000,0.511
	23,feed_forward.down_proj,444.00360107,0.01000,1.158