Does FP8 quantization degrade model output quality?

Extensive evaluations on MMLU and GSM8K demonstrate that modern FP8 (E4M3 format) maintains over 99.5% of original FP16 accuracy while doubling inference speed.

Can I run FP8 models on older NVIDIA RTX 30-series GPUs?

RTX 30-series (Ampere) lacks native FP8 tensor core acceleration. Use INT4 or AWQ quantization on Ampere, reserving native FP8 for RTX 40-series, H100, and Blackwell.

Intermediate · 15 minutes

How to Run Local LLMs with FP8 Quantization on Mac & Linux

Complete step-by-step engineering guide to deploying 14B, 32B, and 70B open-weight LLMs locally with high token throughput using MLX and vLLM.

Step 1: Verify Hardware & Memory Bandwidth Requirements

FP8 quantization cuts memory footprints by 50% compared to 16-bit weights. A 32B model requires approximately 34GB of unified memory or VRAM to run comfortable 8k-token contexts.

python3 -c "import torch; print(f'CUDA Available: {torch.cuda.is_available()}'); print(f'Device Count: {torch.cuda.device_count()}')"

Step 2: Install the Serving Framework (vLLM for Linux / MLX for Apple Silicon)

For Linux/CUDA environments, use vLLM for continuous batching and PagedAttention. For Apple Silicon Macs, use Apple’s native MLX framework.

# On Linux / CUDA:
pip install vllm --upgrade

# On Apple Silicon macOS:
pip install mlx-lm

Step 3: Download and Launch the FP8 Quantized Model

Launch the inference engine pointing to a pre-quantized FP8 HuggingFace repository with an OpenAI-compatible HTTP server endpoint.

# For Linux (vLLM):
vllm serve neuralmagic/Qwen2.5-32B-Instruct-FP8 \
  --port 8000 \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.92 \
  --kv-cache-dtype fp8

# For Apple Silicon (MLX):
mlx_lm.server --model mlx-community/Qwen2.5-32B-Instruct-4bit --port 8000

Step 4: Test Inference via Local OpenAI SDK Client

Point any standard OpenAI-compatible client library to your localhost port 8000.

import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "http://localhost:8000/v1",
  apiKey: "local-dev-token"
});

async function runLocalInference() {
  const completion = await client.chat.completions.create({
    model: "neuralmagic/Qwen2.5-32B-Instruct-FP8",
    messages: [
      { role: "system", content: "You are a concise, factual systems engineer." },
      { role: "user", content: "Explain the latency difference between SRAM and HBM in 3 sentences." }
    ],
    temperature: 0.2
  });

  console.log(completion.choices[0].message.content);
}

runLocalInference();