[QUESTION] Qwen3 Omni VRAM memory leak

[tommyip]

The model is around 70 GiB. I tried running GPTQModel on a RTX PRO 6000 with 96GiB vram but still ran out of memory. Config QuantizeConfig(bits=4, group_size=128).

[Qubitium]

@tommyip What version of gptqmodel are you using? Make sure you install main branch for latest gpu/cpu memory savings. v5.0.0 (main branch) has hundreds of changes and optimizations not yet fully-released yet since we need to pass some critical ci tests.

git clone --depth 1 ...
cd gptqmodel
pip install -v -e . --no-build-isolation

[tommyip]

I did install the main branch with uv pip install -v git+https://github.com/ModelCloud/GPTQModel --no-build-isolation. This should be the same as cloning + build? The version shows as 5.0.0+dev

[Qubitium]

I did install the main branch with uv pip install -v git+https://github.com/ModelCloud/GPTQModel --no-build-isolation. This should be the same as cloning + build? The version shows as 5.0.0+dev

Yes. This is the same as clone + compile. We are good here.

You need to post the last 100 lines or so terminial output before you got the oom. I need to see what's going on and I need more debug info.

[tommyip]

Ahh I terminated the machine I was renting. This is the only screenshot I have:

I can get more logs later today. I'll also try out an h200 to see if it works.

[Qubitium]

@tommyip Make sure to install flash-attention. It may reduce vram usage. Your vram usage and oom is very strange. We did test on A100 with 80GB of memory and it worked so how can it oom on 96GB but we had flash attention enabled and gptqmodel will auto enable it if flash attention is installd. And collect detailed logs and screenshots next time it crashes so we can see at which step and overall situation awareness.

[Jun-Howie]

I remember seeing the 5.0.0 tag yesterday — did something happen to it?

[Qubitium]

I remember seeing the 5.0.0 tag yesterday — did something happen to it?

? 5.0 hasn't been released yet. we are doing final ci test validations.

[tommyip]

Installed flash attention but still getting OOM. Traceback:

Traceback (most recent call last): based config for targeting of modules                                                                            
  File "/workspace/qwen3/quant.py", line 18, in <module>░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░| 0:00:55 / 0:44:00 [1/48] 2.1%
    model.quantize(calibration_dataset, batch_size=1)                                                                                               
  File "/venv/main/lib/python3.12/site-packages/gptqmodel/models/base.py", line 1013, in quantize
    result = module_looper.loop(                                                                                                                    
             ^^^^^^^^^^^^^^^^^^^                                          
  File "/venv/main/lib/python3.12/site-packages/gptqmodel/looper/module_looper.py", line 963, in loop
    return self._loop_impl(fail_safe=fail_safe, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/main/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
    return func(*args, **kwargs)                                          
           ^^^^^^^^^^^^^^^^^^^^^
  File "/venv/main/lib/python3.12/site-packages/gptqmodel/looper/module_looper.py", line 1204, in _loop_impl
    forward_outputs = self._run_forward_batches(
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/main/lib/python3.12/site-packages/gptqmodel/looper/module_looper.py", line 406, in _run_forward_batches
    return self._run_forward_batches_single(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/main/lib/python3.12/site-packages/gptqmodel/looper/module_looper.py", line 521, in _run_forward_batches_single
    module_output = module(*layer_input, **additional_inputs)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/main/lib/python3.12/site-packages/transformers/modeling_layers.py", line 94, in __call__
    return super().__call__(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/main/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/main/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/main/lib/python3.12/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/venv/main/lib/python3.12/site-packages/transformers/models/qwen3_omni_moe/modeling_qwen3_omni_moe.py", line 1535, in forward
    hidden_states, _ = self.self_attn(
                       ^^^^^^^^^^^^^^^
  File "/venv/main/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/main/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/main/lib/python3.12/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/venv/main/lib/python3.12/site-packages/transformers/models/qwen3_omni_moe/modeling_qwen3_omni_moe.py", line 1462, in forward
    attn_output, attn_weights = attention_interface(
                                ^^^^^^^^^^^^^^^^^^^^
  File "/venv/main/lib/python3.12/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward
    attn_output = torch.nn.functional.scaled_dot_product_attention(
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 37.10 GiB. GPU 0 has a total capacity of 94.97 GiB of which 32.70 GiB is free. Including non-PyTorch memory, this process has 62.26 GiB memory in use. Of the allocated memory 58.69 GiB is allocated by PyTorch, and 2.78 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)

Package versions:

gptqmodel 1077c9545f019e29e3eac6313dbf6ed71b4530a9
torch                    2.9.0+cu128
torchvision              0.24.0
transformers             4.57.1   
triton                   3.5.0  
flash-attn               2.8.3 
nvidia-cuda-runtime-cu12 12.8.90 

GPU:

NVIDIA RTX PRO 6000 Blackwell Workstation
Driver Version: 575.57.08
CUDA Version: 12.9

Quant script:

from datasets import load_dataset
from gptqmodel import GPTQModel, QuantizeConfig

model_id = "Qwen/Qwen3-Omni-30B-A3B-Instruct"
quant_path = "Qwen3-Omni-30B-A3B-Instruct-GPTQ-4bit"

calibration_dataset = load_dataset(
    "allenai/c4",
    data_files="en/c4-train.00001-of-01024.json.gz",
    split="train"
  ).select(range(1024))["text"]

quant_config = QuantizeConfig(bits=4, group_size=128)

model = GPTQModel.load(model_id, quant_config)

# increase `batch_size` to match gpu/vram specs to speed up quantization
model.quantize(calibration_dataset, batch_size=1)

model.save(quant_path)

Also when batch_size != 1 I get:

  File "/venv/main/lib/python3.12/site-packages/transformers/models/qwen3_omni_moe/modeling_qwen3_omni_moe.py", line 1462, in forward
    attn_output, attn_weights = attention_interface(
                                ^^^^^^^^^^^^^^^^^^^^
  File "/venv/main/lib/python3.12/site-packages/transformers/integrations/sdpa_attention.py", line 96, in sdpa_attention_forward
    attn_output = torch.nn.functional.scaled_dot_product_attention(
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: The expanded size of the tensor (17640) must match the existing size (8820) at non-singleton dimension 3.  Target sizes: [2, 32, 8820, 17640].  Tensor sizes: [2, 1, 8820, 8820]

[Qubitium]

@tommyip Your dataset contains single row size exceeding 40k in size. Loop over your dataset rows and only select 1024 that is at or below 2048 or 4096 or 8192 in length or something sane. 40K will blow up inferene vram.

[ZX-ModelCloud]

The maximum text length of the calibration_dataset you're using is 40123. This would take up too much memory.
You can limit the maximum length.

from datasets import load_dataset
from gptqmodel import GPTQModel, QuantizeConfig

model_id = "Qwen/Qwen3-Omni-30B-A3B-Instruct"
quant_path = "Qwen3-Omni-30B-A3B-Instruct-GPTQ-4bit"

calibration_dataset = load_dataset(
    "allenai/c4",
    data_files="en/c4-train.00001-of-01024.json.gz",
    split="train"
  )

calibration_dataset = calibration_dataset.filter(lambda x: len(x["text"]) <= 8192)
calibration_dataset = calibration_dataset.select(range(1024))["text"]

quant_config = QuantizeConfig(bits=4, group_size=128)

model = GPTQModel.load(model_id, quant_config)

# increase `batch_size` to match gpu/vram specs to speed up quantization
model.quantize(calibration_dataset, batch_size=1)

model.save(quant_path)

[tommyip]

Thanks. I got it to quantize around half the model but still hit OOM at [22/48] step even at 2048 row size. The vram usage grows slowly overtime - is this suppose to happen?
Did you managed to run the full quant process with the A100?

(Edit: maybe same memory leak as #1686?)

[Qubitium]

Please send us the logs, not oom stacktrace, the actual logs preceding the oom. plus the oom stacktrace as well. If we can see the logs, we can't tell. The terminal logs that shows step, error loss, module name, vram usage per gpu, etc. It has tons of info we need to debug. We can't just go by oom. We know to now exactly where down to the module.

[tommyip]

Stdout/err logs on my RTX 5090. OOMed at layer 3.

gptqmodel.log

[allerou4]

Hi, I encountered the same issue, I have four cards L20 with 48GB×4

and blow is the last couple of lines of the log, the distribution of memory usage is not balanced.

By the way, loss": "999999999.0000000000 means the quantization is bad?

{
"process": "gptq",
"layer": 43,
"module": "mlp.experts.4.down_proj",
"feat: in, out": "768, 2048",
"dtype: size": "bf16: 3.1MB",
"loss": "999999999.0000000000",
"samples": "66189",
"damp": "1.00000",
"time": "2.013",
"fwd_time": "94.184",
"(v)ram": "cuda 44.1G, 21.5G, 12.1G, 3.7G"
}
{
"process": "gptq",
"layer": 43,
"module": "mlp.experts.2.down_proj",
"feat: in, out": "768, 2048",
"dtype: size": "bf16: 3.1MB",
"loss": "999999999.0000000000",
"samples": "66189",
"damp": "1.00000",
"time": "2.018",
"fwd_time": "94.184",
"(v)ram": "cuda 44.1G, 21.5G, 12.1G, 3.7G"
}