[RFC]: Deprecate Legacy Quantization Formats

[robertgshaw2-redhat]

Motivation.

• vLLM supports a large variety of quantization formats. This is hard to maintain and makes the codebase complex
• Many mature frameworks (llm-compressor, modelopt, quark, torchao) have emerged which are general purpose implementations of various quantization schemes
• we have limited usage of older formats per usage stats

Proposed Change.

• deprecate many of the legacy formats

Kept:

• compressed-tensors
• quark
• awq.py (to be deprecated later, too many models exist though --- autoawq no longer maintained)
• bitsandbytes.py
• fp8.py
• quark
• mxfp4.py
• modelopt.py
• gguf.py
• gptq.py (to be deprecated later, too many models exists though) --- autogptq no longer maintained)
• torchao.py

Proposed to be removed (per usage stats):

• auto_round
• awq_marlin (consolidate to awq.py)
• awq_triton (consolidate to awq.py)
• bitblas.py
• cpu_wna16.py
• deepseepfp8.py
• experts_int8.py
• fbgemm_fp8.py
• fp_quant.py
• gptq_bitblas.py
• gptq_marlin.py (consolidate to gptq.py)
• gptq_marlin_24.py
• hqq_marlin.py
• inc.py
• input_quant_fp8.py
• ipex_quant.py
• moe_wna16.py
• petit.py
• ptpc_fp8.py
• rtn.py
• tpu_int8.py

Feedback Period.

2 Weeks

CC List.

@mgoin @pavanimajety

Any Other Things.

The goal is to clean up the codebase:

• reduce mental load
• reduce complexity of implementing features (e.g. FusedMoE refactor)

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[robertgshaw2-redhat]

Please let me know if this is a bad idea! Legacy formats can be changed to plugins or models can be requantized using many of the frameworks above

[tjtanaa]

@robertgshaw2-redhat should we keep the 2:4 sparsity feature? Feels like it will be very important when scaling dense compute becomes harder.

[Isotr0py]

Proposed to be removed (per usage stats):

• ipex_quant.py

I think CPU/XPU can only support quantization through ipex. So perhaps we need to keep this feature for these backends as well.
cc @bigPYJ1151 @jikunshang

[robertgshaw2-redhat]

@robertgshaw2-redhat should we keep the 2:4 sparsity feature? Feels like it will be very important when scaling dense compute becomes harder.

There are ~0 models on the HF hub using the marlin 24 feature and they can only be created via a custom fork of autogptq that I made. This is exactly the type of feature that we should not keep

[robertgshaw2-redhat]

Proposed to be removed (per usage stats):

• ipex_quant.py

I think CPU/XPU can only support quantization through ipex. So perhaps we need to keep this feature for these backends as well. cc @bigPYJ1151 @jikunshang

Ack. Perhaps we can move this into the compressed-tensors implementation

[simon-mo]

+1 on plugin or consolidation into compressed tensors. Additionally, should we organize some of these by data type?

[robertgshaw2-redhat]

+1 on plugin or consolidation into compressed tensors. Additionally, should we organize some of these by data type?

the files refer to the "quantization format"

as part of the quant/fused moe refactor, I plan to split kernels and quantization format into separate concerns

[jikunshang]

Proposed to be removed (per usage stats):

• ipex_quant.py

I think CPU/XPU can only support quantization through ipex. So perhaps we need to keep this feature for these backends as well. cc @bigPYJ1151 @jikunshang

Actually, we are retiring ipex. CPU already removed ipex dependency and XPU will remove and migrate to vllm-xpu-kernels in 26Q1. we need keep ipex_quant here for xpu for a wheel.
But I think we will always need some hardware specific quant config/method to handle their own issue(like current tpu_int8.py). I am not clear what the shape will be.

[zhuohan123]

Thanks for the survey here! We really need to deprecate more things in the code so that the code becomes more managable.