Hi @mrwyattii @delock. Please kindly review, Thanks!

@Yejing-Lai what specific value of module_str and k caused the issue?

@Yejing-Lai what specific value of module_str and k caused the issue?
It is a logic error. The k is contained in module_str. The module_name_matches cannot get the correct fused_type now. This will cause all models to choose bloom_type.

if k in module_str does it mean k is a substring of module_str or k is an element of module_str as a list?

if k in module_str does it mean k is a substring of module_str or k is an element of module_str as a list?
Yes.

For example:
k = "CodeGenBlock"
module_str="
(module): CodeGenForCausalLM(
(transformer): CodeGenModel(
(wte): Embedding(51200, 2560)
(drop): Dropout(p=0.0, inplace=False)
(h): ModuleList(
(0-31): 32 x CodeGenBlock(
(ln_1): LayerNorm((2560,), eps=1e-05, elementwise_affine=True)
(attn): CodeGenAttention(
(attn_dropout): Dropout(p=0.0, inplace=False)
(resid_dropout): Dropout(p=0.0, inplace=False)
(qkv_proj): LinearLayer()
(out_proj): LinearAllreduce()
)
(mlp): CodeGenMLP(
(fc_in): LinearLayer()
(fc_out): LinearAllreduce()
(act): NewGELUActivation()
(dropout): Dropout(p=0.0, inplace=False)
)
)
)
(ln_f): LayerNorm((2560,), eps=1e-05, elementwise_affine=True)
)
(lm_head): LmHeadLinearAllreduce()
)
"
We need to judge if k in module_str rather than module_str in k.
Thanks.

Hi @mrwyattii can you help review this PR? This PR fixed an accuracy issue for various models with fused qkv. i.e. Baichuan, code gen, bloom, mpt.