Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[cli/paraformer] ali-paraformer inference #2067

Merged
merged 21 commits into from
Oct 30, 2023
Merged

[cli/paraformer] ali-paraformer inference #2067

merged 21 commits into from
Oct 30, 2023

Conversation

Mddct
Copy link
Collaborator

@Mddct Mddct commented Oct 20, 2023 •
edited
Loading

TODO: (in this pr)

  • export jit work
  • 简化代码
  • model.decode,
  • 支持cli
  • 精度校验

TODO： （future pr）

NOTE： streaming paraformer not in current plan

@Mddct Mddct changed the title [cli/paraformer] ali-paraformer load and infer work [cli/paraformer] ali-paraformer load and infer work [WIP] Oct 20, 2023
@Mddct Mddct force-pushed the Mddct-cli-paraformer branch from 579e9ff to 6967ef5 Compare October 20, 2023 07:13
@Mddct Mddct marked this pull request as ready for review October 20, 2023 07:14
@robin1001 robin1001 mentioned this pull request Oct 20, 2023
Closed
12 tasks
@Mddct Mddct force-pushed the Mddct-cli-paraformer branch from c3ca2e8 to 87906d5 Compare October 23, 2023 10:29
@Mddct Mddct force-pushed the Mddct-cli-paraformer branch from 87906d5 to c8cccdc Compare October 23, 2023 11:05
@Mddct Mddct force-pushed the Mddct-cli-paraformer branch from 3151899 to e8fa013 Compare October 23, 2023 11:56
@Mddct Mddct force-pushed the Mddct-cli-paraformer branch from 0658057 to e987b00 Compare October 23, 2023 12:29
@Mddct Mddct force-pushed the Mddct-cli-paraformer branch from 7ed212b to 3d25e2e Compare October 23, 2023 16:34
@Mddct
Copy link
Collaborator Author

Mddct commented Oct 24, 2023

Screenshot 2023-10-24 at 18 37 25 cli work

@Mddct Mddct changed the title [cli/paraformer] ali-paraformer load and infer work [WIP] [cli/paraformer] ali-paraformer inference Oct 24, 2023
@Mddct
Copy link
Collaborator Author

Mddct commented Oct 30, 2023 •
edited
Loading

mmexport1698658547189.jpg

recognize.py works

@Mddct
Copy link
Collaborator Author

Mddct commented Oct 30, 2023 •
edited
Loading

decode info: batch_size=100， beam_size=10
aishell:

model greedy_search beam_search
wenet-ali-paraformer 1.96 1.96

decode info: batch_size=1， beam_size=10
aishell:

model greedy_search beam_search
wenet-ali-paraformer 1.95 1.95
funasr-paraformer 1.95 /

@Mddct
Copy link
Collaborator Author

Mddct commented Oct 30, 2023 •
edited
Loading

Screenshot 2023-10-30 at 22 34 12 confidence in search.py works

TODO(next):

  • resolve conflict to merge main

@Mddct
Copy link
Collaborator Author

Mddct commented Oct 30, 2023

Screenshot 2023-10-30 at 23 01 01

it works!

@robin1001
Copy link
Collaborator

assets 是必须的吗？是否可以放到模型中？

robin1001
robin1001 reviewed Oct 30, 2023
View reviewed changes
wenet/cli/transcribe.py Outdated Show resolved Hide resolved
@Mddct
Copy link
Collaborator Author

Mddct commented Oct 30, 2023 •
edited
Loading

assets 是必须的吗？是否可以放到模型中？

目前cmvn 这些已经在模型里边力，不过funasr的conf和wenet conf 格式不一样

导出模型的时候，需要用assets里边的文件，目前是我脚本转的

可以等后边自动从funasr里边的conf力转成wenet格式，然后再删掉

@Mddct Mddct force-pushed the Mddct-cli-paraformer branch from bd2ea92 to b765c12 Compare October 30, 2023 15:38
@robin1001 robin1001 merged commit af1315c into main Oct 30, 2023
5 of 6 checks passed
@robin1001 robin1001 deleted the Mddct-cli-paraformer branch October 30, 2023 15:39
@xingchensong xingchensong mentioned this pull request Nov 1, 2023
Open
14 tasks
@Mddct Mddct mentioned this pull request Nov 18, 2023
Open
16 tasks
@Mddct Mddct mentioned this pull request Jan 18, 2024
Closed
3 tasks
@donstang
Copy link

streaming paraformer 有计划支持吗？

@Mddct
Copy link
Collaborator Author

Mddct commented Apr 29, 2024

streaming paraformer 有计划支持吗？

暂时没计划， streaming的paraformer指标上差了一些 也不是理想中的流模型，

感兴趣的话 可以请关注wenetspeech2.0的进展

TeaPoly
TeaPoly reviewed Jun 2, 2024
View reviewed changes
wenet/cli/paraformer_model.py
def transcribe(self, audio_file: str, tokens_info: bool = False) -> dict:
waveform, sample_rate = torchaudio.load(audio_file, normalize=False)
waveform = waveform.to(torch.float)
feats = kaldi.fbank(waveform,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The default window in the FunASR frontend is hamming. You can find more details here. However, the default window in kaldi.fbank is povey, as specified here. This different window maybe a little mismatch. As mentioned in line 44 of this document:

"povey" is a window I made to be similar to Hamming but to go to zero at the edges, it's pow((0.5 - 0.5cos(n/N2*pi)), 0.85) I just don't think the Hamming window makes sense as a windowing function.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pr welcome

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@TeaPoly TeaPoly TeaPoly left review comments

@robin1001 robin1001 robin1001 approved these changes

@xingchensong xingchensong Awaiting requested review from xingchensong

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@Mddct @robin1001 @donstang @TeaPoly