Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add RMSNorm #834

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
Open

Add RMSNorm #834

wants to merge 2 commits into from

Conversation

ZiyueHuang
Copy link

What does this PR do?

RMSNorm is used in many popular models, e.g., ChatGLM2, Baichuan, Llama.

Made many duplicate codes with triton/k_layer_norm.py for clarity and a neat implementation. Would it be better if reorganizing LayerNorm and RMSNorm into a base class?

Before submitting

  • Did you have fun?
    • Make sure you had fun coding 🙃
  • Did you read the contributor guideline?
  • Was this discussed/approved via a Github issue? (no need for typos, doc improvements)
    • N/A
  • Did you make sure to update the docs?
    • N/A
  • Did you write any new necessary tests?
    • N/A
  • Did you update the changelog? (if needed)
    • N/A

PR review

Anyone in the community is free to review the PR once the tests have passed.
If we didn't discuss your PR in Github issues there's a high chance it will not be merged.

@facebook-github-bot facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Aug 26, 2023
@sebhtml
Copy link

sebhtml commented Nov 12, 2024

image

Can you double-check if you had fun ?

bertmaher pushed a commit to bertmaher/xformers that referenced this pull request Dec 20, 2024
@lw
Make names of Triton kernel functions more descriptive (facebookresea…
b2e8ed8 
…rch#834)

Since these names are the ones that will be displayed in the profiler traces
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@ZiyueHuang @sebhtml @facebook-github-bot