Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix missing scale attributes for GPTJ #3256

Merged
merged 7 commits into from
Apr 21, 2023
Merged

Fix missing scale attributes for GPTJ #3256

Changes from 1 commit
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
565995a
Fix missing scale attributes
cmikeh2 Apr 15, 2023
a488488
Fix double usage of scratch space
cmikeh2 Apr 17, 2023
49f9d5b
Merge branch 'master' into cholmes/gptj-weight-scale-fix
cmikeh2 Apr 17, 2023
4c2b6c1
Merge branch 'master' into cholmes/gptj-weight-scale-fix
tjruwase Apr 19, 2023
0fd4a3f
Merge branch 'master' into cholmes/gptj-weight-scale-fix
mrwyattii Apr 20, 2023
4cc8faa
Merge branch 'master' into cholmes/gptj-weight-scale-fix
mrwyattii Apr 20, 2023
829496f
Merge branch 'master' into cholmes/gptj-weight-scale-fix
mrwyattii Apr 21, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Fix double usage of scratch space
  • Loading branch information
@cmikeh2
cmikeh2 committed Apr 17, 2023
commit a4884888321d1be052d32e21033788bba18ba8d0
4 changes: 2 additions & 2 deletions csrc/transformer/inference/csrc/pt_binding.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -462,9 +462,9 @@ std::vector<at::Tensor> ds_softmax_context(at::Tensor& query_key_value,

T* workspace = (T*)InferenceContext::Instance().GetWorkSpace();
size_t buf_size = bsz * seq_len * hidden_dim;
auto output = torch::from_blob(workspace + 3 * buf_size, {bsz, seq_len, hidden_dim}, options);
auto output = torch::from_blob(workspace + 4 * buf_size, {bsz, seq_len, hidden_dim}, options);

auto query_cont = workspace + 4 * buf_size;
auto query_cont = workspace + 5 * buf_size;
size_t offset =
10 * (hidden_dim * bsz * InferenceContext::Instance().GetMaxTokenLenght()) +
layer_id * 2 * bsz * InferenceContext::Instance().GetMaxTokenLenght() * hidden_dim;
Expand Down