Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(generator): extend construction to any langchain LLM and Embeddings #670

Merged
merged 13 commits into from
Mar 12, 2024

Conversation

mspronesti
Copy link
Contributor

@mspronesti mspronesti commented Feb 27, 2024

User description

The current version of with_openai contains a hardcoded instantiation of langchain_openai.chat_models.ChatOpenAI, which makes TestsetGenerator very limited and not compatible with completion models, Azure OpenAI models, and open-source models.

This PR extends TestsetGenerator to any BaseLanguageModel and Embeddings from langchain for versatility, addressing #230, #342, #635, and #636.

Lastly, I've removed all the occurrences of mutable default arguments (bad antipattern, read here).

@codeant-ai codeant-ai bot added the enhancement New feature or request label Feb 27, 2024
Copy link

codeant-ai bot commented Feb 27, 2024

PR Description updated to latest commit (ca53c81)

@mspronesti mspronesti marked this pull request as ready for review February 28, 2024 09:18
@mspronesti mspronesti changed the title feat(generator): extend construction to Azure OpenAI feat(generator): extend construction to any langchain LLM and Embeddings Feb 28, 2024
@mspronesti
Copy link
Contributor Author

Just finalized the PR - It's ready for review now :)

@shahules786
Copy link
Member

Wow, thanks man @mspronesti Will review it ASAP

@mspronesti
Copy link
Contributor Author

@shahules786 Thanks :)

@chin-jey
Copy link

chin-jey commented Feb 29, 2024

This works very well, thanks @mspronesti!

@mspronesti
Copy link
Contributor Author

Hi there @shahules786, any chance this PR gets reviewed? :)

@shahules786
Copy link
Member

Hi @mspronesti apologies for the late review. @jjmachan was testing this out. Will be reviewed soon.

@shahules786 shahules786 self-requested a review March 10, 2024 23:13
@shahules786
Copy link
Member

Hi @mspronesti just finished testing and making some changes to avoid breaking changes.
I'm not sure if you tested it on any instruct models, but the changes made don't allow the use of instruct models.
Also if you have time can you remove the documentation changes that were made inorder to tackle the breaking change (with_openai)?

@jjmachan you can do the review now.

Copy link
Member

@shahules786 shahules786 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove all changes made to tackle the removal of (with_openai) function

@mspronesti
Copy link
Contributor Author

mspronesti commented Mar 11, 2024

@shahules786 When I split this PR into two (the second part being #684) I didn't realize that completion models relied on the fact that this parameter needs to default to None not to bump into an IndexError from langchain (besides the point of being a mutable default).

I have appended one additional commit. Please try again :)

Regarding the docs, my suggestion would be to mark with_openai for deprecation (e.g., with a decorator) and discontinue it in the next major release. As such, I would advise showcasing the new usage in the docs, not the old one :)

Lastly, I find the name from_default ambiguous. I believe from_langchain was accurate and pythonic. What do you think?

@jjmachan
Copy link
Member

hey @mspronesti - I agree from_defaults might be ambiguous so lets keep it as from_langchain itself as you first proposed.

as for deprecation I've added the code in this PR but let me think we have to remove it in this version

and for the completion bug - nice catch seems like it fixed the issue ❤️

@jjmachan jjmachan requested a review from shahules786 March 12, 2024 07:35
@mspronesti
Copy link
Contributor Author

Hi @jjmachan, thanks for the review :)

I wouldn't remove with_openai in this version, but in the next major release (0.2). I'd decorate it as deprecated now, suggesting from_lanchain as alternative. What do you think? :)

@mspronesti
Copy link
Contributor Author

@jjmachan I've just added an extra commit to make the deprecation decorator more versatile and informative, and I've used it on with_openai to mark its deprecation as "pending" so that you can schedule that whenever convenient.

@jjmachan
Copy link
Member

jjmachan commented Mar 12, 2024

this is awesome @mspronesti - I love the depreciation util func - handy!
Once @shahules786 unblocks us we should be good to go and do a release later today as well

btw would love to hop on a call with you sometime if you are interested @mspronesti 🙂

@shahules786
Copy link
Member

@mspronesti Yes, that actually makes sense. Let's do that. I'll also add some more docs to help community use non-open ai models with it.

@jjmachan jjmachan merged commit 746d723 into explodinggradients:main Mar 12, 2024
14 of 15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants