Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: Asynchronous connection error when using databases other than Postgres or SQLite #5555

Open
wants to merge 6 commits into
base: main
Choose a base branch
from

Conversation

MarceloNunesAlves
Copy link
Contributor

This PR introduces a new environment variable, LANGFLOW_DATABASE_DRIVER_ASYNC, to specify the database driver used for asynchronous database connections. This allows for greater flexibility in supporting different database systems without requiring code changes.

Motivation:

Currently, asynchronous database connections were limited to Postgres and SQLite. This restriction made it difficult to use Langflow with other database systems.

Below is the error message:

Screenshot from 2025-01-06 11-09-10

Changes:

  • Added the LANGFLOW_DATABASE_DRIVER_ASYNC environment variable.

  • Updated the asynchronous connection logic to dynamically select the appropriate database driver based on the value of this environment variable.

  • Added error handling for unsupported database drivers.

Example:

To connect to a MySQL database asynchronously, set the environment variable as follows:

LANGFLOW_DATABASE_DRIVER_ASYNC=mysql+aiomysql

@dosubot dosubot bot added size:S This PR changes 10-29 lines, ignoring generated files. bug Something isn't working labels Jan 6, 2025
@github-actions github-actions bot added bug Something isn't working and removed bug Something isn't working labels Jan 6, 2025
@dosubot dosubot bot added size:M This PR changes 30-99 lines, ignoring generated files. and removed size:S This PR changes 10-29 lines, ignoring generated files. labels Jan 6, 2025
@github-actions github-actions bot added bug Something isn't working and removed bug Something isn't working labels Jan 6, 2025
@ogabrielluiz ogabrielluiz requested a review from cbornet January 6, 2025 17:52
Copy link

codspeed-hq bot commented Jan 6, 2025

CodSpeed Performance Report

Merging #5555 will degrade performances by 58.3%

Comparing MarceloNunesAlves:fix/db_async_driver (fc82703) with main (16ff8eb)

Summary

⚡ 1 improvements
❌ 2 regressions
✅ 12 untouched benchmarks

⚠️ Please fix the performance issues or acknowledge them on CodSpeed.

Benchmarks breakdown

Benchmark main MarceloNunesAlves:fix/db_async_driver Change
test_get_and_cache_all_types_dict 1 ms 2.5 ms -58.3%
test_successful_run_with_input_type_text 271.3 ms 208 ms +30.44%
test_successful_run_with_output_type_any 260.3 ms 294.5 ms -11.61%

@cbornet
Copy link
Collaborator

cbornet commented Jan 7, 2025

Does it work ? I think to use mysql+aiomysql, you need the aiomysql package to be installed which is not the case in the default LangFlow.

@MarceloNunesAlves
Copy link
Contributor Author

Does it work ? I think to use mysql+aiomysql, you need the aiomysql package to be installed which is not the case in the default LangFlow.

Yes! You need to install the driver of your choice. And I did it this way because it is generic for other DB, such as Oracle and SQLServer.
I think the idea of the project has always been to allow the use of any relational database. And frankly, the way this method works with these different conditional structures, I don't think it's the most appropriate. but I kept it so as not to harm others who already use this version in production.

@qq745639151
Copy link
Contributor

qq745639151 commented Jan 13, 2025

I tried this solution . it doesn't work. this Exception is still raised.

sqlalchemy.exc.MissingGreenlet: greenlet_spawn has not been called; can't call await_only() here. Was IO attempted in an unexpected place? (Background on this error at: https://sqlalche.me/e/20/xd2s)

@MarceloNunesAlves
Copy link
Contributor Author

MarceloNunesAlves commented Jan 13, 2025

I tried this solution . it doesn't work. this Exception is still raised.

sqlalchemy.exc.MissingGreenlet: greenlet_spawn has not been called; can't call await_only() here. Was IO attempted in an unexpected place? (Background on this error at: https://sqlalche.me/e/20/xd2s)

OK! What did you put in the environment variables: LANGFLOW DATABASE_URL and LANGFLOW_DATABASE_DRIVER_ASYNC? Have you tested with my branch?

@qq745639151
Copy link
Contributor

I tried this solution . it doesn't work. this Exception is still raised.

sqlalchemy.exc.MissingGreenlet: greenlet_spawn has not been called; can't call await_only() here. Was IO attempted in an unexpected place? (Background on this error at: https://sqlalche.me/e/20/xd2s)

OK! What did you put in the environment variables: LANGFLOW DATABASE_URL and LANGFLOW_DATABASE_DRIVER_ASYNC? Have you tested with my branch?

I tried it on Windows 11 . I installed the langflow 1.1.1 and then I merge your code in it . Then I run it with this command:

python -m langflow run --env-file=.env

in the .envfile . I set the environment variables as follow:

LANGFLOW_DATABASE_DRIVER_ASYNC=mysql+aiomysql
LANGFLOW_DATABASE_URL=mysql+aiomysql://{my_user}:{mypassword}@{my_ip}:{my_port}/langflow_newest_dev

@qq745639151
Copy link
Contributor

qq745639151 commented Jan 14, 2025

I tried this solution . it doesn't work. this Exception is still raised.

sqlalchemy.exc.MissingGreenlet: greenlet_spawn has not been called; can't call await_only() here. Was IO attempted in an unexpected place? (Background on this error at: https://sqlalche.me/e/20/xd2s)

OK! What did you put in the environment variables: LANGFLOW DATABASE_URL and LANGFLOW_DATABASE_DRIVER_ASYNC? Have you tested with my branch?

I tried it on Windows 11 . I installed the langflow 1.1.1 and then I merge your code in it . Then I run it with this command:

python -m langflow run --env-file=.env

in the .envfile . I set the environment variables as follow:

LANGFLOW_DATABASE_DRIVER_ASYNC=mysql+aiomysql
LANGFLOW_DATABASE_URL=mysql+aiomysql://{my_user}:{mypassword}@{my_ip}:{my_port}/langflow_newest_dev

I fixed it. the engine mysql+aiomysql is not available to function inspect and run_migrations_online . we should use mysql+pymysql

@MarceloNunesAlves
Copy link
Contributor Author

@qq745639151,

Yes! That's why I asked you for both configurations, because one must use mysql+pymysql (For the non-asynchronous engine) and mysql+aiomysql (For the asynchronous engine). In your case the configuration would be like this:

LANGFLOW_DATABASE_DRIVER_ASYNC=mysql+aiomysql
LANGFLOW_DATABASE_URL=mysql+pymysql://{my_user}:{mypassword}@{my_ip}:{my_port}/langflow_newest_dev

"postgresql+psycopg://"
if url_components[0].startswith("postgresql")
else f"{self.database_driver_async}://"
if self.database_driver_async
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If provided, database_driver_async should probably be the first choice.

Comment on lines +73 to 75
"""Database driver async."""
database_driver_async: str | None = None
"""Database URL for Langflow. If not provided, Langflow will use a SQLite database."""
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Param description should be switched.

MarceloNunesAlves and others added 2 commits January 14, 2025 10:57
Co-authored-by: Christophe Bornet <cbornet@hotmail.com>
Co-authored-by: Christophe Bornet <cbornet@hotmail.com>
@github-actions github-actions bot added bug Something isn't working and removed bug Something isn't working labels Jan 14, 2025
@github-actions github-actions bot added the bug Something isn't working label Jan 14, 2025
@dosubot dosubot bot added size:S This PR changes 10-29 lines, ignoring generated files. and removed size:M This PR changes 30-99 lines, ignoring generated files. labels Jan 14, 2025
@github-actions github-actions bot added bug Something isn't working and removed bug Something isn't working labels Jan 14, 2025
@qq745639151
Copy link
Contributor

@qq745639151,

Yes! That's why I asked you for both configurations, because one must use mysql+pymysql (For the non-asynchronous engine) and mysql+aiomysql (For the asynchronous engine). In your case the configuration would be like this:

LANGFLOW_DATABASE_DRIVER_ASYNC=mysql+aiomysql LANGFLOW_DATABASE_URL=mysql+pymysql://{my_user}:{mypassword}@{my_ip}:{my_port}/langflow_newest_dev

OK, Thank you for your help ! I understand it.

@qq745639151
Copy link
Contributor

@qq745639151,
Yes! That's why I asked you for both configurations, because one must use mysql+pymysql (For the non-asynchronous engine) and mysql+aiomysql (For the asynchronous engine). In your case the configuration would be like this:
LANGFLOW_DATABASE_DRIVER_ASYNC=mysql+aiomysql LANGFLOW_DATABASE_URL=mysql+pymysql://{my_user}:{mypassword}@{my_ip}:{my_port}/langflow_newest_dev

OK, Thank you for your help ! I understand it.

the engine problem is solved. there is another problem .

sqlalchemy.exc.OperationalError: (pymysql.err.OperationalError) (1170, "BLOB/TEXT column 'description' used in key specification without a key length")
[SQL: CREATE INDEX ix_flow_description ON flow (description)]

I fixed it in 260dbcc8b680_adds_tables.py
first import text

from sqlalchemy.sql.expression import text

then change the code in line 116:
the origin code:

batch_op.create_index(batch_op.f("ix_flow_description"), ["description"], unique=False)

the modified code

batch_op.create_index(batch_op.f("ix_flow_description"),[text("description(10)")], unique=False)

I specified a length for the description column. but I don't know if that length is appropriate for the index.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working size:S This PR changes 10-29 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants