Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add libpq interface for pipeline mode #89

Merged
merged 3 commits into from
Oct 1, 2021
Merged

Conversation

dlax
Copy link
Contributor

@dlax dlax commented Oct 1, 2021

First step towards #74 adding libpq bindings and extending CI checks with libpq 14.


Next working on implementing a Python interface; as a first (simple) draft, I was thinking of something like:

with conn.pipeline() as pipeline:
   cur.execute(...)
   cur2.execute(...)
   pipeline.sync()  # calls PQpipelineSync
   r1 = cur.fetchone()
   r2 = cur2.fetchall()

If I understand correctly, we need to keep track of cursors executing queries during a pipeline session and process their result when asked; which mean fetch*() methods would emit PQgetResult calls (instead of execute() currently). Since multiple cursors might be involved, we need a queue of active cursors to later bind pipeline results to their respective cursor.

@dvarrazzo
Copy link
Member

Hello, thank you for looking into this. I think it would be an amazing feature to provide.

As a first MR, please feel free to propose one only adding the psycopg.pq objects, because their implementation (which you already have is straighforward). What should be built on top of it will be much more open to discussion...

@dlax
Copy link
Contributor Author

dlax commented Oct 1, 2021

As a first MR, please feel free to propose one only adding the psycopg.pq objects, because their implementation (which you already have is straighforward).

Ok, sounds good.
Do you want only the first commit with bindings only? Or also libpq 14 installation in CI?

tests/pq/test_pipeline.py Show resolved Hide resolved
@dvarrazzo
Copy link
Member

Do you want only the first commit with bindings only? Or also libpq 14 installation in CI?

I think having a test in the CI is important but maybe let's have a few jobs with libpq 14 and a few with whatever the system throws in. Adding a few comments to the MR...

import psycopg
from psycopg import pq

pytestmark = pytest.mark.libpq(">= 14")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you please add tests to verify that on libpq < 14 the functions raise the expected exception? Thank you

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

dlax added 3 commits October 1, 2021 16:32
Bindings are conditional so that compilation works with libpq version
older than 14.

Add a couple of tests inspired by PostgreSQL's tests.
We add "libpq" matrix axis and select version 14 against various server
versions.
@dlax dlax changed the title add support for pipeline mode add libpq interface for pipeline mode Oct 1, 2021
@dlax dlax marked this pull request as ready for review October 1, 2021 14:45
@dvarrazzo dvarrazzo merged commit 23d6d5f into psycopg:master Oct 1, 2021
@dlax dlax deleted the pipeline branch October 2, 2021 10:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants