Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support free-threaded Python 3.13 #925

Merged
merged 10 commits into from
Jan 15, 2025
+35 −8
Merged

Support free-threaded Python 3.13 #925

Show file tree
Hide file tree
Changes from 8 commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
87257a2
Support free-threaded Python 3.13
ngoldbaum Nov 25, 2024
fb0606b
fix issues with tox.ini
ngoldbaum Nov 25, 2024
b7f6fbd
add multithreading test
ngoldbaum Nov 26, 2024
f0f4c17
fix pep8
ngoldbaum Nov 26, 2024
95d99e1
fix ruff format
ngoldbaum Nov 26, 2024
9f49f5a
name linux CI run based on python VERSION
ngoldbaum Dec 6, 2024
b557966
clarify use of pytest-run-parallel marks
ngoldbaum Jan 14, 2025
1a26622
refactor test_multithreading
ngoldbaum Jan 14, 2025
7465f54
simplify test_multithreading
ngoldbaum Jan 14, 2025
46fa817
remove pytest-run-parallel use
ngoldbaum Jan 15, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 7 additions & 4 deletions .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ jobs:
PYTHON:
- {VERSION: "3.8", TOXENV: "py38"}
- {VERSION: "3.13", TOXENV: "py313"}
- {VERSION: "3.13t", TOXENV: "py313"}
MACOS:
- macos-13
- macos-latest
Expand All @@ -24,7 +25,7 @@ jobs:
- uses: actions/checkout@v4.2.2
- name: Setup python
id: setup-python
uses: actions/setup-python@v5.3.0
uses: quansight-labs/setup-python@v5.3.1
with:
python-version: ${{ matrix.PYTHON.VERSION }}
- uses: actions/cache@v4.2.0
Expand Down Expand Up @@ -53,12 +54,13 @@ jobs:
PYTHON:
- {VERSION: "3.8", TOXENV: "py38"}
- {VERSION: "3.13", TOXENV: "py313"}
- {VERSION: "3.13t", TOXENV: "py313"}
name: "Python ${{ matrix.PYTHON.VERSION }} on ${{ matrix.WINDOWS.WINDOWS }}"
steps:
- uses: actions/checkout@v4.2.2
- name: Setup python
id: setup-python
uses: actions/setup-python@v5.3.0
uses: quansight-labs/setup-python@v5.3.1
with:
python-version: ${{ matrix.PYTHON.VERSION }}
architecture: ${{ matrix.WINDOWS.ARCH }}
Expand Down Expand Up @@ -92,19 +94,20 @@ jobs:
- {VERSION: "3.11", TOXENV: "py311"}
- {VERSION: "3.12", TOXENV: "py312"}
- {VERSION: "3.13", TOXENV: "py313"}
- {VERSION: "3.13t", TOXENV: "py313"}
- {VERSION: "pypy-3.9", TOXENV: "pypy3"}
- {VERSION: "pypy-3.10", TOXENV: "pypy3"}

# MSRV
- {VERSION: "3.13", TOXENV: "py313", RUST_VERSION: "1.64.0"}
- {VERSION: "3.13", TOXENV: "py313", RUST_VERSION: "beta"}
- {VERSION: "3.13", TOXENV: "py313", RUST_VERSION: "nightly"}
name: "${{ matrix.PYTHON.TOXENV }} on linux, Rust ${{ matrix.PYTHON.RUST_VERSION || 'stable' }}"
name: "${{ matrix.PYTHON.VERSION }} on linux, Rust ${{ matrix.PYTHON.RUST_VERSION || 'stable' }}"
steps:
- uses: actions/checkout@v4.2.2
- name: Setup python
id: setup-python
uses: actions/setup-python@v5.3.0
uses: quansight-labs/setup-python@v5.3.1
with:
python-version: ${{ matrix.PYTHON.VERSION }}
- uses: actions/cache@v4.2.0
Expand Down
2 changes: 1 addition & 1 deletion README.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -285,7 +285,7 @@ Compatibility
-------------

This library should be compatible with py-bcrypt and it will run on Python
3.6+, and PyPy 3.
3.8+ (including free-threaded builds), and PyPy 3.

Security
--------
Expand Down
2 changes: 1 addition & 1 deletion src/_bcrypt/src/lib.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -182,7 +182,7 @@ fn kdf<'p>(
})
}

#[pyo3::pymodule]
#[pyo3::pymodule(gil_used = false)]
mod _bcrypt {
use pyo3::types::PyModuleMethods;

Expand Down
37 changes: 35 additions & 2 deletions tests/test_bcrypt.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,6 @@
import uuid
from concurrent.futures import ThreadPoolExecutor

import pytest

import bcrypt
Expand Down Expand Up @@ -171,7 +174,7 @@
]


def test_gensalt_basic(monkeypatch):
def test_gensalt_basic():
salt = bcrypt.gensalt()
assert salt.startswith(b"$2b$12$")

Expand Down Expand Up @@ -219,7 +222,7 @@ def test_gensalt_bad_prefix():
bcrypt.gensalt(prefix=b"bad")


def test_gensalt_2a_prefix(monkeypatch):
alex marked this conversation as resolved.
Show resolved Hide resolved
def test_gensalt_2a_prefix():
salt = bcrypt.gensalt(prefix=b"2a")
assert salt.startswith(b"$2a$12$")

Expand Down Expand Up @@ -464,6 +467,8 @@ def test_kdf_no_warn_rounds():
bcrypt.kdf(b"password", b"salt", 10, 10, True)


# Python warning state is global
@pytest.mark.thread_unsafe()
alex marked this conversation as resolved.
Show resolved Hide resolved
def test_kdf_warn_rounds():
with pytest.warns(UserWarning):
bcrypt.kdf(b"password", b"salt", 10, 10)
Expand Down Expand Up @@ -494,3 +499,31 @@ def test_2a_wraparound_bug():
)
== b"$2a$04$R1lJ2gkNaoPGdafE.H.16.1MKHPvmKwryeulRe225LKProWYwt9Oi"
)


# this test spawns threads and is slow, so don't run it in many threads
@pytest.mark.parallel_threads(1)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll be honest, I'm really struggling to understand the pytest-run-parallel docs on what this marker does.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right now I'm setting --parallel-threads=10 in the call to pytest in tox. WIth that setting, for each test, pytest-run-parallel creates a thread pool that, by default, simultaneously executes every test in the test suite in 10 threads.

pytest-run-parallel also additionally lets you repeatedly run the test for a number of iterations in each spawned thread. By default it runs the test once in each thread. I'm not using that feature in this PR.

The pytest.mark.parallel_threads(1) decorator overrides the global default we're setting in the tox call to pytest and tells pytest-run-parallel to spawn a single thread for this test. I'm also not overriding the default iterations, so it runs the test once, just like if pytest-run-parallel wasn't turned on in the test session.

I think the pytest-run-parallel docs might be clearer if the readme included some example code using concurrent.futures.ThreadPoolExecutor that is equivalent to what pytest-run-parallel is doing. Thanks for the feedback that the docs are unclear.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok. I'm now following.

pytest-run-parallel's behavior is not what I expected based on either its name or reading the README. What I thought it was doing was running all the tests in parallel, whereas what it's actually doing is running each test multiple times in parallel with itself.

Based on this, I think I'd actually recommend dropping pytest-run-parallel and simply having only test_multithreading, which exercises the same behavior in a much clearer way. (Seperately, I'd love it if pytest had in-process test parallelism).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(Seperately, I'd love it if pytest had in-process test parallelism).

Maybe free-threaded Python will change things, but making pytest truly thread safe would be a really big undertaking. I suspect someone will try in the next year or two, or maybe try writing a thread-safe pytest-compatible test runner. Either way - big undertaking!

What does work is to spawn threads inside each test, but not share any of the Pytest testing primitives between threads. This is more-or-less what pytest-run-parallel is doing. This works doing one test at a time, but actually running multiple tests (which might share marks and fixtures) will probably lead to bad times.

All that said - separate from performance and concurrency, I think it would be really useful to run multiple tests simultaneously to discover races and use of global state. I found several novel races in CPython working on PyO3 because cargo runs tests simultaneously in a thread pool.

def test_multithreading():
class User:
def __init__(self, pw):
self.salt = bcrypt.gensalt(4)
self.hash_ = bcrypt.hashpw(pw, self.salt)
self.key = bcrypt.kdf(pw, self.salt, 32, 50)
Copy link
Contributor Author

@ngoldbaum ngoldbaum Dec 6, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I chose these parameters to make this run in a reasonable time. Still, it's a very slow test compared with the rest of the test suite.

assert self.check(pw)

def check(self, pw):
return bcrypt.checkpw(pw, self.hash_)

# use UUIDs as both ID and passwords
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This comment no longer makes a ton of sense

num_users = 50
user_creator = ThreadPoolExecutor(max_workers=4)
pws = [uuid.uuid4().bytes for _ in range(num_users)]

futures = [user_creator.submit(User, pw) for pw in pws]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we need a user class at all, just a function that returns a tuple of values.


users = [future.result() for future in futures]

for pw, user in zip(pws, users):
assert bcrypt.hashpw(pw, user.salt) == user.hash_
assert user.check(pw)
assert bcrypt.kdf(pw, user.salt, 32, 50) == user.key
3 changes: 2 additions & 1 deletion tox.ini
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,10 +6,11 @@ extras =
tests
deps =
coverage
pytest-run-parallel
passenv =
RUSTUP_HOME
commands =
coverage run -m pytest --strict-markers {posargs}
coverage run -m pytest --parallel-threads=10 --strict-markers {posargs}
ngoldbaum marked this conversation as resolved.
Show resolved Hide resolved
coverage combine
coverage report -m --fail-under 100

Expand Down
Loading