feat(flags): Add integration for custom tracking of flag evaluations #3860

aliu39 · 2024-12-05T19:57:14Z

Follow-up to

Add LaunchDarkly and OpenFeature integration #3648

Adds an integration for buffering feature flags manually with an API (add_feature_flag), and automatically capturing them on error events.
Adds improved test coverage to launchdarkly and openfeature by testing end to end with capture_events fixture, rather than asserting the flags are stored to scope.

codecov · 2024-12-05T20:01:54Z

❌ 1 Tests Failed:

Tests completed	Failed	Passed	Skipped
13801	1	13800	4139

View the top 1 failed tests by shortest run time

tests.integrations.launchdarkly.test_launchdarkly test_launchdarkly_integration[False]

Stack Traces | 0.337s run time

.../integrations/launchdarkly/test_launchdarkly.py:48: in test_launchdarkly_integration
    assert len(events) == 1
E   AssertionError: assert 2 == 1
E    +  where 2 = len([{'breadcrumbs': {'values': []}, 'contexts': {'runtime': {'build': '3.12.8 (main, Dec  4 2024, 06:20:10) [GCC 9.4.0]', 'name': 'CPython', 'version': '3.12.8'}, 'trace': {'parent_span_id': None, 'span_id': '91e2f801598b1cc5', 'trace_id': 'ec3ee1fd20f64251aee47ef1bfa3b656'}}, 'environment': 'production', 'event_id': '22cbdbda740f4333bc4a24d9477cd58f', ...}, {'breadcrumbs': {'values': []}, 'contexts': {'flags': {'values': [{'flag': 'hello', 'result': True}, {'flag': 'world', 'result': True}, {'flag': 'other', 'result': False}]}, 'runtime': {'build': '3.12.8 (main, Dec  4 2024, 06:20:10) [GCC 9.4.0]', 'name': 'CPython', 'version': '3.12.8'}, 'trace': {'parent_span_id': None, 'span_id': '91e2f801598b1cc5', 'trace_id': 'ec3ee1fd20f64251aee47ef1bfa3b656'}}, 'environment': 'production', 'event_id': 'b5b7269cd45740ad89573390823dbcbd', ...}])

To view more test analytics, go to the Test Analytics Dashboard
📢 Thoughts on this report? Let us know!

sentry_sdk/integrations/featureflags.py

tests/integrations/featureflags/test_featureflags.py

tests/test_lru_cache.py

sentry_sdk/_lru_cache.py

sentry_sdk/integrations/featureflags.py

tests/integrations/featureflags/test_featureflags.py

aliu39 · 2024-12-06T20:20:31Z

tests/integrations/featureflags/test_featureflags.py

+
+
+@pytest.fixture
+def uninstall_integration():


Note on this fixture: I found using the reset_integrations fixture in conftest.py caused issues with the default dedupe integration. We just want the integration being tested to be setup at the start of every new test.

Without using this, the error processor isn't added correctly. This isn't a concern for real use, since user apps only call init once

antonpirker

Looks OK

antonpirker · 2024-12-18T08:11:21Z

We also need to add docs for this, can you prepare a PR in the docs repo?

aliu39 · 2024-12-18T08:14:07Z

Sounds good! I've opened getsentry/sentry-docs#12152, will do a last check in the morning to make sure it's up to date!

aliu39 · 2024-12-18T08:14:29Z

Do you know anything about the failing ubuntu tests @antonpirker ?

antonpirker · 2024-12-18T08:19:12Z

Do you know anything about the failing ubuntu tests @antonpirker ?

In Python 3.6 there is probably no asyncio.run().

Can you fix this somehow, or skip this test in Python 3.6 (then this integration will only be supported form Python 3.7 and up)

aliu39 added 3 commits December 4, 2024 14:38

Add new integration and unit tests

3652ce3

Test flag values for LD and OF threaded/asyncio, not just flag names

625969e

update ffIntegration test to be e2e, and fix LRU copy bug

beb9512

aliu39 requested a review from cmanallen December 5, 2024 19:57

cmanallen requested changes Dec 5, 2024

View reviewed changes

sentry_sdk/integrations/featureflags.py Show resolved Hide resolved

tests/integrations/featureflags/test_featureflags.py Outdated Show resolved Hide resolved

tests/test_lru_cache.py Outdated Show resolved Hide resolved

sentry_sdk/_lru_cache.py Outdated Show resolved Hide resolved

aliu39 changed the title ~~Aliu/ff integration and added coverage~~ feat(flags): Add integration for custom tracking of flag evaluations Dec 5, 2024

aliu39 added 3 commits December 6, 2024 10:28

Merge branch 'master' into aliu/ff-integration-and-added-coverage

296545d

make a helper fixture and test error processor in original thread

921b133

Move api to top-level, rename to add_flag

4651b6a

aliu39 force-pushed the aliu/ff-integration-and-added-coverage branch from 320e49e to 4651b6a Compare December 6, 2024 19:05

aliu39 added 4 commits December 6, 2024 11:19

Add docstrs

36bc869

Rename to add_feature_flag

81c1761

Rm extra import in test_lru_cache

42f76a3

Revert lru comment

ef01990

aliu39 requested a review from cmanallen December 6, 2024 19:31

aliu39 marked this pull request as ready for review December 6, 2024 19:31

Type annotate

381ccc1

aliu39 requested review from antonpirker and a team December 6, 2024 19:57

cmanallen requested changes Dec 6, 2024

View reviewed changes

sentry_sdk/integrations/featureflags.py Outdated Show resolved Hide resolved

tests/integrations/featureflags/test_featureflags.py Outdated Show resolved Hide resolved

Review comments

c76192e

aliu39 requested a review from cmanallen December 6, 2024 20:18

cmanallen approved these changes Dec 6, 2024

View reviewed changes

aliu39 commented Dec 6, 2024

View reviewed changes

Update launchdarkly and openfeature tests to be e2e

b63982b

aliu39 requested a review from a team December 10, 2024 21:36

billyvg mentioned this pull request Dec 11, 2024

Create an RFC for the SDK and protocol where we can capture FF with events getsentry/sentry#77041

Closed

Merge branch 'master' into aliu/ff-integration-and-added-coverage

af1128c

aliu39 mentioned this pull request Dec 16, 2024

feat(flags): document python FeatureFlagsIntegration for custom flag tracking getsentry/sentry-docs#12152

Closed

Merge branch 'master' into aliu/ff-integration-and-added-coverage

4c6f08a

antonpirker approved these changes Dec 18, 2024

View reviewed changes

Update docstrs

b4eb421

aliu39 added 7 commits December 18, 2024 09:58

Skip threading test for <3.7

043e298

Merge branch 'master' into aliu/ff-integration-and-added-coverage

41fdde5

Skip ffs asyncio test if 3.6

2dc679d

undo 'skip threading test'

af9e92d

Try commenting out asyncio

2cea37b

Use importorskip

fea761c

Import order

752ce7d

aliu39 enabled auto-merge (squash) December 19, 2024 04:00

aliu39 disabled auto-merge December 19, 2024 04:00

aliu39 enabled auto-merge (squash) December 19, 2024 04:01

Also use skipif

d77ebc3

aliu39 merged commit 50222ca into master Dec 19, 2024
134 of 135 checks passed

aliu39 deleted the aliu/ff-integration-and-added-coverage branch December 19, 2024 04:33

aliu39 mentioned this pull request Jan 6, 2025

ref(flags): underscore featureflags module and identifier #3902

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(flags): Add integration for custom tracking of flag evaluations #3860

feat(flags): Add integration for custom tracking of flag evaluations #3860

aliu39 commented Dec 5, 2024 •

edited

Loading

codecov bot commented Dec 5, 2024 •

edited

Loading

aliu39 Dec 6, 2024 •

edited

Loading

antonpirker left a comment

antonpirker commented Dec 18, 2024

aliu39 commented Dec 18, 2024

aliu39 commented Dec 18, 2024

antonpirker commented Dec 18, 2024



		@pytest.fixture
		def uninstall_integration():

feat(flags): Add integration for custom tracking of flag evaluations #3860

feat(flags): Add integration for custom tracking of flag evaluations #3860

Conversation

aliu39 commented Dec 5, 2024 • edited Loading

codecov bot commented Dec 5, 2024 • edited Loading

❌ 1 Tests Failed:

aliu39 Dec 6, 2024 • edited Loading

Choose a reason for hiding this comment

antonpirker left a comment

Choose a reason for hiding this comment

antonpirker commented Dec 18, 2024

aliu39 commented Dec 18, 2024

aliu39 commented Dec 18, 2024

antonpirker commented Dec 18, 2024

aliu39 commented Dec 5, 2024 •

edited

Loading

codecov bot commented Dec 5, 2024 •

edited

Loading

aliu39 Dec 6, 2024 •

edited

Loading