Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add tenant context #2596

Merged
merged 23 commits into from
Oct 10, 2024
Merged

Add tenant context #2596

Changes from 1 commit
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
53b72ba
add proper tenant context to background tasks
pablonyx Sep 29, 2024
35ede0f
update for new session logic
pablonyx Sep 29, 2024
be1caab
remove unnecessary functions
pablonyx Sep 29, 2024
0983d83
add additional tenant context
pablonyx Sep 29, 2024
6aea755
update ports
pablonyx Sep 29, 2024
2acc6bf
proper format / directory structure
pablonyx Sep 30, 2024
27095b5
update ports
pablonyx Sep 30, 2024
b8f42bd
ensure tenant context properly passed to ee bg tasks
pablonyx Oct 1, 2024
bc42cfa
add user provisioning
pablonyx Oct 8, 2024
7cb58ee
nit
pablonyx Oct 8, 2024
62efe54
validated for multi tenant
pablonyx Oct 8, 2024
e62357f
auth
pablonyx Oct 8, 2024
58b429c
nit
pablonyx Oct 8, 2024
613ea34
nit
pablonyx Oct 8, 2024
0bf512d
nit
pablonyx Oct 8, 2024
a19d7d7
nit
pablonyx Oct 9, 2024
dcb97cc
validate pruning
pablonyx Oct 9, 2024
24e2c7e
evaluate integration tests
pablonyx Oct 9, 2024
9ea11a8
at long last, validated celery beat
pablonyx Oct 10, 2024
1321ba0
nit: minor edge case patched
pablonyx Oct 10, 2024
a645c0c
minor
pablonyx Oct 10, 2024
a4dad03
validate update
pablonyx Oct 10, 2024
4ec418c
nit
pablonyx Oct 10, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
nit
  • Loading branch information
@pablonyx
pablonyx committed Oct 10, 2024
commit 0bf512dac8387b59fef083f7e0bba5c991ff0b81
13 changes: 10 additions & 3 deletions backend/danswer/background/celery/tasks/pruning/tasks.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -142,7 +142,9 @@ def try_creating_prune_generator_task(
celery_app.send_task(
"connector_pruning_generator_task",
kwargs=dict(
connector_id=cc_pair.connector_id, credential_id=cc_pair.credential_id, tenant_id=tenant_id
connector_id=cc_pair.connector_id,
credential_id=cc_pair.credential_id,
tenant_id=tenant_id,
),
queue=DanswerCeleryQueues.CONNECTOR_PRUNING,
task_id=custom_task_id,
Expand All @@ -153,8 +155,11 @@ def try_creating_prune_generator_task(
r.set(rcp.fence_key, 1)
return 1


@shared_task(name="connector_pruning_generator_task", soft_time_limit=JOB_TIMEOUT)
def connector_pruning_generator_task(connector_id: int, credential_id: int, tenant_id: str | None) -> None:
def connector_pruning_generator_task(
connector_id: int, credential_id: int, tenant_id: str | None
) -> None:
"""connector pruning task. For a cc pair, this task pulls all document IDs from the source
and compares those IDs to locally stored documents and deletes all locally stored IDs missing
from the most recently pulled document ID list"""
Expand Down Expand Up @@ -219,7 +224,9 @@ def redis_increment_callback(amount: int) -> None:
task_logger.info(
f"RedisConnectorPruning.generate_tasks starting. cc_pair_id={cc_pair.id}"
)
tasks_generated = rcp.generate_tasks(celery_app, db_session, r, None, tenant_id)
tasks_generated = rcp.generate_tasks(
celery_app, db_session, r, None, tenant_id
)
if tasks_generated is None:
return None

Expand Down