Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Feature/log despam #2022

Merged
merged 2 commits into from
Aug 2, 2024
Merged

Feature/log despam #2022

merged 2 commits into from
Aug 2, 2024

Conversation

rkuo-danswer
Copy link
Contributor

@rkuo-danswer rkuo-danswer commented Aug 1, 2024 •
edited
Loading

Description

reorganizes info level logging during indexing to be more focused around normal indexing operation and progress.

Fixes DAN-360.

How Has This Been Tested?

Ran background indexing as usual and watched logging.

Accepted Risk

[Any know risks or failure modes to point out to reviewers]

Related Issue(s)

[If applicable, link to the issue(s) this PR addresses]

Checklist:

  • All of the automated tests pass
  • All PR comments are addressed and marked resolved
  • If there are migrations, they have been rebased to latest main
  • If there are new dependencies, they are added to the requirements
  • If there are new environment variables, they are added to all of the deployment methods
  • If there are new APIs that don't require auth, they are added to PUBLIC_ENDPOINT_SPECS
  • Docker images build and basic functionalities work
  • Author has done a final read through of the PR right before merge

@vercel Vercel
Copy link

vercel bot commented Aug 1, 2024

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name Status Preview Comments Updated (UTC)
internal-search ✅ Ready (Inspect) Visit Preview 💬 Add feedback Aug 1, 2024 11:51pm

@rkuo-danswer rkuo-danswer marked this pull request as ready for review August 2, 2024 01:12
@rkuo-danswer
Copy link
Contributor Author

Example logging:

08/01/2024 04:43:12 PM update.py 441 : Starting indexing service
08/01/2024 04:48:54 PM run_indexing.py 330 : [Attempt ID: 703] Indexing starting: connector='danswer-ai/danswer' config='{'repo_name': 'danswer', 'repo_owner': 'danswer-ai', 'include_prs': True, 'include_issues': True}' credentials='14'
08/01/2024 04:48:54 PM run_indexing.py 82 : [Attempt ID: 703] Polling for updates between 2024-08-01 23:08:38.035345+00:00 and 2024-08-01 23:48:54.454872+00:00
08/01/2024 04:49:05 PM timing.py 39 : [Attempt ID: 703] index_doc_batch took 0.011157035827636719 seconds
08/01/2024 04:49:05 PM run_indexing.py 281 : [Attempt ID: 703] Connector succeeded: docs=7 chunks=0 elapsed=10.68s
08/01/2024 04:49:05 PM run_indexing.py 339 : [Attempt ID: 703] Indexing finished: connector='danswer-ai/danswer' config='{'repo_name': 'danswer', 'repo_owner': 'danswer-ai', 'include_prs': True, 'include_issues': True}' credentials='14'
08/01/2024 04:49:05 PM update.py 335 : [Attempt ID: 703] Indexing dispatch starts: pending=1
08/01/2024 04:49:05 PM update.py 341 : [Attempt ID: 703] Indexing dispatched: connector='danswer-ai/danswer' config='{'repo_name': 'danswer', 'repo_owner': 'danswer-ai', 'include_prs': True, 'include_issues': True}' credentials='8'
08/01/2024 04:49:05 PM update.py 350 : [Attempt ID: 703] Indexing dispatch results: initial_pending=1 started=1 remaining=0
08/01/2024 04:49:05 PM run_indexing.py 330 : [Attempt ID: 704] Indexing starting: connector='WebConnector-https://docs.danswer.dev/introduction' config='{'base_url': 'https://docs.danswer.dev/introduction', 'web_connector_type': 'single'}' credentials='1'
08/01/2024 04:49:05 PM connector.py 263 : [Attempt ID: 704] Visiting https://docs.danswer.dev/introduction
08/01/2024 04:49:07 PM document.py 244 : [Attempt ID: 704] Upserted 1 document store entries into DB
08/01/2024 04:49:08 PM timing.py 39 : [Attempt ID: 704] index_doc_batch took 1.6120450496673584 seconds
08/01/2024 04:49:08 PM run_indexing.py 281 : [Attempt ID: 704] Connector succeeded: docs=1 chunks=2 elapsed=3.60s
08/01/2024 04:49:08 PM run_indexing.py 339 : [Attempt ID: 704] Indexing finished: connector='WebConnector-https://docs.danswer.dev/introduction' config='{'base_url': 'https://docs.danswer.dev/introduction', 'web_connector_type': 'single'}' credentials='1'

@yuhongsun96 yuhongsun96 added this pull request to the merge queue Aug 2, 2024
Merged via the queue into main with commit 6a61331 Aug 2, 2024
5 checks passed
@yuhongsun96 yuhongsun96 deleted the feature/log_despam branch August 2, 2024 21:08
@onimsha onimsha mentioned this pull request Sep 4, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@yuhongsun96 yuhongsun96 yuhongsun96 approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@rkuo-danswer @yuhongsun96