effective changes:
------------------

- introduces a multithreaded webcrawler specialized to scrape
  documentation
- adds the following backend endpoints:
  * `GET /docs/sync url==<string>`: index a new doc source
  * `GET|DELETE /docs/:id`: crud ops on indexed doc sources
  * `GET /docs/:id/resync`: update an existing doc source
  * `GET /docs/:id/fetch relative_url==<string>`: fetch one webpage from
    a doc source, the page is broken down into headings & sections
  * `GET /docs/:id/search`: semantic search over a doc source, the
    granularity here is induvidual sections of each page

implementation details:
-----------------------

- article scraper is a port of a python library: newspaper3k
- websites are scraped and broken down into sections based on html
  header tags; using tree-sitter
- a new qdrant collection is initialized on startup: `web`
- document sources are tracked on sqlite, but text content of each page
  is stored solely in qdrant

studio crud ops
---------------

- introduces a new `doc_context` to studio snapshots and studios
- `doc_context` may be populated in a similar fashion to `context`,
  through the `patch` method, sample request:

    http PATCH :7878/api/studio/1
    {
        "doc_context": [
            {
                "doc_id": 3,
                "doc_source": "https://docs.rs/qdrant-client/latest/qdrant_client/",
                "relative_url": "qdrant/struct.PayloadIncludeSelector.html",
                "ranges": [
                    "eaefb40a-13a3-4c2e-a0b3-4ffa3670bfa4",
                    "9b0f63bb-9ecd-46ee-8315-23065df418ce"
                ],
                "hidden": false
            }
        ]
    }
- the uuids in the `ranges` field correspond to the sections in a
  webpage
- to display a webpage with its active an inactive sections, use the
  `fetch` endpoint, which lists every section of the page in-order.
  among these, the active sections are those which are present in the
  studio context

token counting
--------------

- token counts for docs are calculated separately and added under the
  `doc_context` field
- token counts include the headers for each section as well

- also add support for metadata scraping, yet to be integrated into the
  doc index however

also updates db to incorporate metadata changeset

- return stream errors as EventSource messages
- return end of stream only /after/ all tantivy commits have completed
- include link dedup check in `verify` endpoint
- track absolute url in the index instead of constructing on the fly
- attach newlines if not available in code fence blocks
- and several others, idk look at the diff

- different url schemes used in linking pages
- improve article main node detector

- also add Sse keep-alive to all streaming endpoints
- commit just once at the end of the index step

* doc indexing ui

* add doc sections panel

* styling for doc panel

* two-step popup

* use sse for syncing docs

* add filtering to doc providers on search

* search sections

* fix sections selection

* show a list of docs for provider

* add docs to studio context

* add tokens for docs

* ui fixes

* get token count for doc

* some design updates

* rework arrow key navigation in doc context modal

* update designs for doc panel

* add title and icon to docs in code studio

* minor fixes

* rework arrow navigation for repo context modal

* small fix for long doc urls

* show error if doc page in context is unavailable

* set max width for the info message and add three dots to very long urls to make it less jittery

* allow multiselect for context files

* add breadcrumbs for doc sections on search, fix scrolling issues in doc modal

* rework section search UI

* minor UI fixes

* change doc remove hotkey

* make sure buttons are not selectable

* disable html in markdown for parsed docs

* reset indexing state on SSE error

* add tooltip to breadcrumbs, fix clicking on section breadcrumbs, open link in doc panel

* manually handle links in docs panel

* use absolute url

* remove console log

- make doc_context non-null
- use sql transaction to prevent stream errors from landing us in an
  incongruent state
- do not check host str when making relative urls
- no more warnings