[wip] adopting huggingface embedding

zylon-ai · zhygallo · Aug 7, 2024 · Aug 10, 2024 · Aug 12, 2024 · Sep 5, 2024
commit 0dc4e2781e71ed5995db1aa27f1809f82ee04c57
diff --git a/.docker/router.yml b/.docker/router.yml
@@ -13,4 +13,4 @@ http:
   routers:
     ollama-router:
       rule: "PathPrefix(`/`)"
-      service: ollama
+      service: ollama
diff --git a/.dockerignore b/.dockerignore
@@ -9,4 +9,4 @@ local_data
 terraform
 tests
 Dockerfile
-Dockerfile.*
+Dockerfile.*
diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md
@@ -34,4 +34,4 @@ Please describe the tests that you ran to verify your changes. Provide instructi
 - [ ] I have added tests that prove my fix is effective or that my feature works
 - [ ] New and existing unit tests pass locally with my changes
 - [ ] Any dependent changes have been merged and published in downstream modules
-- [ ] I ran `make check; make test` to ensure mypy and tests pass
+- [ ] I ran `make check; make test` to ensure mypy and tests pass
diff --git a/.github/workflows/actions/install_dependencies/action.yml b/.github/workflows/actions/install_dependencies/action.yml
@@ -27,4 +27,3 @@ runs:
     - name: Install Dependencies
       run: poetry install --extras "ui vector-stores-qdrant" --no-root
       shell: bash
-
diff --git a/.github/workflows/fern-check.yml b/.github/workflows/fern-check.yml
@@ -6,7 +6,7 @@ on:
       - main
     paths:
       - "fern/**"
-      
+
 jobs:
   fern-check:
     runs-on: ubuntu-latest
@@ -18,4 +18,4 @@ jobs:
         run:  npm install -g fern-api
 
       - name: Check Fern API is valid
-        run: fern check
+        run: fern check
diff --git a/.github/workflows/generate-release.yml b/.github/workflows/generate-release.yml
@@ -80,4 +80,4 @@ jobs:
 
       - name: Version output
         id: version
-        run: echo "version=${{ steps.meta.outputs.version }}" >> "$GITHUB_OUTPUT"
+        run: echo "version=${{ steps.meta.outputs.version }}" >> "$GITHUB_OUTPUT"
diff --git a/.github/workflows/publish-docs.yml b/.github/workflows/publish-docs.yml
@@ -1,8 +1,8 @@
 name: publish docs
 
-on: 
-  push: 
-    branches: 
+on:
+  push:
+    branches:
       - main
     paths:
       - "fern/**"

diff --git a/.github/workflows/release-please.yml b/.github/workflows/release-please.yml
@@ -16,4 +16,4 @@ jobs:
       - uses: google-github-actions/release-please-action@v3
         with:
           release-type: simple
-          version-file: version.txt
+          version-file: version.txt
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
@@ -40,4 +40,4 @@ repos:
     pass_filenames: false
     language: system
     types: [python]
-    stages: [push]
+    stages: [push]
diff --git a/Dockerfile.llamacpp-cpu b/Dockerfile.llamacpp-cpu
@@ -59,4 +59,4 @@ COPY --chown=worker *.yaml ./
 COPY --chown=worker scripts/ scripts
 
 USER worker
-ENTRYPOINT python -m private_gpt
+ENTRYPOINT python -m private_gpt
diff --git a/README.md b/README.md
@@ -110,10 +110,10 @@ typing checks, just run `make check` before committing to make sure your code is
 Remember to test your code! You'll find a tests folder with helpers, and you can run
 tests using `make test` command.
 
-Don't know what to contribute? Here is the public 
-[Project Board](https://github.com/users/imartinez/projects/3) with several ideas. 
+Don't know what to contribute? Here is the public
+[Project Board](https://github.com/users/imartinez/projects/3) with several ideas.
 
-Head over to Discord 
+Head over to Discord
 #contributors channel and ask for write permissions on that GitHub project.
 
 ## 💬 Community
@@ -122,7 +122,7 @@ Join the conversation around PrivateGPT on our:
 - [Discord](https://discord.gg/bK6mRVpErU)
 
 ## 📖 Citation
-If you use PrivateGPT in a paper, check out the [Citation file](CITATION.cff) for the correct citation.  
+If you use PrivateGPT in a paper, check out the [Citation file](CITATION.cff) for the correct citation.
 You can also use the "Cite this repository" button in this repo to get the citation in different formats.
 
 Here are a couple of examples:
@@ -150,7 +150,7 @@ PrivateGPT is actively supported by the teams behind:
 * [Fern](https://buildwithfern.com/), providing Documentation and SDKs
 * [LlamaIndex](https://www.llamaindex.ai/), providing the base RAG framework and abstractions
 
-This project has been strongly influenced and supported by other amazing projects like 
+This project has been strongly influenced and supported by other amazing projects like
 [LangChain](https://github.com/hwchase17/langchain),
 [GPT4All](https://github.com/nomic-ai/gpt4all),
 [LlamaCpp](https://github.com/ggerganov/llama.cpp),

diff --git a/docker-compose.yaml b/docker-compose.yaml
@@ -99,4 +99,4 @@ services:
               count: 1
               capabilities: [gpu]
     profiles:
-      - ollama-cuda
+      - ollama-cuda
diff --git a/fern/README.md b/fern/README.md
@@ -7,15 +7,15 @@ Fern is basically transforming your `.md` and `.mdx` files into a static website
 The configuration of your documentation is done in the `./docs.yml` file.
 There, you can configure the navbar, tabs, sections and pages being rendered.
 
-The documentation of fern (and the syntax of its configuration `docs.yml`) is 
+The documentation of fern (and the syntax of its configuration `docs.yml`) is
 available there [docs.buildwithfern.com](https://docs.buildwithfern.com/).
 
 ## How to run fern
 
 **You cannot render your documentation locally without fern credentials.**
 
 To see how your documentation looks like, you **have to** use the CICD of this
-repository (by opening a PR, CICD job will be executed, and a preview of 
+repository (by opening a PR, CICD job will be executed, and a preview of
 your PR's documentation will be deployed in vercel automatically, through fern).
 
 The only thing you can do locally, is to run `fern check`, which check the syntax of
@@ -36,4 +36,4 @@ navigation:
           - page: My new page display name
             # The path of the page, relative to `fern/`
             path: ./docs/pages/my-existing-tab/new-page-content.mdx
-```
+```
diff --git a/fern/docs/pages/api-reference/api-reference.mdx b/fern/docs/pages/api-reference/api-reference.mdx
@@ -11,4 +11,4 @@ The API is divided in two logical blocks:
 2. Low-level API, allowing advanced users to implement their own complex pipelines:
     - Embeddings generation: based on a piece of text.
     - Contextual chunks retrieval: given a query, returns the most relevant chunks of text from the ingested
-      documents.
+      documents.
diff --git a/fern/docs/pages/installation/concepts.mdx b/fern/docs/pages/installation/concepts.mdx
@@ -64,4 +64,4 @@ In order for HuggingFace LLM to work (the second option), you need to download t
 poetry run python scripts/setup
 ```
 ### Vector stores
-The vector stores supported (Qdrant, Milvus, ChromaDB and Postgres) run locally by default.
+The vector stores supported (Qdrant, Milvus, ChromaDB and Postgres) run locally by default.
diff --git a/fern/docs/pages/installation/troubleshooting.mdx b/fern/docs/pages/installation/troubleshooting.mdx
@@ -46,4 +46,4 @@ huggingface:
 embedding:
   embed_dim: 384
 ```
-</Callout>
+</Callout>
diff --git a/fern/docs/pages/manual/ingestion-reset.mdx b/fern/docs/pages/manual/ingestion-reset.mdx
@@ -11,4 +11,4 @@ make wipe
 # Advanced usage
 
 You can actually delete your documents from your storage by using the
-API endpoint `DELETE` in the Ingestion API.
+API endpoint `DELETE` in the Ingestion API.
diff --git a/fern/docs/pages/manual/ingestion.mdx b/fern/docs/pages/manual/ingestion.mdx
@@ -134,4 +134,3 @@ explanatory error asking you to download the necessary dependencies to install t
 **Other file formats might work**, but they will be considered as plain text
 files (in other words, they will be ingested as `.txt` files).
 </Callout>
-
diff --git a/fern/docs/pages/manual/llms.mdx b/fern/docs/pages/manual/llms.mdx
@@ -117,7 +117,7 @@ azopenai:
   azure_endpoint: <your_azopenai_endpoint> # You could skip this configuration and use the AZ_OPENAI_ENDPOINT env var instead
   api_version: <api_version> # The API version to use. Default is "2023_05_15"
   embedding_deployment_name: <your_embedding_deployment_name> # You could skip this configuration and use the AZ_OPENAI_EMBEDDING_DEPLOYMENT_NAME env var instead
-  embedding_model: <openai_embeddings_to_use> # Optional model to use. Default is "text-embedding-ada-002" 
+  embedding_model: <openai_embeddings_to_use> # Optional model to use. Default is "text-embedding-ada-002"
   llm_deployment_name: <your_model_deployment_name> # You could skip this configuration and use the AZ_OPENAI_LLM_DEPLOYMENT_NAME env var instead
   llm_model: <openai_model_to_use> # Optional model to use. Default is "gpt-35-turbo"
 ```
@@ -231,4 +231,3 @@ or
 
 When the server is started it will print a log *Application startup complete*.
 Navigate to http://localhost:8001 to use the Gradio UI or to http://localhost:8001/docs (API section) to try the API.
-
diff --git a/fern/docs/pages/manual/nodestore.mdx b/fern/docs/pages/manual/nodestore.mdx
@@ -57,10 +57,10 @@ Given the above configuration, Two PostgreSQL tables will be created upon succes
 ```
 postgres=# \dt private_gpt.*
                   List of relations
-   Schema    |      Name       | Type  |    Owner     
+   Schema    |      Name       | Type  |    Owner
 -------------+-----------------+-------+--------------
  private_gpt | data_docstore   | table | postgres
  private_gpt | data_indexstore | table | postgres
 
-postgres=# 
+postgres=#
 ```
diff --git a/fern/docs/pages/manual/reranker.mdx b/fern/docs/pages/manual/reranker.mdx
@@ -33,4 +33,4 @@ rag:
   rerank:
     enabled: true
     top_n: 3  # Number of top-ranked documents to use for generating the answer
-```
+```
diff --git a/fern/docs/pages/manual/settings.mdx b/fern/docs/pages/manual/settings.mdx
@@ -82,4 +82,4 @@ Missing variables with no default will produce an error.
 ```yaml
 server:
   port: ${PORT:8001}
-```
+```
diff --git a/fern/docs/pages/manual/vectordb.mdx b/fern/docs/pages/manual/vectordb.mdx
@@ -116,7 +116,7 @@ postgres=# \d private_gpt.data_embeddings
 Indexes:
     "data_embeddings_pkey" PRIMARY KEY, btree (id)
 
-postgres=# 
+postgres=#
 ```
 The dimensions of the embeddings columns will be set based on the `embedding.embed_dim` value.  If the embedding model changes this table may need to be dropped and recreated to avoid a dimension mismatch.
 

diff --git a/fern/docs/pages/overview/welcome.mdx b/fern/docs/pages/overview/welcome.mdx
@@ -39,4 +39,4 @@ Get started by understanding the [Main Concepts and Installation](/installation)
   />
 </Cards>
 
-<br />
+<br />
diff --git a/fern/docs/pages/quickstart/quickstart.mdx b/fern/docs/pages/quickstart/quickstart.mdx
@@ -102,4 +102,4 @@ or with a specific profile:
 ```sh
 docker-compose --profile <profile_name> up --build
 ```
-Replace `<profile_name>` with the desired profile.
+Replace `<profile_name>` with the desired profile.
diff --git a/fern/docs/pages/recipes/summarize.mdx b/fern/docs/pages/recipes/summarize.mdx
@@ -17,4 +17,3 @@ This can be applied in various scenarios, such as summarizing research papers, n
 
 If you have ideas for improving the Summarize or want to add new features, feel free to contribute!
 You can submit your enhancements via a pull request on our [GitHub repository](https://github.com/zylon-ai/private-gpt).
-
diff --git a/fern/docs/pages/ui/gradio.mdx b/fern/docs/pages/ui/gradio.mdx
@@ -68,4 +68,4 @@ their manner of speaking and vocabulary.
 * You are a -role- bot, respond with -response criteria needed-. If no -response criteria- is needed,
 respond with -alternate response-.
     * Example: You are a grammar checking bot, respond with any grammatical corrections needed. If no corrections
-    are needed, respond with "verified".
+    are needed, respond with "verified".
diff --git a/fern/fern.config.json b/fern/fern.config.json
@@ -1,4 +1,4 @@
 {
   "organization": "privategpt",
   "version": "0.31.17"
-}
+}
diff --git a/fern/openapi/openapi.json b/fern/openapi/openapi.json
@@ -1267,4 +1267,4 @@
       }
     }
   }
-}
+}
diff --git a/local_data/.gitignore b/local_data/.gitignore
@@ -1,2 +1,2 @@
 *
-!.gitignore
+!.gitignore
diff --git a/models/.gitignore b/models/.gitignore
@@ -1,2 +1,2 @@
 *
-!.gitignore
+!.gitignore
Original file line number	Diff line number	Diff line change
Expand Up		@@ -27,4 +27,3 @@ runs:
		- name: Install Dependencies
		run: poetry install --extras "ui vector-stores-qdrant" --no-root
		shell: bash
Original file line number	Diff line number	Diff line change
Expand Up		@@ -134,4 +134,3 @@ explanatory error asking you to download the necessary dependencies to install t
		Other file formats might work, but they will be considered as plain text
		files (in other words, they will be ingested as `.txt` files).
		</Callout>
-Original file line number
+Diff line change
@@ Expand Up / @@ -33,4 +33,4 @@ rag: @@
       rerank:
         enabled: true
         top_n: 3  # Number of top-ranked documents to use for generating the answer
-    ```
+    ```
Original file line number	Diff line number	Diff line change
Expand Up		@@ -17,4 +17,3 @@ This can be applied in various scenarios, such as summarizing research papers, n

		If you have ideas for improving the Summarize or want to add new features, feel free to contribute!
		You can submit your enhancements via a pull request on our [GitHub repository](https://github.com/zylon-ai/private-gpt).