Skip to content

Commit

Permalink
Update documentation on Vertex AI (langchain4j#709)
Browse files Browse the repository at this point in the history 
Provide instructions on creating Google Cloud Platform account and
establishing a new project with use of Vertex AI API. Add basic code
samples for using google's PaLM2, Gemini and Embedding models. List
available model names.

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit


- **Documentation**
- Updated the guide on Google Vertex AI Embedding Models with
dependencies, example code, and available models.
	- Added a link to an article about image generation with Imagen.
- Expanded the documentation for Google Gemini with setup instructions,
authentication strategies, example uses, and early access information.
- Detailed instructions for getting started with Google Vertex AI PaLM
2, including dependencies, example code, and model information.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

Co-authored-by: Lize Raes <49833622+LizeRaes@users.noreply.github.com>
  • Loading branch information
@OTR @LizeRaes
OTR and LizeRaes authored Mar 8, 2024
1 parent 3535903 commit e62ea2f
Show file tree
Hide file tree
Showing 4 changed files with 333 additions and 9 deletions.
86 changes: 82 additions & 4 deletions docs/docs/integrations/embedding-models/google-vertex-ai.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,10 +2,88 @@
sidebar_position: 9
---

# Google Vertex AI
# Google Vertex AI Embedding Models

[Google Vertex AI Documentation](https://cloud.google.com/vertex-ai)
## Get started

- [Embeddings](https://cloud.google.com/vertex-ai/docs/generative-ai/embeddings/get-text-embeddings)
To get started follow the steps outlined in the `Get started` section of [Vertex AI Gemini integration tutorial](../language-models/google-gemini) to create a
Google Cloud Platform account and establish a new project with access to Vertex AI API.

Coming soon
## Add dependencies

Add the following dependencies to your project's `pom.xml`:

```xml
<dependency>
<groupId>dev.langchain4j</groupId>
<artifactId>langchain4j-vertex-ai</artifactId>
<version>{your-version}</version> <!-- Specify langchain4j version here -->
</dependency>
```

or project's `build.gradle`:

```groovy
implementation 'dev.langchain4j:langchain4j-vertex-ai:{your-version}'
```

### Try out an example code:

[An Example of using Vertex AI Embedding Model](https://github.com/langchain4j/langchain4j-examples/blob/main/other-examples/src/main/java/embedding/model/VertexAiEmbeddingModelExample.java)

The `PROJECT_ID` field represents the variable you set when creating a new Google Cloud project.

```java
import dev.langchain4j.data.embedding.Embedding;
import dev.langchain4j.model.embedding.EmbeddingModel;
import dev.langchain4j.model.output.Response;
import dev.langchain4j.model.vertexai.VertexAiEmbeddingModel;

public class VertexAiEmbeddingModelExample {

private static final String PROJECT_ID = "YOUR-PROJECT-ID";
private static final String MODEL_NAME = "textembedding-gecko@latest";

public static void main(String[] args) {

EmbeddingModel embeddingModel = VertexAiEmbeddingModel.builder()
.endpoint("us-central1-aiplatform.googleapis.com:443")
.project(PROJECT_ID)
.location("us-central1")
.publisher("google")
.modelName(MODEL_NAME)
.build();

Response<Embedding> response = embeddingModel.embed("Hello, how are you?");

Embedding embedding = response.content();

int dimension = embedding.dimension(); // 768
float[] vector = embedding.vector(); // [-0.06050122, -0.046411075, ...

System.out.println(dimension);
System.out.println(embedding.vectorAsList());
}
}
```

### Available Embedding models

| Model name | Description |
|-----------------------------------------|------------------------------------------------------------|
| textembedding-gecko@latest | the newest stable embedding model with enhanced AI quality |
| textembedding-gecko-multilingual@latest | optimized for a wide range of non-English languages. |

[List of supported languages for multi lingual model](https://cloud.google.com/vertex-ai/generative-ai/docs/embeddings/get-text-embeddings#language_coverage_for_textembedding-gecko-multilingual_models)

Model names suffixed with `@latest` reference the most recent version of the model.

The API accepts a maximum of 3,072 input tokens and outputs 768-dimensional vector embeddings.

### References

[Google Codelab on Vertex AI Embedding Model](https://codelabs.developers.google.com/codelabs/genai-chat-java-palm-langchain4j)

[Available stable Embedding Models](https://cloud.google.com/vertex-ai/generative-ai/docs/model-reference/text-embeddings#model_versions)

[Latest Embedding Models version](https://cloud.google.com/vertex-ai/generative-ai/docs/learn/model-versioning#palm-latest-models)
2 changes: 2 additions & 0 deletions docs/docs/integrations/image-models/imagen.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,4 +4,6 @@ sidebar_position: 6

# Google Imagen

[An article about image generation with Imagen](https://glaforge.dev/posts/2024/02/01/image-generation-with-imagen-and-langchain4j/)

Coming soon
150 changes: 149 additions & 1 deletion docs/docs/integrations/language-models/google-gemini.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,4 +2,152 @@
sidebar_position: 9
---

# Google Vertex AI Gemini
# Google Vertex AI Gemini

Vertex AI is a product built on top of the Google Cloud Platform that provides access to Google's large generative models, including the older generation (PaLM2) and the newer generation (Gemini).

To utilize Vertex AI, one must first create a Google Cloud Platform account.

## Get started

### Create Google Cloud Account

If you're new to Google Cloud, you can create a new account by clicking on the `[create an account]` button located under `Get set up on Google Cloud` dropdown menu on the following page:

[Create an account](https://cloud.google.com/vertex-ai/generative-ai/docs/start/quickstarts/quickstart-multimodal#new-to-google-cloud)

### Create a project within your Google Cloud Platform account.

Within your Google Cloud Account create a new project and enable the Vertex AI APIs by following the steps outlined below:

[Create a new project](https://cloud.google.com/vertex-ai/docs/start/cloud-environment#set_up_a_project)

Note your `PROJECT_ID` as it will be required for future API calls..

### Select the Google Cloud authentication strategy

There are several ways on how your application authenticates to Google Cloud services and APIs. For example, you can create a [service account](https://cloud.google.com/docs/authentication/provide-credentials-adc#local-key) and set up environment variable `GOOGLE_APPLICATION_CREDENTIALS` to the path of the JSON file that contains your credentials.

You can discover all the authentication strategies [here](https://cloud.google.com/docs/authentication/provide-credentials-adc). But for simplicity of local testing we will be using authentication via `gcloud` utility.

### Install Google Cloud CLI (Optional)

To access your cloud projects locally, you can install `gcloud` tool by following the [installation instructions](https://cloud.google.com/sdk/docs/install). For GNU/Linux operating systems, the installation steps are as follows:

1. Download SDK:

```bash
curl -O https://dl.google.com/dl/cloudsdk/channels/rapid/downloads/google-cloud-cli-467.0.0-linux-x86_64.tar.gz
```

2. Extract an archive:

```bash
tar -xf google-cloud-cli-467.0.0-linux-x86_64.tar.gz
```
3. Run an installation script:

```bash
cd google-cloud-sdk/
./install.sh
```

4. Run the following command to set up a default project and authentication credentials:

```bash
gcloud auth application-default login
```

This authentication method is compatible with both the `vertex-ai` (Embedding models, PaLM2) and `vertex-ai-gemini` (Gemini) packages.

## Add dependencies

To get started, add the following dependencies to your project's `pom.xml`:

```xml
<dependency>
<groupId>dev.langchain4j</groupId>
<artifactId>langchain4j-vertex-ai-gemini</artifactId>
<version>{your-version}</version> <!-- Specify langchain4j version here -->
</dependency>
```

or project's `build.gradle`:

```groovy
implementation 'dev.langchain4j:langchain4j-vertex-ai-gemini:{your-version}'
```

### Try out an example code:

[Example of using chat model for text prediction](https://github.com/langchain4j/langchain4j-examples/blob/main/vertex-ai-gemini-examples/src/main/java/VertexAiGeminiChatModelExamples.java)

[Gemini Pro Vision with Image input](https://github.com/langchain4j/langchain4j/blob/657aac9519b57afc04ea434ddcfa70d701923b91/langchain4j-vertex-ai-gemini/src/test/java/dev/langchain4j/model/vertexai/VertexAiGeminiChatModelIT.java#L123)

The `PROJECT_ID` field represents the variable you set when creating a new Google Cloud project.

```java
import dev.langchain4j.data.message.AiMessage;
import dev.langchain4j.data.message.ImageContent;
import dev.langchain4j.data.message.TextContent;
import dev.langchain4j.data.message.UserMessage;
import dev.langchain4j.model.chat.ChatLanguageModel;
import dev.langchain4j.model.output.Response;
import dev.langchain4j.model.vertexai.VertexAiGeminiChatModel;

public class GeminiProVisionWithImageInput {

private static final String PROJECT_ID = "YOUR-PROJECT-ID";
private static final String LOCATION = "us-central1";
private static final String MODEL_NAME = "gemini-pro-vision";
private static final String CAT_IMAGE_URL = "https://upload.wikimedia.org/" +
"wikipedia/commons/e/e9/" +
"Felis_silvestris_silvestris_small_gradual_decrease_of_quality.png";

public static void main(String[] args) {
ChatLanguageModel visionModel = VertexAiGeminiChatModel.builder()
.project(PROJECT_ID)
.location(LOCATION)
.modelName(MODEL_NAME)
.build();

Response<AiMessage> response = visionModel.generate(
UserMessage.from(
ImageContent.from(CAT_IMAGE_URL),
TextContent.from("What do you see?")
)
);

System.out.println(response.content().text());
}
}
```

### Available models

|Model name| Description |Properties|
|----------|----------------|----------|
|gemini-pro| Designed to handle natural language tasks, multiturn text and code chat, and code generation.|Max total tokens (input and output): 32,760. Max output tokens: 8,192|
|gemini-pro-vision|Supports multimodal prompts. You can include text, images, and video in your prompt requests and get text or code responses.|Max total tokens (input and output): 16,384. Max output tokens: 2,048|
|gemini-ultra|Google's most capable multimodal model, optimized for complex tasks including instruction, code, and reasoning, with support for multiple languages.|Max tokens input: 8,192. Max tokens output: 2,048|
|gemini-ultra-vision|Google's most capable multimodal vision model, optimized to support text, images, videos, and multi-turn chat. |Max tokens input: 8,192. Max tokens output: 2,048|

Note that in March 2024, the Ultra version has private access with an allow list. Therefore, you may receive an exception similar to this:

```text
Caused by: io.grpc.StatusRuntimeException:
FAILED_PRECONDITION: Project `1234567890` is not allowed to use Publisher Model
`projects/{YOUR_PROJECT_ID}/locations/us-central1/publishers/google/models/gemini-ultra`
```

## Apply for early access

[Early access for Gemma](https://docs.google.com/forms/d/e/1FAIpQLSe0grG6mRFW6dNF3Rb1h_YvKqUp2GaXiglZBgA2Os5iTLWlcg/viewform)

[Early access for Gemini 1.5 Pro](https://aistudio.google.com/app/waitlist/97445851)

## References

[Available locations](https://cloud.google.com/vertex-ai/generative-ai/docs/learn/locations#available-regions)

[Multimodal capabilities](https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/overview#multimodal_models)
104 changes: 100 additions & 4 deletions docs/docs/integrations/language-models/google-palm.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,9 +4,105 @@ sidebar_position: 9

# Google Vertex AI PaLM 2

[Google Vertex AI Documentation](https://cloud.google.com/vertex-ai)
## Get started

- [Chats](https://cloud.google.com/vertex-ai/docs/generative-ai/chat/chat-prompts)
- [Completions](https://cloud.google.com/vertex-ai/docs/generative-ai/text/text-overview)
To get started follow the steps outlined in the `Get started` section of [Vertex AI Gemini integration tutorial](/google-gemini) to create a
Google Cloud Platform account and establish a new project with access to Vertex AI API.

Tutorial coming soon
## Add dependencies

Add the following dependencies to your project's `pom.xml`:

```xml
<dependency>
<groupId>dev.langchain4j</groupId>
<artifactId>langchain4j-vertex-ai</artifactId>
<version>{your-version}</version> <!-- Specify langchain4j version here -->
</dependency>
```

or project's `build.gradle`:

```groovy
implementation 'dev.langchain4j:langchain4j-vertex-ai:{your-version}'
```

### Try out an example code:

[An example of using Vertex AI Embedding Model](https://github.com/langchain4j/langchain4j-examples/blob/main/other-examples/src/main/java/embedding/model/VertexAiEmbeddingModelExample.java)

The `PROJECT_ID` field represents the variable you set when creating a new Google Cloud project.

```java
import dev.langchain4j.data.message.AiMessage;
import dev.langchain4j.data.message.UserMessage;
import dev.langchain4j.model.chat.ChatLanguageModel;
import dev.langchain4j.model.output.Response;
import dev.langchain4j.model.vertexai.VertexAiChatModel;

public class ChatLanguageModel {

private static final String PROJECT_ID = "YOUR-PROJECT-ID";
// `chat-bison` means PaLM2 general purpose chat model
private static final String MODEL_NAME = "chat-bison";

public static void main(String[] args) {
ChatLanguageModel model = VertexAiChatModel.builder()
.endpoint("us-central1-aiplatform.googleapis.com:443")
.location("us-central1")
.publisher("google")
.project(PROJECT_ID)
.modelName(MODEL_NAME)
.temperature(0.0)
.build();

Response<AiMessage> response = model.generate(
UserMessage.from(
"Describe in several sentences what language model you are: \n" +
"Describe in several sentences what is your code name: "
)
);
System.out.println(response.content().text());

// I am a large language model, trained by Google.
// I am a transformer-based language model that has been trained
// on a massive dataset of text and code.
// I am able to understand and generate human language,
// and I can also write code in a variety of programming languages.
//
// My code name is PaLM 2, which stands for Pathways Language Model 2.

}

}
```

### Available chat models

Chat models are optimized for multi-turn chat, where the model keeps track of previous messages in the chat and uses it as context for generating new responses.

|Model name|Description| Properties |
|----------|-----------|---------------------------------------------------------------|
|chat-bison|Fine-tuned for multi-turn conversation use cases.| Maximum input tokens: 8192. Maximum output tokens: 2048 |
|chat-bison-32k|Fine-tuned for multi-turn conversation use cases.| Max tokens (input + output): 32,768. Max output tokens: 8,192 |
|codechat-bison|A model fine-tuned for chatbot conversations that help with code-related questions.| Maximum input tokens: 6144. Maximum output tokens: 1024 |
|codechat-bison-32k|A model fine-tuned for chatbot conversations that help with code-related questions.| Max tokens (input + output): 32,768. Max output tokens: 8,192 |

You can use bare model name e.g. `chat-bison` or specify a stable version,
like `chat-bison@002`.

### Available text models

Text models are optimized for performing natural language tasks, such as classification, summarization, extraction, content creation, and ideation.

Use the `VertexAiLanguageModel` [class](https://github.com/langchain4j/langchain4j/blob/main/langchain4j-vertex-ai/src/test/java/dev/langchain4j/model/vertexai/VertexAiLanguageModelIT.java) for text models such as `text-bison`, `text-bison-32k`, and `text-unicorn`.

## Reference

[Google Codelab on Vertex AI PaLM 2 Model](https://codelabs.developers.google.com/codelabs/genai-text-gen-java-palm-langchain4j)

[PaLM2 generation models](https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models#palm-models)

[Model naming explanation](https://cloud.google.com/vertex-ai/generative-ai/docs/language-model-overview#model_naming_scheme)

[Available PalM stable versions](https://cloud.google.com/vertex-ai/generative-ai/docs/learn/model-versioning#palm-stable-versions-available)

0 comments on commit e62ea2f

Please sign in to comment.