Skip to content

Commit

Permalink
Jinja templates for Instruct and Chat (oobabooga#4874)
Browse files Browse the repository at this point in the history
  • Loading branch information
oobabooga authored Dec 12, 2023
1 parent aab0dd9 commit 39d2fe1
Show file tree
Hide file tree
Showing 71 changed files with 1,771 additions and 515 deletions.
5 changes: 1 addition & 4 deletions docs/01 - Chat Tab.md
Original file line number Diff line number Diff line change
Expand Up @@ -97,10 +97,7 @@ The "Chat" option should typically be used only for base models or non-instruct

Used for talking to an instruction-following model using the prompt format defined under "Parameters" > "Instruction template". Think of this option as an offline ChatGPT.

The prompt format is defined by the following adjustable parameters in "Parameters" > "Instruction template":

* **Context**: appears at the top of the prompt exactly as it is written, including the newline characters at the end (if any). Often the context includes a customizable system message. For instance, instead of "Answer the questions." for Llama-2-chat, you can write "Answer the questions as if you were a pirate.", and the model will comply.
* **Turn template**: defines a single input/reply turn. In this string, `<|user|>` and `<|bot|>` are placeholders that get replaced with whatever you type in the **User string** and **Bot string** fields respectively; they are mandatory and should be present even if those fields are empty. `<|user-message|>` and `<|bot-message|>` get replaced with the user and bot messages at that turn. If the prompt format uses newline characters, they should be written inline as `\n` in the turn template.
The prompt format is defined by the **Instruction template string** parameter in "Parameters" > "Instruction template", which represents a A Jinja2 template.

Note that when you load a model in the "Model" tab, the web UI will try to automatically detect its instruction template (if any), and will update the values under "Parameters" > "Instruction template" accordingly. This is done using a set of regular expressions defined in `models/config.yaml`. This detection is not guaranteed to be accurate. You should check the model card on Hugging Face to see if you are using the correct prompt format.

Expand Down
8 changes: 2 additions & 6 deletions docs/03 - Parameters Tab.md
Original file line number Diff line number Diff line change
Expand Up @@ -101,13 +101,9 @@ So you can use those special placeholders in your character definitions. They ar

Defines the instruction template that is used in the Chat tab when "instruct" or "chat-instruct" are selected under "Mode".

* **Instruction template**: A dropdown menu where you can select from saved templates, save a new template (💾 button), and delete the currently selected template (🗑️).
* **Instruction template**: A dropdown menu where you can load a saved template, save a new template (💾 button), and delete the currently selected template (🗑️).
* **Custom system message**: A message that defines the personality of the chatbot, replacing its default "System message" string. Example: "You are a duck."
* **Turn template**: Defines the positioning of spaces and new line characters in a single turn of the dialogue. `<|user-message|>` gets replaced with the user input, `<|bot-message|>` gets replaced with the bot reply, `<|user|>` gets replaced with the "User string" below, and `<|bot|>` gets replaced with "Bot string" below. The `<|user|>` and `<|bot|>` placeholders must be included even if "User string" and "Bot string" are empty, as they are used to split the template in parts in the backend.
* **User string**: Replaces `<|user|>` in the turn template.
* **Bot string**: Replaces `<|bot|>` in the turn template.
* **Context**: A string that appears as-is at the top of the prompt, including the new line characters at the end (if any). The `<|system-message|>` placeholder gets replaced with the "System message" string below, unless "Custom system message" is not empty, in which case it is used instead.
* **System message**: A default message recommended by the model creator(s) to define the personality of the chatbot.
* **Instruction template string**: A Jinja2 template that defines the prompt format for the instruction-following conversation.
* **Send to default**: Send the full instruction template in string format to the Default tab.
* **Send to notebook**: Send the full instruction template in string format to the Notebook tab.
* **Send to negative prompt**: Send the full instruction template in string format to the "Negative prompt" field under "Parameters" > "Generation".
Expand Down
30 changes: 15 additions & 15 deletions extensions/openai/completions.py
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,8 @@
from modules.chat import (
generate_chat_prompt,
generate_chat_reply,
load_character_memoized
load_character_memoized,
load_instruction_template_memoized
)
from modules.presets import load_preset_memoized
from modules.text_generation import decode, encode, generate_reply
Expand Down Expand Up @@ -195,21 +196,23 @@ def chat_completions_common(body: dict, is_legacy: bool = False, stream=False) -
continue_ = body['continue_']

# Instruction template
instruction_template = body['instruction_template'] or shared.settings['instruction_template']
instruction_template = "Alpaca" if instruction_template == "None" else instruction_template
name1_instruct, name2_instruct, _, _, context_instruct, turn_template, system_message = load_character_memoized(instruction_template, '', '', instruct=True)
name1_instruct = body['name1_instruct'] or name1_instruct
name2_instruct = body['name2_instruct'] or name2_instruct
turn_template = body['turn_template'] or turn_template
context_instruct = body['context_instruct'] or context_instruct
system_message = body['system_message'] or system_message
if body['instruction_template_str']:
instruction_template_str = body['instruction_template_str']
elif body['instruction_template']:
instruction_template = body['instruction_template']
instruction_template = "Alpaca" if instruction_template == "None" else instruction_template
instruction_template_str = load_instruction_template_memoized(instruction_template)
else:
instruction_template_str = shared.settings['instruction_template_str']

chat_template_str = body['chat_template_str'] or shared.settings['chat_template_str']
chat_instruct_command = body['chat_instruct_command'] or shared.settings['chat-instruct_command']

# Chat character
character = body['character'] or shared.settings['character']
character = "Assistant" if character == "None" else character
name1 = body['name1'] or shared.settings['name1']
name1, name2, _, greeting, context, _, _ = load_character_memoized(character, name1, '', instruct=False)
name1, name2, _, greeting, context = load_character_memoized(character, name1, '')
name2 = body['name2'] or name2
context = body['context'] or context
greeting = body['greeting'] or greeting
Expand All @@ -223,12 +226,9 @@ def chat_completions_common(body: dict, is_legacy: bool = False, stream=False) -
'name2': name2,
'context': context,
'greeting': greeting,
'name1_instruct': name1_instruct,
'name2_instruct': name2_instruct,
'context_instruct': context_instruct,
'system_message': system_message,
'instruction_template_str': instruction_template_str,
'custom_system_message': custom_system_message,
'turn_template': turn_template,
'chat_template_str': chat_template_str,
'chat-instruct_command': chat_instruct_command,
'history': history,
'stream': stream
Expand Down
9 changes: 3 additions & 6 deletions extensions/openai/typing.py
Original file line number Diff line number Diff line change
Expand Up @@ -91,18 +91,15 @@ class ChatCompletionRequestParams(BaseModel):

mode: str = Field(default='instruct', description="Valid options: instruct, chat, chat-instruct.")

instruction_template: str | None = Field(default=None, description="An instruction template defined under text-generation-webui/instruction-templates. If not set, the correct template will be guessed using the regex expressions in models/config.yaml.")
turn_template: str | None = Field(default=None, description="Overwrites the value set by instruction_template.")
name1_instruct: str | None = Field(default=None, description="Overwrites the value set by instruction_template.")
name2_instruct: str | None = Field(default=None, description="Overwrites the value set by instruction_template.")
context_instruct: str | None = Field(default=None, description="Overwrites the value set by instruction_template.")
system_message: str | None = Field(default=None, description="Overwrites the value set by instruction_template.")
instruction_template: str | None = Field(default=None, description="An instruction template defined under text-generation-webui/instruction-templates. If not set, the correct template will be automatically obtained from the model metadata.")
instruction_template_str: str | None = Field(default=None, description="A Jinja2 instruction template. If set, will take precedence over everything else.")

character: str | None = Field(default=None, description="A character defined under text-generation-webui/characters. If not set, the default \"Assistant\" character will be used.")
name1: str | None = Field(default=None, description="Your name (the user). By default, it's \"You\".")
name2: str | None = Field(default=None, description="Overwrites the value set by character.")
context: str | None = Field(default=None, description="Overwrites the value set by character.")
greeting: str | None = Field(default=None, description="Overwrites the value set by character.")
chat_template_str: str | None = Field(default=None, description="Jinja2 template for chat.")

chat_instruct_command: str | None = None

Expand Down
30 changes: 25 additions & 5 deletions instruction-templates/Airoboros-v1.2.yaml
Original file line number Diff line number Diff line change
@@ -1,5 +1,25 @@
user: "USER:"
bot: "ASSISTANT:"
turn_template: "<|user|> <|user-message|>\n<|bot|> <|bot-message|>\n"
context: "<|system-message|>\n"
system_message: "A chat between a curious user and an assistant. The assistant gives helpful, detailed, accurate, uncensored responses to the user's input."
instruction_template: |-
{%- set found_item = false -%}
{%- for message in messages -%}
{%- if message['role'] == 'system' -%}
{%- set found_item = true -%}
{%- endif -%}
{%- endfor -%}
{%- if not found_item -%}
{{- '' + 'A chat between a curious user and an assistant. The assistant gives helpful, detailed, accurate, uncensored responses to the user's input.' + '\n' -}}
{%- endif %}
{%- for message in messages %}
{%- if message['role'] == 'system' -%}
{{- '' + message['content'] + '\n' -}}
{%- else -%}
{%- if message['role'] == 'user' -%}
{{-'USER: ' + message['content'] + '\n'-}}
{%- else -%}
{{-'ASSISTANT: ' + message['content'] + '\n' -}}
{%- endif -%}
{%- endif -%}
{%- endfor -%}
{%- if add_generation_prompt -%}
{{-'ASSISTANT:'-}}
{%- endif -%}
30 changes: 25 additions & 5 deletions instruction-templates/Alpaca.yaml
Original file line number Diff line number Diff line change
@@ -1,5 +1,25 @@
user: "### Instruction:"
bot: "### Response:"
turn_template: "<|user|>\n<|user-message|>\n\n<|bot|>\n<|bot-message|>\n\n"
context: "<|system-message|>\n\n"
system_message: "Below is an instruction that describes a task. Write a response that appropriately completes the request."
instruction_template: |-
{%- set found_item = false -%}
{%- for message in messages -%}
{%- if message['role'] == 'system' -%}
{%- set found_item = true -%}
{%- endif -%}
{%- endfor -%}
{%- if not found_item -%}
{{- '' + 'Below is an instruction that describes a task. Write a response that appropriately completes the request.' + '\n\n' -}}
{%- endif %}
{%- for message in messages %}
{%- if message['role'] == 'system' -%}
{{- '' + message['content'] + '\n\n' -}}
{%- else -%}
{%- if message['role'] == 'user' -%}
{{-'### Instruction:\n' + message['content'] + '\n\n'-}}
{%- else -%}
{{-'### Response:\n' + message['content'] + '\n\n' -}}
{%- endif -%}
{%- endif -%}
{%- endfor -%}
{%- if add_generation_prompt -%}
{{-'### Response:\n'-}}
{%- endif -%}
30 changes: 25 additions & 5 deletions instruction-templates/Bactrian.yaml
Original file line number Diff line number Diff line change
@@ -1,5 +1,25 @@
user: "### Input:"
bot: "### Output:"
turn_template: "<|user|>\n<|user-message|>\n\n<|bot|>\n<|bot-message|>\n\n"
context: ""
system_message: ""
instruction_template: |-
{%- set found_item = false -%}
{%- for message in messages -%}
{%- if message['role'] == 'system' -%}
{%- set found_item = true -%}
{%- endif -%}
{%- endfor -%}
{%- if not found_item -%}
{{- '' + '' + '' -}}
{%- endif %}
{%- for message in messages %}
{%- if message['role'] == 'system' -%}
{{- '' + message['content'] + '' -}}
{%- else -%}
{%- if message['role'] == 'user' -%}
{{-'### Input:\n' + message['content'] + '\n\n'-}}
{%- else -%}
{{-'### Output:\n' + message['content'] + '\n\n' -}}
{%- endif -%}
{%- endif -%}
{%- endfor -%}
{%- if add_generation_prompt -%}
{{-'### Output:\n'-}}
{%- endif -%}
30 changes: 25 additions & 5 deletions instruction-templates/Baichuan Chat.yaml
Original file line number Diff line number Diff line change
@@ -1,5 +1,25 @@
user: "<reserved_102>"
bot: "<reserved_103>"
turn_template: "<|user|><|user-message|><|bot|><|bot-message|></s>"
context: ""
system_message: ""
instruction_template: |-
{%- set found_item = false -%}
{%- for message in messages -%}
{%- if message['role'] == 'system' -%}
{%- set found_item = true -%}
{%- endif -%}
{%- endfor -%}
{%- if not found_item -%}
{{- '' + '' + '' -}}
{%- endif %}
{%- for message in messages %}
{%- if message['role'] == 'system' -%}
{{- '' + message['content'] + '' -}}
{%- else -%}
{%- if message['role'] == 'user' -%}
{{-'<reserved_102>' + message['content'] + ''-}}
{%- else -%}
{{-'<reserved_103>' + message['content'] + '</s>' -}}
{%- endif -%}
{%- endif -%}
{%- endfor -%}
{%- if add_generation_prompt -%}
{{-'<reserved_103>'-}}
{%- endif -%}
30 changes: 25 additions & 5 deletions instruction-templates/Baize.yaml
Original file line number Diff line number Diff line change
@@ -1,5 +1,25 @@
user: "[|Human|]"
bot: "[|AI|]"
turn_template: "<|user|><|user-message|>\n<|bot|><|bot-message|>\n"
context: "<|system-message|>\n"
system_message: "The following is a conversation between a human and an AI assistant named Baize (named after a mythical creature in Chinese folklore). Baize is an open-source AI assistant developed by UCSD and Sun Yat-Sen University. The human and the AI assistant take turns chatting. Human statements start with [|Human|] and AI assistant statements start with [|AI|]. The AI assistant always provides responses in as much detail as possible, and in Markdown format. The AI assistant always declines to engage with topics, questions and instructions related to unethical, controversial, or sensitive issues. Complete the transcript in exactly that format.\n[|Human|]Hello!\n[|AI|]Hi!"
instruction_template: |-
{%- set found_item = false -%}
{%- for message in messages -%}
{%- if message['role'] == 'system' -%}
{%- set found_item = true -%}
{%- endif -%}
{%- endfor -%}
{%- if not found_item -%}
{{- '' + 'The following is a conversation between a human and an AI assistant named Baize (named after a mythical creature in Chinese folklore). Baize is an open-source AI assistant developed by UCSD and Sun Yat-Sen University. The human and the AI assistant take turns chatting. Human statements start with [|Human|] and AI assistant statements start with [|AI|]. The AI assistant always provides responses in as much detail as possible, and in Markdown format. The AI assistant always declines to engage with topics, questions and instructions related to unethical, controversial, or sensitive issues. Complete the transcript in exactly that format.\n[|Human|]Hello!\n[|AI|]Hi!' + '\n' -}}
{%- endif %}
{%- for message in messages %}
{%- if message['role'] == 'system' -%}
{{- '' + message['content'] + '\n' -}}
{%- else -%}
{%- if message['role'] == 'user' -%}
{{-'[|Human|]' + message['content'] + '\n'-}}
{%- else -%}
{{-'[|AI|]' + message['content'] + '\n' -}}
{%- endif -%}
{%- endif -%}
{%- endfor -%}
{%- if add_generation_prompt -%}
{{-'[|AI|]'-}}
{%- endif -%}
30 changes: 25 additions & 5 deletions instruction-templates/Bluemoon.yaml
Original file line number Diff line number Diff line change
@@ -1,5 +1,25 @@
user: "LEAD:"
bot: "ASSOCIATE:"
turn_template: "<|user|> <|user-message|>\n<|bot|> <|bot-message|></s>\n"
context: "<|system-message|>\n"
system_message: "A transcript of a roleplay between two players, LEAD and ASSOCIATE. LEAD sets up a scenario and the characters, from which ASSOCIATE then assumes a character role and continues the story for that role in response to description given by LEAD. The story and characters are developed by exchange of detailed event descriptions and character dialogs, successively given by both LEAD and ASSOCIATE."
instruction_template: |-
{%- set found_item = false -%}
{%- for message in messages -%}
{%- if message['role'] == 'system' -%}
{%- set found_item = true -%}
{%- endif -%}
{%- endfor -%}
{%- if not found_item -%}
{{- '' + 'A transcript of a roleplay between two players, LEAD and ASSOCIATE. LEAD sets up a scenario and the characters, from which ASSOCIATE then assumes a character role and continues the story for that role in response to description given by LEAD. The story and characters are developed by exchange of detailed event descriptions and character dialogs, successively given by both LEAD and ASSOCIATE.' + '\n' -}}
{%- endif %}
{%- for message in messages %}
{%- if message['role'] == 'system' -%}
{{- '' + message['content'] + '\n' -}}
{%- else -%}
{%- if message['role'] == 'user' -%}
{{-'LEAD: ' + message['content'] + '\n'-}}
{%- else -%}
{{-'ASSOCIATE: ' + message['content'] + '</s>\n' -}}
{%- endif -%}
{%- endif -%}
{%- endfor -%}
{%- if add_generation_prompt -%}
{{-'ASSOCIATE:'-}}
{%- endif -%}
30 changes: 25 additions & 5 deletions instruction-templates/ChatGLM.yaml
Original file line number Diff line number Diff line change
@@ -1,5 +1,25 @@
user: "[Round <|round|>]\n问:"
bot: "答:"
turn_template: "<|user|><|user-message|>\n<|bot|><|bot-message|>\n"
context: ""
system_message: ""
instruction_template: |-
{%- set found_item = false -%}
{%- for message in messages -%}
{%- if message['role'] == 'system' -%}
{%- set found_item = true -%}
{%- endif -%}
{%- endfor -%}
{%- if not found_item -%}
{{- '' + '' + '' -}}
{%- endif %}
{%- for message in messages %}
{%- if message['role'] == 'system' -%}
{{- '' + message['content'] + '' -}}
{%- else -%}
{%- if message['role'] == 'user' -%}
{{-'[Round <|round|>]\n问:' + message['content'] + '\n'-}}
{%- else -%}
{{-'答:' + message['content'] + '\n' -}}
{%- endif -%}
{%- endif -%}
{%- endfor -%}
{%- if add_generation_prompt -%}
{{-'答:'-}}
{%- endif -%}
31 changes: 25 additions & 6 deletions instruction-templates/ChatML.yaml
Original file line number Diff line number Diff line change
@@ -1,6 +1,25 @@
user: user
bot: assistant
turn_template: <|im_start|><|user|>\n<|user-message|><|im_end|>\n<|im_start|><|bot|>\n<|bot-message|><|im_end|>\n
context: |
<|im_start|>system
<|system-message|><|im_end|>
instruction_template: |-
{%- set found_item = false -%}
{%- for message in messages -%}
{%- if message['role'] == 'system' -%}
{%- set found_item = true -%}
{%- endif -%}
{%- endfor -%}
{%- if not found_item -%}
{{- '<|im_start|>system\n' + '' + '<|im_end|>\n' -}}
{%- endif %}
{%- for message in messages %}
{%- if message['role'] == 'system' -%}
{{- '<|im_start|>system\n' + message['content'] + '<|im_end|>\n' -}}
{%- else -%}
{%- if message['role'] == 'user' -%}
{{-'<|im_start|>user\n' + message['content'] + '<|im_end|>\n'-}}
{%- else -%}
{{-'<|im_start|>assistant\n' + message['content'] + '<|im_end|>\n' -}}
{%- endif -%}
{%- endif -%}
{%- endfor -%}
{%- if add_generation_prompt -%}
{{-'<|im_start|>assistant\n'-}}
{%- endif -%}
Loading

0 comments on commit 39d2fe1

Please sign in to comment.