Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Response Bot using GPT and Webpage Sources #7518

Merged
merged 41 commits into from
Jul 21, 2023
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
41 commits
Select commit Hold shift + click to select a range
3cbbc3a
poc: Search support articles via OpenAI embeddings
sojan-official May 22, 2023
fe86e78
Merge branch 'develop' into spike/CW-1690-embeddings-for-articles
sojan-official May 29, 2023
3e0a71d
Chore: basic prototype
sojan-official Mar 20, 2023
8fbfe44
chore: GPT bot with article recommendations
sojan-official May 29, 2023
44a24bb
Merge branch 'develop' into spike/CW-1690-embeddings-for-articles
sojan-official May 29, 2023
cdaa40b
chore: minor formatting
sojan-official May 29, 2023
a728bb2
Merge branch 'develop' into spike/CW-1690-embeddings-for-articles
sojan-official Jun 21, 2023
f26649a
Merge branch 'develop' into spike/CW-1690-embeddings-for-articles
sojan-official Jul 5, 2023
c921aec
chore: Response Sources
sojan-official Jul 10, 2023
ce872de
Merge branch 'spike/CW-1690-embeddings-for-articles' into chore/respo…
sojan-official Jul 10, 2023
870d2e6
chore: fi
sojan-official Jul 13, 2023
41a7a25
chore: clean up
sojan-official Jul 13, 2023
a732ec1
chore: clean up
sojan-official Jul 13, 2023
78814eb
Merge branch 'develop' into chore/response-sources-with-embeddings
sojan-official Jul 13, 2023
bf6d69a
chore: fix
sojan-official Jul 13, 2023
dc01127
chore: endpoint to create sources
sojan-official Jul 13, 2023
590cff4
chor: fixes
sojan-official Jul 13, 2023
11a16a8
chore: The bot
sojan-official Jul 13, 2023
6510886
chore: reoder gemfile
sojan-official Jul 13, 2023
cc61674
chore: fix
sojan-official Jul 13, 2023
b7c7b09
chore: APIs for add and remove documents
sojan-official Jul 14, 2023
ada6c9c
Chore: clean up
sojan-official Jul 14, 2023
3729a9f
Merge branch 'develop' into chore/response-sources-with-embeddings
sojan-official Jul 14, 2023
dfa9b27
chore: fixes
sojan-official Jul 17, 2023
fce44b7
Merge branch 'develop' into chore/response-sources-with-embeddings
sojan-official Jul 17, 2023
c459f42
chore: clean up
sojan-official Jul 17, 2023
98e3ae0
chore: clean up
sojan-official Jul 17, 2023
f63feff
chore: refactor
sojan-official Jul 17, 2023
fec1e8a
Merge branch 'develop' into chore/response-sources-with-embeddings
pranavrajs Jul 19, 2023
30a2009
Merge branch 'develop' into chore/response-sources-with-embeddings
pranavrajs Jul 19, 2023
3ca3f9d
chore: add feature lock
sojan-official Jul 20, 2023
4538d35
chore: clean up
sojan-official Jul 20, 2023
1db6b2a
chore: fix specs
sojan-official Jul 20, 2023
e99a8ee
Merge branch 'develop' into chore/response-sources-with-embeddings
sojan-official Jul 21, 2023
4b05257
chore: fix code climate
sojan-official Jul 21, 2023
45a2754
chore: fix specs
sojan-official Jul 21, 2023
455df91
chore: test latest postgres
sojan-official Jul 21, 2023
ede19fe
chore: tests
sojan-official Jul 21, 2023
441da57
chore: update
sojan-official Jul 21, 2023
fe86348
Empty-Commit
sojan-official Jul 21, 2023
acf5b66
chore: clean up
sojan-official Jul 21, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
chore: fixes
  • Loading branch information
sojan-official committed Jul 17, 2023
commit dfa9b27bec1226fd1b1d95b41bd064820df272fd
Original file line number Diff line number Diff line change
@@ -1 +1 @@
json.partial! 'api/v1/models/response_source', formats: [:json], resource: @response_source
json.partial! 'api/v1/models/response_source', formats: [:json], resource: @response_source
35 changes: 35 additions & 0 deletions config/initializers/monkey_patches/schema_dumper.rb
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
# When working with experimental extensions, which doesn't have support on all providers
# This monkey patch will help us to ignore the extensions when dumping the schema
# Additionally we will also ignore the tables associated with those features and exentions

# Once the feature stabilizes, we can remove the tables/extension from the ignore list
# Ensure you write appropriate migrations when you do that.

module ActiveRecord
module ConnectionAdapters
module PostgreSQL
class SchemaDumper < ConnectionAdapters::SchemaDumper
cattr_accessor :ignore_extentions, default: []

private

def extensions(stream)
extensions = @connection.extensions
return unless extensions.any?

stream.puts ' # These are extensions that must be enabled in order to support this database'
extensions.sort.each do |extension|
stream.puts " enable_extension #{extension.inspect}" unless ignore_extentions.include?(extension)
end
stream.puts
end
end
end
end
end

## Extentions / Tables to be ignored
ActiveRecord::ConnectionAdapters::PostgreSQL::SchemaDumper.ignore_extentions << 'vector'
ActiveRecord::SchemaDumper.ignore_tables << 'responses'
ActiveRecord::SchemaDumper.ignore_tables << 'response_sources'
ActiveRecord::SchemaDumper.ignore_tables << 'response_documents'
5 changes: 0 additions & 5 deletions db/migrate/20230509122602_enable_pgvector_extension.rb

This file was deleted.

48 changes: 0 additions & 48 deletions db/migrate/20230709113830_create_responses.rb

This file was deleted.

39 changes: 0 additions & 39 deletions db/schema.rb
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,6 @@
enable_extension "pg_trgm"
enable_extension "pgcrypto"
enable_extension "plpgsql"
enable_extension "vector"

create_table "access_tokens", force: :cascade do |t|
t.string "owner_type"
Expand Down Expand Up @@ -814,44 +813,6 @@
t.index ["user_id"], name: "index_reporting_events_on_user_id"
end

create_table "response_documents", force: :cascade do |t|
t.bigint "response_source_id", null: false
t.string "document_link"
t.string "document_type"
t.bigint "document_id"
t.text "content"
t.bigint "account_id", null: false
t.datetime "created_at", null: false
t.datetime "updated_at", null: false
t.index ["document_type", "document_id"], name: "index_response_documents_on_document"
t.index ["response_source_id"], name: "index_response_documents_on_response_source_id"
end

create_table "response_sources", force: :cascade do |t|
t.integer "source_type", default: 0, null: false
t.string "name", null: false
t.string "source_link"
t.string "source_model_type"
t.bigint "source_model_id"
t.bigint "account_id", null: false
t.bigint "inbox_id", null: false
t.datetime "created_at", null: false
t.datetime "updated_at", null: false
t.index ["source_model_type", "source_model_id"], name: "index_response_sources_on_source_model"
end

create_table "responses", force: :cascade do |t|
t.bigint "response_document_id"
t.string "question", null: false
t.text "answer", null: false
t.bigint "account_id", null: false
t.vector "embedding", limit: 1536
t.datetime "created_at", null: false
t.datetime "updated_at", null: false
t.index ["embedding"], name: "index_responses_on_embedding", using: :ivfflat
t.index ["response_document_id"], name: "index_responses_on_response_document_id"
end

create_table "sla_policies", force: :cascade do |t|
t.string "name", null: false
t.float "frt_threshold"
Expand Down
76 changes: 76 additions & 0 deletions enterprise/app/services/features/response_bot_service.rb
Original file line number Diff line number Diff line change
@@ -0,0 +1,76 @@
class Features::ResponseBotService
MIGRATION_VERSION = ActiveRecord::Migration[7.0]

def enable_in_installation
enable_vector_extension
create_tables
end

def enable_vector_extension
MIGRATION_VERSION.enable_extension 'vector'
end

def disable_vector_extension
MIGRATION_VERSION.disable_extension 'vector'
end

def vector_extension_enabled?
raise 'Vector extension not available' unless ActiveRecord::Base.connection.extension_enabled?('vector')

true
end

def create_tables
vector_extension_enabled?
%i[response_sources response_documents responses].each do |table|
send("create_#{table}_table")
end
end

def drop_tables
%i[responses response_documents response_sources].each do |table|
MIGRATION_VERSION.drop_table table
end
end

private

def create_response_sources_table
MIGRATION_VERSION.create_table :response_sources do |t|
t.integer :source_type, null: false, default: 0
t.string :name, null: false
t.string :source_link
t.references :source_model, polymorphic: true
t.bigint :account_id, null: false
t.bigint :inbox_id, null: false
t.timestamps
end
end

def create_response_documents_table
MIGRATION_VERSION.create_table :response_documents do |t|
t.bigint :response_source_id, null: false
t.string :document_link
t.references :document, polymorphic: true
t.text :content
t.bigint :account_id, null: false
t.timestamps
end

MIGRATION_VERSION.add_index :response_documents, :response_source_id
end

def create_responses_table
MIGRATION_VERSION.create_table :responses do |t|
t.bigint :response_document_id
t.string :question, null: false
t.text :answer, null: false
t.bigint :account_id, null: false
t.vector :embedding, limit: 1536
t.timestamps
end

MIGRATION_VERSION.add_index :responses, :response_document_id
MIGRATION_VERSION.add_index :responses, :embedding, using: :ivfflat, opclass: :vector_l2_ops
end
end