Is SQLLite really the best database for long term text storage? I'd think elastisearch...

Also we are getting close to needing a docker compose file.

Also see #122

@ryanpeach elasticsearch is very heavy. they don't have a long term free version on the cloud like pinecone does, so putting the api key of the cloud offering wouldn't be very useful for most people.
Putting it as a docker file I am not sure...
But a big benefit of elasticsearch is that it does good keyword search and it allows people to not have to embed their texts.

But I would put opensearch though, Elasticsearch is under an ambiguous license, so people forking autogpt and using it for comercial use with the elasticsearch integration might run into issues.

Sqlite3 is part of python standard library. That's why I chose it as a sensible default. It sidesteps both license and installation issues. If they have python 3, they have sqlite3.
https://docs.python.org/3/library/persistence.html
It was either that or pickle.

Thank you, big shot

!!

To be clear, I'm pro-vector inputs. However, vectored inputs can be added in addition to this. This is more like a "persistent log".

I also think that vector memory is one thing and long term text retrieval is another, like @EricFedrowisch says. However, I'd also be fine saving long term text to a text file and searching it with an nlp library or regex. I don't think SQL is a good choice for long term text storage because it doesn't add value in querying the text, and it doesn't add value in storing the text over raw text on disk storage.

I'd totally be in favor of opensearch in docker though.

@ryanpeach Wow! Then you should totally write some code, Ryan. Like just do it. No one is stopping you.

@EricFedrowisch Dude i have several PR's. I'm contributing to the discussion on this one. That kind of unprofessional comment is very toxic.

Choosing the appropriate, limited, set of databases for this repo will be important for its future. It should probably be discussed. I'd rather that discussion be had before writing code to implement.

@ryanpeach Weird. I don't see any pull requests here by you. I see a lot of issues about code styling, which feels pointless when half the point of GPT-4 is to have the AI write the code. You can call me toxic if you like, but I don't see how you materially improved the conversation in any way. If you have better ideas, then write them and have people use them.
Several people are already using mine: #125 (comment)
And they seem to like it.

I hadn't seen this prior to implementing something similar. I basically built the ability to save "snapshots", which include the history and memory; which allows you to stop/start in the same state, and potentially course-correct manually. I had to do some refactoring to make it clean enough that we can swap out data stores (you should be able to change the snapshot file to point to a different data store).

Joe0#9

@Joe0 Framing the data as snapshots seems very useful. The persistent memory imp I roughed out uses "sessions", with basically each run of the program creating a new session. There is a command to get_session(id), so you can retrieve whatever session you want in plain text. If you can decouple the snapshot from data stores, that would be huge.

@EricFedrowisch ya, I'm not sure what the best approach is. We can store the state transitions to reconstruct the internal state at some point in time (which is initially how I approached it), but it was getting annoying, so I switched to just taking snapshots of the state after each transition.

Maybe the right idea is to just build some tooling around re-creation snapshots based on the state transitions. Though, it could be useful for people collaborating to be able to resume from snapshots.

@Joe0 I think you are onto something. Still reading and trying to understand your pr.

Out of curiousity, what was your motivation for focusing on state transitions?

@EricFedrowisch if you store the transitions themselves, you get some nice properties like full replays, and can potentially do some level of caching. Snapshots would also give you full replayability, but you can't really do caching with them, unless the full states are equal (which seems much less likely). The benefit of snapshots is that you don't need to replay or reconstruct the current state, and just have it.

I had a few motivations behind the implementation, one of which was the ability to save states and create test cases. Another was the ability to manually course-correct the system by exiting and modifying the snapshot manually. It also creates a nice debugging tool, where you can effectively see what happened, all I need to add is the outputs of the commands themselves.

The core of the PR is really just refactoring the way we manage message history and memory so that we can "hook" all the operations and perform side effects (saving). Then the rest is just saving/loading. I'm working on a cleaner PR that's branched off master of this repo.

@ryanpeach Several people have confirmed that I was being toxic. Please accept my apology. Your desire to fully explore the questions this project raises are reasonable. Again, I'm sorry. I'll try and do better in the future.

I like your class.
I just put together the code mentioned above on #122 for using pinecone.
It might make sense to put these both on the same interface and/or we could consider something like dumping text to sql and just putting the embeddings in pinecone.
Currently, I am sending up the raw text as meta data and it would be better to store that locally so we don't store more crap in pinecone than we need to and so we can query stuff locally.

It might be worth doing something like:

1. Store raw memory data locally in sql mapped to a vector id
2. Query for relevant memories in pinecone which returns vector id so we can query sql from it.
3. Make a start up option to push local sql to pinecone embeddings.

Just some thoughts. Also, wanted to make sure we could make the merge conflict situation less painful and make sure we work together on whatever solution we come up with if we do take both PRs

@dschonholtz Glad to see #122 . I'd see this pr as analogous to your SimpleMemory class and/or a local cache. I'm not familiar with pinecone, currently using llama_index for my experiments. So, gonna leave decisions revolving around that to others with more experience.
This repo is a viral success, pull requests and issues are coming in way faster than one person can likely deal with. So many changes are coming in, so fast that it seems likely that merges are going to be painful for a while. @Torantulino needs some time to have a chance to communicate his vision.

I tested this code and believe there's a bug. The memory addresses are being input to the prompt, rather than the memory itself.

Example:
{'role': 'system', 'content': 'Permanent memory: <memory.MemoryDB object at 0x00000174FEDD3F40>'}

If we do choose to use a SQL database, I'd at least petition that we use an adaptor. I have plans to use autogpt on k8s with multiple instances talking to one another in the future. I believe the database should be Horizontally Autoscalable. In the SQL world, the best database for that is Postgres. If we implemented the option to use sqlalchemy, or something similar, then the database could be customizable by the user.

I think another thing is making the memory modular. We will apparently obviously disagree on database implementations. Maybe the only solution to that is to create a database folder, a class interface, and then just a bunch of implementations, with a CLI flag or config file for the user choice.

A discussion should be had with this PR about such a design as well:
#122

I'll check it out, @jdonkers and do another commit. My guess is I need to do a repr so that it can return a string for the current session text.