Open
Description
This is very similar to #2847.
Information about peers (their addresses, the protocols they have, their public keys) are currently stored in memory and need to be written to disk to avoid consuming excess memory.
I havent started this one yet and would appreciate some help getting this done.
Activity
Kubuxu commentedon Jul 6, 2016
With @whyrusleeping we decided to bump it off the 0.4.3, it uses only 6MiB of RAM on long running nodes.
csasarak commentedon Jul 28, 2016
I know this was removed from the milestones, but I wouldn't mind digging a little deeper into the structures IPFS uses if you still want the help @whyrusleeping
whyrusleeping commentedon Aug 4, 2016
@csasarak that would be very helpful! I'd love to have some help with this :)
csasarak commentedon Aug 4, 2016
Cool, I'm actually traveling starting Saturday through to next Friday so unfortunately I won't be able to help until then. I've been thinking about this though and was thinking there might be some way to make some reified caching strategy object (if one doesn't exist already) or interface. That way things could get a bit smarter than simply writing entries to disk when the finger table gets larger than a certain point.
Also, crazy idea, but would it be possible or desirable to store peerstore data in IPFS itself? I'm wondering if that might be an efficient way to pass around peerstores or parts of peerstores to other nodes that request them?
whyrusleeping commentedon Aug 4, 2016
The thought of storing the information in IPFS itself is very appealing, but it has a few challenges off the bat, the first of which is that the peerstore data is very mutable in nature, addresses change quickly, new peers come online and leave and our knowledge of things changes with nearly every RPC made (at an extreme). IPFS objects are immutable, so every change that gets made will end up creating a new object (unless optimized similarly to how the mfs bubbling up happens).
The second concern there is about privacy, its not necessarily a problem, but something that needs to be thought through. If you put all the peerstore data in ipfs, it becomes accessible to everyone.
Anyways, Safe travels and i'll hopefully hear from you after next friday :)
csasarak commentedon Aug 4, 2016
I figured there'd be something wrong with it, and it's probably a spec question rather than an issue for go-ipfs anyways. Anyways, I will get back to you - have a good week!
jbenet commentedon Aug 4, 2016
not distributed)
consistency critical)
On Thu, Aug 4, 2016 at 11:31 Christopher Sasarak notifications@github.com
wrote:
csasarak commentedon Aug 17, 2016
@whyrusleeping I've returned. @jbenet Are you saying that we should try to implement it this way? I'd assume that storing it in IPFS would achieve the goal of writing it to disk.
Or should we open a discussion elsewhere about storing data in IPFS and go with the more naive implementation for now?
whyrusleeping commentedon Aug 17, 2016
@csasarak for now lets not store the data in ipfs, we need to figure a lot of stuff out first.
Lets just put this data in leveldb for now, But before we get to that, we need to make the go-libp2p-peerstore support writing into a go-datastore. You should go open an issue there to start discussing the design/changes
15 remaining items