Skip to content

Store peerstore data to disk #2848

Open
@whyrusleeping

Description

This is very similar to #2847.

Information about peers (their addresses, the protocols they have, their public keys) are currently stored in memory and need to be written to disk to avoid consuming excess memory.

I havent started this one yet and would appreciate some help getting this done.

Activity

added
help wantedSeeking public contribution on this issue
exp/expertHaving worked on the specific codebase is important
on Jun 13, 2016
added this to the Ipfs 0.4.3 milestone on Jun 21, 2016
Kubuxu

Kubuxu commented on Jul 6, 2016

@Kubuxu
Member

With @whyrusleeping we decided to bump it off the 0.4.3, it uses only 6MiB of RAM on long running nodes.

removed this from the ipfs-0.4.3 milestone on Jul 6, 2016
csasarak

csasarak commented on Jul 28, 2016

@csasarak
Contributor

I know this was removed from the milestones, but I wouldn't mind digging a little deeper into the structures IPFS uses if you still want the help @whyrusleeping

whyrusleeping

whyrusleeping commented on Aug 4, 2016

@whyrusleeping
MemberAuthor

@csasarak that would be very helpful! I'd love to have some help with this :)

csasarak

csasarak commented on Aug 4, 2016

@csasarak
Contributor

Cool, I'm actually traveling starting Saturday through to next Friday so unfortunately I won't be able to help until then. I've been thinking about this though and was thinking there might be some way to make some reified caching strategy object (if one doesn't exist already) or interface. That way things could get a bit smarter than simply writing entries to disk when the finger table gets larger than a certain point.

Also, crazy idea, but would it be possible or desirable to store peerstore data in IPFS itself? I'm wondering if that might be an efficient way to pass around peerstores or parts of peerstores to other nodes that request them?

whyrusleeping

whyrusleeping commented on Aug 4, 2016

@whyrusleeping
MemberAuthor

The thought of storing the information in IPFS itself is very appealing, but it has a few challenges off the bat, the first of which is that the peerstore data is very mutable in nature, addresses change quickly, new peers come online and leave and our knowledge of things changes with nearly every RPC made (at an extreme). IPFS objects are immutable, so every change that gets made will end up creating a new object (unless optimized similarly to how the mfs bubbling up happens).

The second concern there is about privacy, its not necessarily a problem, but something that needs to be thought through. If you put all the peerstore data in ipfs, it becomes accessible to everyone.

Anyways, Safe travels and i'll hopefully hear from you after next friday :)

csasarak

csasarak commented on Aug 4, 2016

@csasarak
Contributor

I figured there'd be something wrong with it, and it's probably a spec question rather than an issue for go-ipfs anyways. Anyways, I will get back to you - have a good week!

jbenet

jbenet commented on Aug 4, 2016

@jbenet
Member
  • the peerstore data in IPFS makes sense, like the pinset records (private,
    not distributed)
  • with a lazy write (i.e. No need to flush on every write because it's not
    consistency critical)
  • would enable in memory peerstore to become an LRU cache
  • step towards peerdiscovery protocol based on previously seen nodes
    On Thu, Aug 4, 2016 at 11:31 Christopher Sasarak notifications@github.com
    wrote:

I figured there'd be something wrong with it, and it's probably a spec
question rather than an issue for go-ipfs anyways. Anyways, I will get back
to you - have a good week!


You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#2848 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAIcoVOeMe67wEqxjnnQEdrESp_UkrtIks5qcgW9gaJpZM4I0sTr
.

csasarak

csasarak commented on Aug 17, 2016

@csasarak
Contributor

@whyrusleeping I've returned. @jbenet Are you saying that we should try to implement it this way? I'd assume that storing it in IPFS would achieve the goal of writing it to disk.
Or should we open a discussion elsewhere about storing data in IPFS and go with the more naive implementation for now?

whyrusleeping

whyrusleeping commented on Aug 17, 2016

@whyrusleeping
MemberAuthor

@csasarak for now lets not store the data in ipfs, we need to figure a lot of stuff out first.

Lets just put this data in leveldb for now, But before we get to that, we need to make the go-libp2p-peerstore support writing into a go-datastore. You should go open an issue there to start discussing the design/changes

15 remaining items

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    exp/expertHaving worked on the specific codebase is importantstatus/blockedUnable to be worked further until needs are met

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions

      Store peerstore data to disk · Issue #2848 · ipfs/kubo