@brutuscat thanks for this, I'll check it out over the weekend!

I am inclined towards only supporting each_key in those cases where the chain of key transformers can be reversed. So I think this means all the key transformers need to be either :encode or :compress types. If not, then the resulting store should behave as if it doesn't support each_key.

This will work for all the Moneta.new cases I can think of off the top of my head, as they mostly just use Marshal.

What do you reckon?

@asppsa something like what I just did? I move the compilation into the built transformer class, so it can add the :each_key feature at "build time" of the concrete transformer class.

Then it is as you said, either supports it or not based on a rule set in the config.rb

@asppsa not sure why mongodb specs are failing...

I reverted my changes to bump the supported ruby version (already some active-support gems, e.g. rails stuff, are using 2.3) given that for some reason kyotocabinet-ruby-reanimated does not compile in ruby 2.7.

See https://travis-ci.org/moneta-rb/moneta/jobs/643129451

kyotocabinet.cc:4038:7: error: ‘ruby_nonempty_memcpy’ is not a member of ‘std’
745       std::memcpy(kbuf_, kbuf, ksiz);
746       ^
747kyotocabinet.cc:4038:7: note: suggested alternative:
748In file included from
749/home/travis/.rvm/rubies/ruby-2.7.0/include/ruby-2.7.0/ruby.h:33:0,
750                 from kyotocabinet.cc:22:
751/home/travis/.rvm/rubies/ruby-2.7.0/include/ruby-2.7.0/ruby/ruby.h:1758:1: note:
752‘ruby_nonempty_memcpy’
753 ruby_nonempty_memcpy(void *dest, const void *src, size_t n)
754 ^

@brutuscat I'll open a separate PR for the Ruby version switch-over. If kyoto cabinet doesn't work any more, we can mark it as unsupported in 2.7+

@asppsa thx. Could you maybe look into what's going on with the MongoDB specs?

Sorry it took me so long to check this. The issue with the MongoDB specs appears to be that they are all using the "standard_mongo" database. When the specs get run in parallel the results will be unpredictable as a result (especially in the case of each_key, evidently). Hopefully this will go away if you make the database name for each file in spec/moneta/adapters/mongo different - best practice is to use a database name that corresponds to the spec's filename - e.g. spec/moneta/adapters/mongo/standard_mongo_moped_spec.rb should use the standard_mongo_moped db.

@asppsa Moped seems to have been forgotten... Since Mongo v3

Starting in version 3.0, MongoDB deprecates direct access to the system.indexes collection, which had previously been used to list all indexes in a database

And in their code you see:

    def create(key, options = {})
      spec = options.merge(ns: namespace, key: key)
      spec[:name] ||= key.to_a.join("_")

      database.session.with(write: { w: 1 }) do |_s|
        _s[:"system.indexes"].insert(spec)
      end
    end

I will add a warning as it was done before for MongoDB version < 2.2.

@asppsa I need more help with the specs :(

Firstly, a huge thank you for all your work here, and a big apology from me that I've been unable to take a look at this sooner.

What I would really love is if we are able to split out some of the things that are going on in this PR into separate PRs. In particular the Mongo and Couch stuff should probably be separated, so we can take a look at it in isolation. There's also a couple of small things like adding rspec-performance that I would want to consider more fully before adding. Let me know if this is too much work though.

The second thing is that I think we need to work out what the expected behaviour of each_key is when someone is using different transformers for different keys in the same database. The obvious example to me is the prefix transformer - one might use the same database with various prefixes in order to separate out the keys. I guess this is probably applicable to other key transforms as well though. Possible behaviours that I think would be acceptable are:

• we say that this isn't supported and throw an error if each_key encounters a key that it can't transform
• we support it, and filter out keys that can't be transformed.

Your thoughts on this are appreciated.

Also, I'll see if I can work out a way to get CouchDB 3.0 working ...

Actually, the mongo issues are directly connected to the each_key stuff, are they? In that case I retract my comments about a separate mongo PR!

@asppsa I reverted most of the changes commented here. The good thing is that now we kinda have fixes for all of them ;)

@asppsa alright I fixed the problem of each_key with a "prefix" transformation. I added a "TEST" code in the transformers config and with that it filters the non prefixed keys. Please review.

BTW I tested the couchdb specs in my local with the latest version and it works fine too.

@asppsa regarding the each_key and the transformers:

we say that this isn't supported and throw an error if each_key encounters a key that it can't transform

I think this only makes sense for prefix. Marshal and other serialisers will fail to "load" and they we will get the exception right there. The same for the encoders I think...

So I compiled a specific code for these where when these fail if won't break the chain. I also added on marshal->prefix->base64 spec that also tests the each_key with the "wrong" prefix

I've fixed CouchDB (and also added Ruby 2.7 testing) in master now. Hopefully if you merge with master, we'll have this PR passing again.

Retrying the build. There was a weird Ruby 2.3 failure in rspec ./spec/moneta/adapters/restclient/standard_restclient_spec.rb[1:1:1], I think that it was just unfortunate because it passed in all the other builds... The rest seems ok.

@brutuscat yes, there is unfortunately some unreliability in the test suite still. If it happens again this time, I can restart just the test(s) that fail instead of rerunning everything.

Concerning the size stuff, I think it's a good addition. If the change is required in two files, so be it.

Ok, I added the changes so to honor the adapter eager size block passed in the enum_for. Please review.

@asppsa could u retry the dying Ruby 2.4 specs please? They seemed have timed out or something weird. I see no rspec error trace.