Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a pluggable strategy for storage #129

Merged
merged 1 commit into from
Oct 25, 2017
Merged

Add a pluggable strategy for storage #129

merged 1 commit into from
Oct 25, 2017

Conversation

jonathansp
Copy link
Contributor

@jonathansp jonathansp commented Oct 6, 2017

We'd like to suggest an alternative approach to solve #85. This PR adds the ability to plug an external strategy for configurable-http-proxy.

Currently, we have been working on an extension for redis: https://github.com/globocom/configurable-http-proxy-redis-backend

@yuvipanda
Copy link
Contributor

Is this for use with JupyterHub? Or for other use cases?

@yuvipanda
Copy link
Contributor

(also, yay for your first PR to this repo! welcome :D)

@jonathansp
Copy link
Contributor Author

@yuvipanda we gonna use with jupyter hub, although it can be used in any case.

@yuvipanda
Copy link
Contributor

Personally, for JupyterHub I'd highly recommend implementing a custom proxy backend instead (http://jupyterhub.readthedocs.io/en/latest/reference/proxy.html) and using it with a different proxy (such as traefik) rather than adding more complexity to CHP. There's a half complete one for traefik + etcd in https://github.com/yuvipanda/jupyterhub-traefik - and more ones are easy to make. IMO that's the way forward rather than building and maintaining our own complex proxy implementation.

@andrelu
Copy link

andrelu commented Oct 6, 2017

I think it would be nice to have a pluggable storage options. I think it does not add complexity to this project because the implementation would be outside.

@jonathansp
Copy link
Contributor Author

jonathansp commented Oct 11, 2017

@yuvipanda implementing a custom proxy was our first thought. However, as discussed in #85, and it seems to be a recurrent debate ( @minrk @rgbkrk @willingc @narenst @andrelu @gbedin ), CHP is a great software and would be nice to have more backends to work with. By following simple steps, we might build a redis-backend, a postgres-backend, an NFS backend etc.

lib/store.js Outdated
@@ -78,4 +78,4 @@ class MemoryStore extends BaseStore {
}
}

exports.MemoryStore = MemoryStore;
module.exports = MemoryStore;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this should be:

exports.BaseStore = BaseStore;
exports.MemoryStore = MemoryStore;

so that the BaseStore is exported and can be used.

Copy link
Contributor Author

@jonathansp jonathansp Oct 11, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@minrk, by doing this I also need to move BaseStore to another file. The change I made expects a class to be exported, not a module. I did it because we don't need to know the class name we are dynamically importing (RedisStore, PGStore etc). Take a look:

const loadStorage = (options) => {
  const BackendStorageClass = require(options.storageBackend || "./store.js");
  return new BackendStorageClass(options);
};

Also, I'm not sure it is a good an idea for a plugin to use BaseStore. We might have cyclic references. What do you think? IMHO, It's more like a duck-typing style than inheritance.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Duck typing is fine, but BaseStore exists for the purpose of being used in different implementations, potentially to reduce how much has to be duplicated. In particular, it implements some basic and second-order functionality that reduces what implementations must provide (e.g. providers don't strictly need to implement get, only getAll). It also gives us a place to add new second-order methods that providers would get for free.

If we aren't exporting BaseStore, there's no reason for BaseStore to exist and it can be subsumed into MemoryStore as a single class, rather than the sole subclass of an unusable base.

We can split the files if we really have to, but maybe it would be better to allow it to be an attribute rather than requiring all providers to use a dedicated module that only exports one class? Perhaps something like backend = './store.js:MemoryStore':

[modName, className] = backend.split(':');
backendMod = require(modName);
if (className) {
  backendClass = backendMod[className];
else {
  backendClass = backendMod;
}

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@minrk since it is not a javascript pattern (I suppose), I don't think it is a good idea. Unless you want to extend your storage strategies inside the library, I vote for removing BaseStore. It's going to leave the code simpler.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the main issue here is that this is going to cause a major bump in semver when we could easily make this a minor bump adding the functionality. Keep the original keys on the exported interface and it will be alright.

Copy link
Contributor Author

@jonathansp jonathansp Oct 23, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@minrk @rgbkrk store.js exports only MemoryStore https://github.com/jupyterhub/configurable-http-proxy/blob/master/lib/store.js#L81, not BaseStore.

In order to not make a brake change, what I can do is: keep exports.MemoryStore and handle this import in loadStorage. It will keep the same contract but sounds weird, since plugins must export a class and our own module exports an object.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, my only suggestion then is to keep the original MemoryStore export, we can defer the BaseStore to outside the scope of this PR to at least finish bringing in the great work you've already done.

I agree with you about the potential for cyclic issues and for why the base store would need to end up in a separate file. I also see the ease at which a default export would provide. For now, let's keep it backwards compatible while moving forward.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@rgbkrk @minrk I made the changes. Could you please check again?

@minrk
Copy link
Member

minrk commented Oct 11, 2017

This looks great! I'm very happy that you found a way to load the storage backend as a plugin without requiring it to be added to CHP itself. I left a minor comment inline. I imagine that at least some custom providers would want to extend BaseStore, so making sure that's exported is probably important.

Re: custom proxies, JupyterHub 0.8 introduces a Python API for its proxy requirements, making it more feasible to provide alternate implementations, as the Hub's API requirements are more clear.

@jonathansp
Copy link
Contributor Author

Before merging this PR, let me add one more test case. I'll work on it tomorrow morning.

@jonathansp
Copy link
Contributor Author

@minrk
Copy link
Member

minrk commented Oct 25, 2017

Cool!

@minrk minrk merged commit 5d7f58b into jupyterhub:master Oct 25, 2017
@willingc
Copy link
Contributor

Thanks @jonathansp for the contribution 👍

@jonathansp
Copy link
Contributor Author

hi everybody, we would like to let you know we finished our configurable-proxy-redis-backend. Please, feel free to make any suggestion!

@jonathansp
Copy link
Contributor Author

@willingc @minrk have you an idea when it's going to be released?

@willingc
Copy link
Contributor

willingc commented Nov 1, 2017

@jonathansp I return from vacation tomorrow. I’ll make sure to touch base and see what else, if anything, @minrk would like in the release. Thanks again for the contributions.

@minrk
Copy link
Member

minrk commented Nov 2, 2017

I think we can release 3.1 straight away with this (#131)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants