-
Notifications
You must be signed in to change notification settings - Fork 130
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add a pluggable strategy for storage #129
Conversation
Is this for use with JupyterHub? Or for other use cases? |
(also, yay for your first PR to this repo! welcome :D) |
@yuvipanda we gonna use with jupyter hub, although it can be used in any case. |
Personally, for JupyterHub I'd highly recommend implementing a custom proxy backend instead (http://jupyterhub.readthedocs.io/en/latest/reference/proxy.html) and using it with a different proxy (such as traefik) rather than adding more complexity to CHP. There's a half complete one for traefik + etcd in https://github.com/yuvipanda/jupyterhub-traefik - and more ones are easy to make. IMO that's the way forward rather than building and maintaining our own complex proxy implementation. |
I think it would be nice to have a pluggable storage options. I think it does not add complexity to this project because the implementation would be outside. |
@yuvipanda implementing a custom proxy was our first thought. However, as discussed in #85, and it seems to be a recurrent debate ( @minrk @rgbkrk @willingc @narenst @andrelu @gbedin ), CHP is a great software and would be nice to have more backends to work with. By following simple steps, we might build a redis-backend, a postgres-backend, an NFS backend etc. |
lib/store.js
Outdated
@@ -78,4 +78,4 @@ class MemoryStore extends BaseStore { | |||
} | |||
} | |||
|
|||
exports.MemoryStore = MemoryStore; | |||
module.exports = MemoryStore; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think this should be:
exports.BaseStore = BaseStore;
exports.MemoryStore = MemoryStore;
so that the BaseStore is exported and can be used.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@minrk, by doing this I also need to move BaseStore
to another file. The change I made expects a class to be exported, not a module. I did it because we don't need to know the class name we are dynamically importing (RedisStore, PGStore etc). Take a look:
const loadStorage = (options) => {
const BackendStorageClass = require(options.storageBackend || "./store.js");
return new BackendStorageClass(options);
};
Also, I'm not sure it is a good an idea for a plugin to use BaseStore
. We might have cyclic references. What do you think? IMHO, It's more like a duck-typing style than inheritance.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Duck typing is fine, but BaseStore exists for the purpose of being used in different implementations, potentially to reduce how much has to be duplicated. In particular, it implements some basic and second-order functionality that reduces what implementations must provide (e.g. providers don't strictly need to implement get
, only getAll
). It also gives us a place to add new second-order methods that providers would get for free.
If we aren't exporting BaseStore, there's no reason for BaseStore to exist and it can be subsumed into MemoryStore as a single class, rather than the sole subclass of an unusable base.
We can split the files if we really have to, but maybe it would be better to allow it to be an attribute rather than requiring all providers to use a dedicated module that only exports one class? Perhaps something like backend = './store.js:MemoryStore'
:
[modName, className] = backend.split(':');
backendMod = require(modName);
if (className) {
backendClass = backendMod[className];
else {
backendClass = backendMod;
}
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@minrk since it is not a javascript pattern (I suppose), I don't think it is a good idea. Unless you want to extend your storage strategies inside the library, I vote for removing BaseStore. It's going to leave the code simpler.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think the main issue here is that this is going to cause a major bump in semver when we could easily make this a minor bump adding the functionality. Keep the original keys on the exported interface and it will be alright.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@minrk @rgbkrk store.js
exports only MemoryStore
https://github.com/jupyterhub/configurable-http-proxy/blob/master/lib/store.js#L81, not BaseStore
.
In order to not make a brake change, what I can do is: keep exports.MemoryStore
and handle this import in loadStorage
. It will keep the same contract but sounds weird, since plugins must export a class and our own module exports an object.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ok, my only suggestion then is to keep the original MemoryStore export, we can defer the BaseStore
to outside the scope of this PR to at least finish bringing in the great work you've already done.
I agree with you about the potential for cyclic issues and for why the base store would need to end up in a separate file. I also see the ease at which a default export would provide. For now, let's keep it backwards compatible while moving forward.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This looks great! I'm very happy that you found a way to load the storage backend as a plugin without requiring it to be added to CHP itself. I left a minor comment inline. I imagine that at least some custom providers would want to Re: custom proxies, JupyterHub 0.8 introduces a Python API for its proxy requirements, making it more feasible to provide alternate implementations, as the Hub's API requirements are more clear. |
Before merging this PR, let me add one more test case. I'll work on it tomorrow morning. |
Cool! |
Thanks @jonathansp for the contribution 👍 |
hi everybody, we would like to let you know we finished our configurable-proxy-redis-backend. Please, feel free to make any suggestion! |
@jonathansp I return from vacation tomorrow. I’ll make sure to touch base and see what else, if anything, @minrk would like in the release. Thanks again for the contributions. |
I think we can release 3.1 straight away with this (#131) |
We'd like to suggest an alternative approach to solve #85. This PR adds the ability to plug an external strategy for configurable-http-proxy.
Currently, we have been working on an extension for redis: https://github.com/globocom/configurable-http-proxy-redis-backend