Automated message from Dropbox CLA bot

@jros2300, it looks like you've already signed the Dropbox CLA. Thanks!

Looks like the CI error is this -- probably need to add redis_cache to requirements.txt?

+python manage.py migrate --noinput
Traceback (most recent call last):
  File "manage.py", line 24, in <module>
    execute_from_command_line(sys.argv)
  File "/srv/zulip-venv/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 338, in execute_from_command_line
    utility.execute()
  File "/srv/zulip-venv/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 312, in execute
    django.setup()
  File "/srv/zulip-venv/local/lib/python2.7/site-packages/django/__init__.py", line 18, in setup
    apps.populate(settings.INSTALLED_APPS)
  File "/srv/zulip-venv/local/lib/python2.7/site-packages/django/apps/registry.py", line 85, in populate
    app_config = AppConfig.create(entry)
  File "/srv/zulip-venv/local/lib/python2.7/site-packages/django/apps/config.py", line 86, in create
    module = import_module(entry)
  File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
    __import__(name)
ImportError: No module named redis_cache
Traceback (most recent call last):
  File "provision.py", line 177, in <module>
    sys.exit(main())
  File "provision.py", line 172, in main
    sh.do_destroy_rebuild_database(**LOUD)
  File "/home/travis/virtualenv/python2.7.10/lib/python2.7/site-packages/pbs.py", line 456, in __call__
    return RunningCommand(command_ran, process, call_args, actual_stdin)
  File "/home/travis/virtualenv/python2.7.10/lib/python2.7/site-packages/pbs.py", line 168, in __init__
    self._handle_exit_code(self.process.wait())
  File "/home/travis/virtualenv/python2.7.10/lib/python2.7/site-packages/pbs.py", line 235, in _handle_exit_code
    raise get_rc_exc(rc)(self.command_ran, self._stdout, self._stderr)
pbs.ErrorReturnCode_1: 
Ran: './tools/do-destroy-rebuild-database'

Also for the production installation use case, we will need to package django-redis for Debian/Ubuntu...

Tim, please, can you help me?

The build in Travis is failing (in my dev environment it's working the Zulip server and all the tests). This is the failed log: https://travis-ci.org/zulip/zulip/jobs/97313710#L1520

In the line 1203 it finishes installing django-redis, and then it fails in the line 2990 because it cannot find the module django_redis (this is the name of the mudule in django-redis), I've tried in my dev and I can import that module name.

Is there something special to do for Travis?

Thanks, tell me when you can add the package to your ubuntu repository and I'll do the tests.

OK I've uploaded python-django-redis to the apt repository.

It's still failing, now it looks like it finds an old version of redis-py, where the redis.connection.ConnectionPool has no attribute 'from_url', but in the last version it has the method: https://github.com/andymccurdy/redis-py/blob/2.10.5/redis/connection.py

Ahh; looks like we need to backport python-redis from a newer version of Ubuntu to Trusty: http://packages.ubuntu.com/search?keywords=python-redis

OK @jros2300 I backported the redis package to trusty and put it in this repository for now: https://launchpad.net/~tabbott/+archive/ubuntu/zulip-testing
Can you test that this works? Once it's confirmed working, I can release it into the main PPA but I figure best to avoid changing what people are using in production until we've tested.

@timabbott I've updated the package and now all the tests passed

@jros2300 this has some relatively minor merge conflicts with #436 which I just merged, sorry about that!

@jros2300 I pulled your changes in as best I could on top of the current head, but there were some puppet configs that I don't know if I got right due to underlying changes. Can you take a look at https://github.com/007/zulip/commit/3e4585466fb3bb46ae65e228951c40c5e1cec5e5 to see if I got it right?

I also purged all of the other memcache* references throughout the codebase. The rest of the changes are in 007/zulip/redis-only and broken out to super-granular changes for @timabbott's pleasure.

Ping @jros2300 and @timabbott, what's the best way to get this (or my version) merged so we can eliminate one extraneous dependency for a zulip deploy?

We've had our memcached get killed by OOM a couple times this year, I'd really like the ability to remove it completely and focus memory on zulip, nginx, redis, postgres and rabbitmq.

Hi @007, there are 3 issues from my perspective still open:

• Redis replacement for Memcached #378 (comment) is one issue
• performance validation -- we make heavy use of get_many calls to fetch 1000 items from the cache, and there need to be a test for whether that's much slower with redis than memcached (which is probably more about the python bindings' performance, than about the actual backends)
• I would certainly be a lot more comfortable with a model where this migration were implemented as an option for a transitional period, so that if the migration is causing problems, there's an easy way for a team to switch back while we debug the issues.

For #​1, cache eviction should be able to be handled at the redis config level, but depending on your access patterns you might be able to get away with any of the eviction algorithms. If you absolutely have to keep some data vs some other data, this would need to be reworked to use two separate redis instances (with separate eviction policies) or to explicitly specify TTLs for volatile cache data vs non-volatile and set one of the volatile-* eviction policies.

For #​2, redis supports MGET as an analog to get_many, but I don't know if that's used as part of the cache binding further down the Django stack. My assumption would be "yes", but it's reasonable to require some kind of test.

Feature flipping is the bane of my existence 😆. In this case, however, I'd be happy to do the cache name refactoring as a separate merge, and then figure out what is required to implement this as a flagged feature. I think having everything named $VERB_memcache_$THING is deceptive, only because ~90% of the interfaces are using generic cache methods even if they happen to be operating on memcached under-the-hood.

Does that last point sound reasonable regardless of memcache-vs-redis? If so I can get started later today and should have something ready later this week. I'll check my available bandwidth to see if I can tackle #​2 anytime soon, and if that looks good I should have a better understanding to tackle point #​1.

@007 sorry for the slow reply; I've been a bit swamped with the flood of GSOC traffic. I'm definitely happy to take the changes to remove the hardcoding of "memcached cache" in the variable names, which will at least shrink the size of the patch to merge significantly.

I think it's probably possible to do the right thing with TTL or other explicit eviction policies, and it's probably better to do that than move 2 redis instances. Let me know if you run into any questions!

@007 If you want to talk about this pull request in realtime: We're hosting a livechat "office hour" at the open Zulip realm https://zulip.tabbott.net on Monday, August 15th, 10am-11am Pacific Time / 1pm-2pm Eastern Time / 10:30-11:30pm India time. Swing by if you want to talk about this, or about keyboard shortcuts for muting/unmuting, the experimental new desktop app, helping us figure out what new users need to know, etc. We'd be happy to see you.

Hello @jros2300, a Zulip maintainer reviewed this pull request over a week ago, but you haven't updated your pull request since. Please take a look at the requested changes and update your pull request accordingly.

Thank you for your valuable contributions to Zulip!

Hmm, well, the bot probably shouldn't have commented on something this old :/. I think I'll close this, since we're unlikely to merge it directly given the scale of conflicts. Thanks again for working on this though @jros2300!

(I have been having to resurrect this, this week after the sprint)