puppet: Update memcached config to turn on this decade's technology.

We've been running this change on zulipchat.com for a couple of months
now.  Before then, we used to regularly get exceptions like this:

     File "./zerver/views/messages.py", line 749, in get_messages_backend
       setter=stringify_message_dict)
     File "./zerver/lib/cache.py", line 275, in generic_bulk_cached_fetch
       cache_set_many(items_for_remote_cache)
     File "./zerver/lib/cache.py", line 215, in cache_set_many
       get_cache_backend(cache_name).set_many(items, timeout=timeout)
     File "/home/zulip/deployments/2017-09-28-21-04-12/zulip-py3-venv/lib/python3.5/site-packages/django/core/cache/backends/memcached.py", line 150, in set_many
       self._cache.set_multi(safe_data, self.get_backend_timeout(timeout))
   pylibmc.Error: error 48 from memcached_set_multi

This error means memcached was unable to find space for the new value.
You might think that because memcached provides an LRU cache, this
shouldn't happen because it would just evict something... but in fact
  * memcached splits its data into "slabs" by object size, and
  * until recently, once a 1MiB "chunk" is allocated to a given "slab"
    i.e. size class, it wouldn't be reclaimed to allocate to another.

So once the cache has been filled up with objects of some distribution
of sizes, if some objects come in that would go in a different size
class, we have no chunks for that size class / slab, and can't get one.
And that's exactly what was happening on zulipchat.com.

Useful background can be found in
  https://github.com/memcached/memcached/wiki/ServerMaint#slab-imbalance
  https://github.com/memcached/memcached/wiki/ReleaseNotes1411
  https://github.com/memcached/memcached/wiki/ReleaseNotes1425
  https://github.com/memcached/memcached/wiki/ReleaseNotes150
We're already running v1.4.25, which provides an "automover" that should
be well equipped to fix this; v1.5.0 turns it on by default.

With this commit, adopt the "modern start line" recommended in the
release notes for our v1.4.25, including turning on the automover.
This commit is contained in:
Greg Price
2018-02-08 15:33:47 -08:00
parent 4414f2b24b
commit 23e6a2e579

View File

@@ -34,6 +34,9 @@ logfile /var/log/memcached.log
# it's listening on a firewalled interface.
-l 127.0.0.1
# Use "modern" features.
-o slab_reassign,slab_automove,lru_crawler,lru_maintainer,maxconns_fast,hash_algorithm=murmur3
# Limit the number of simultaneous incoming connections. The daemon default is 1024
# -c 1024