Reverse commits changing sorted set code for 2.2

This also optimizes ZREVRANGEBYSCORE for pathological cases where a
sorted set contains many elements with the same score. Previously,
it would traverse the list from back to front in such a case.

00-RELEASENOTES

@@ -1,4 +1,26 @@
-Welcome to Redis 2.1.9 (2.2 Release Candidate 1)
+Redis 2.2 release notes
+
+Migrating from 2.0 to 2.2
+=========================
+
+Redis 2.0 is mostly a strict subset of 2.2. Some return value changed in edge
+cases, basicaly it is very unlikely that you will experience any problem
+upgrading your 2.0 instances to 2.2, as 2.2 can work as a drop in replacement
+for 2.0.
+
+What's new in Redis 2.1.10 (2.2 Release Candidate 2)
+====================================================
+
+Redis 2.2 RC2 is exactly like RC1 with the following minor changes:
+
+* Added evicted keys counter separated from expired keys.
+* Overflow detection in INCR family functions.
+
+Enjoy,
+Salvatore
+
+What's new in Redis 2.1.9 (2.2 Release Candidate 1)
+===================================================
 
 This is the first Release Candidate of Redis 2.2, in our experience the
 server is very stable, but in the latest weeks we rewrote part of the internals
@@ -7,7 +29,8 @@ a BGREWRITEAOF or a BGSAVE, so handle with care for a couple of weeks.
 
 Oh, and I've some very good news: the majority of apps can work if you simply replace 2.2 in your old 2.0 environment. I can't think of any breakage.
 
-WHAT'S NEW IN REDIS 2.2.x
+WHAT'S NEW IN REDIS 2.2 compared to the 2.0 version?
+====================================================
 
 * Specially encoded data types, small lists and sets can now use up to an order of magnitude less memory.
 * VM partial rewrite for code cleaness and memory usage.
@@ -26,10 +49,11 @@ WHAT'S NEW IN REDIS 2.2.x
 * BRPOPLPUSH (Thanks to Michel Martens and Damian Janowski)
 * Much more interesting informations in the INFO output.
 * Sorted sets are now less memory hungry.
-* Non blocking loading of .rdb / AOF file on startup, with process information in the INFO output.
+* Non blocking loading of .rdb / AOF file on startup, with progress information in the INFO output.
 * Now Redis has a clean, powerful, supported C library: hiredis.
 * Code layout completely new, the 2.0.x huge redis.c file is now splitted in many parts.
 * Redis-benchmark rewritten to be faster and in order to use hiredis as well.
+* Ability to rename or disable commands from the config file.
 * Endless other CPU optimizations and bugs fixed.
 
 Credits: Where not specified the implementation and design are done by Salvatore Sanfilippo and Pieter Noordhuis. Thanks to VMware for making all this possible. Also many thanks to all the other contributors and the amazing community we have.

TODO

@@ -1,66 +1,4 @@
-Redis TODO and Roadmap
-----------------------
+This is a stable release! No TODO file here.
+Please check the TODO file in the master branch on github.
 
-VERSION 2.2 TODO (Optimizations and latency)
-============================================
-
-* Support for syslog(3).
-* Change the implementation of ZCOUNT to use the augmented skiplist in order to be much faster.
-* Add an explicit test for MULTI/EXEC reloaded in the AOF.
-* Command table -> hash table, with support for command renaming
-
-VM TODO
-=======
-
-* Use multiple open FDs against the VM file, one for thread.
-* Check what happens performance-wise if instead of creating threads again and again the same threads are reused forever. Note: this requires a way to disable this clients in the child, but waiting for empty new jobs queue can be enough.
-
-STRING COMMANDS
-===============
-
-* Implement STRLEN, PEEK, POKE, SETBIT, GETBIT
-
-OTHER IMPORTANT THINGS THAT WILL BE ADDED BUT I'M NOT SURE WHEN
-===============================================================
-
-BIG ONES:
-
-* BRPOPLPUSH
-* Specially encoded memory-saving integer sets.
-* A command to export a JSON dump (there should be mostly working patch needing major reworking).
-* Specially encoded sets of integers (this includes a big refactoring providing an higher level layer for Sets manipulation)
-
-SMALL ONES:
-
-* If sizeof(double) == sizeof(void*) we could store the double value of sorted sets directly in place of the pointer instead of allocating it in the heap.
-* Delete on writes against expire policy should only happen after argument parsing for commands doing their own arg parsing stuff.
-* Give errors when incrementing a key that does not look like an integer, when providing as a sorted set score something can't be parsed as a double, and so forth.
-* MSADD (n keys) (n values). See this thread in the Redis google group: http://groups.google.com/group/redis-db/browse_thread/thread/e766d84eb375cd41
-* Don't save empty lists / sets / zsets on disk with snapshotting.
-* Remove keys when a list / set / zset reaches length of 0.
-* An option to exec a command slave-side if the master connection is lost: even cooler: if the script returns "0" the slave elects itself as master, otherwise continue trying to reconnect.
-* PING the master from time to time to check if it's gone.
-
-THE "MAYBE" TODO LIST: things that may or may not get implemented
-=================================================================
-
-Most of this can be seen just as proposals, the fact they are in this list
-it's not a guarantee they'll ever get implemented ;)
-
-* SORT: Don't copy the list into a vector when BY argument is constant.
-* Write the hash table size of every db in the dump, so that Redis can resize the hash table just one time when loading a big DB.
-* Byte Array type (BA prefixed commands): BASETBIT BAGETBIT BASETU8 U16 U32 U64 S8 S16 S32 S64, ability to atomically INCRBY all the base types. BARANGE to get a range of bytes as a bulk value, BASETRANGE to set a range of bytes.
-* Read-only mode.
-* Kill the delete-on-write behavior of expires, replicating DELs
-* Multiple BY in SORT.
-
-KNOWN BUGS
-==========
-
-* LRANGE and other commands are using 32 bit integers for ranges, and overflows are not detected. So LRANGE mylist 0 23498204823094823904823904 will have random effects.
-
-REDIS CLI TODO
-==============
-
-* Computer parsable output generation
-* Memoize return values so that they can be used later as arguments, like $1
+    https://github.com/antirez/redis/raw/master/TODO

redis.conf

@@ -365,17 +365,12 @@ vm-max-threads 4
 
 ############################### ADVANCED CONFIG ###############################
 
-# Glue small output buffers together in order to send small replies in a
-# single TCP packet. Uses a bit more CPU but most of the times it is a win
-# in terms of number of queries per second. Use 'yes' if unsure.
-glueoutputbuf yes
-
 # Hashes are encoded in a special way (much more memory efficient) when they
 # have at max a given numer of elements, and the biggest element does not
 # exceed a given threshold. You can configure this limits with the following
 # configuration directives.
-hash-max-zipmap-entries 64
-hash-max-zipmap-value 512
+hash-max-zipmap-entries 512
+hash-max-zipmap-value 64
 
 # Similarly to hashes, small lists are also encoded in a special way in order
 # to save a lot of space. The special representation is only used when

src/Makefile

@@ -8,9 +8,11 @@ OPTIMIZATION?=-O2
 ifeq ($(uname_S),SunOS)
   CFLAGS?= -std=c99 -pedantic $(OPTIMIZATION) -Wall -W -D__EXTENSIONS__ -D_XPG6
   CCLINK?= -ldl -lnsl -lsocket -lm -lpthread
+  DEBUG?= -g -ggdb 
 else
   CFLAGS?= -std=c99 -pedantic $(OPTIMIZATION) -Wall -W $(ARCH) $(PROF)
   CCLINK?= -lm -pthread
+  DEBUG?= -g -rdynamic -ggdb 
 endif
 
 ifeq ($(USE_TCMALLOC),yes)
@@ -18,7 +20,6 @@ ifeq ($(USE_TCMALLOC),yes)
   CFLAGS+= -DUSE_TCMALLOC
 endif
 CCOPT= $(CFLAGS) $(CCLINK) $(ARCH) $(PROF)
-DEBUG?= -g -rdynamic -ggdb 
 
 PREFIX= /usr/local
 INSTALL_BIN= $(PREFIX)/bin

src/anet.c

@@ -64,11 +64,11 @@ int anetNonBlock(char *err, int fd)
      * Note that fcntl(2) for F_GETFL and F_SETFL can't be
      * interrupted by a signal. */
     if ((flags = fcntl(fd, F_GETFL)) == -1) {
-        anetSetError(err, "fcntl(F_GETFL): %s\n", strerror(errno));
+        anetSetError(err, "fcntl(F_GETFL): %s", strerror(errno));
         return ANET_ERR;
     }
     if (fcntl(fd, F_SETFL, flags | O_NONBLOCK) == -1) {
-        anetSetError(err, "fcntl(F_SETFL,O_NONBLOCK): %s\n", strerror(errno));
+        anetSetError(err, "fcntl(F_SETFL,O_NONBLOCK): %s", strerror(errno));
         return ANET_ERR;
     }
     return ANET_OK;
@@ -79,7 +79,7 @@ int anetTcpNoDelay(char *err, int fd)
     int yes = 1;
     if (setsockopt(fd, IPPROTO_TCP, TCP_NODELAY, &yes, sizeof(yes)) == -1)
     {
-        anetSetError(err, "setsockopt TCP_NODELAY: %s\n", strerror(errno));
+        anetSetError(err, "setsockopt TCP_NODELAY: %s", strerror(errno));
         return ANET_ERR;
     }
     return ANET_OK;
@@ -89,7 +89,7 @@ int anetSetSendBuffer(char *err, int fd, int buffsize)
 {
     if (setsockopt(fd, SOL_SOCKET, SO_SNDBUF, &buffsize, sizeof(buffsize)) == -1)
     {
-        anetSetError(err, "setsockopt SO_SNDBUF: %s\n", strerror(errno));
+        anetSetError(err, "setsockopt SO_SNDBUF: %s", strerror(errno));
         return ANET_ERR;
     }
     return ANET_OK;
@@ -99,7 +99,7 @@ int anetTcpKeepAlive(char *err, int fd)
 {
     int yes = 1;
     if (setsockopt(fd, SOL_SOCKET, SO_KEEPALIVE, &yes, sizeof(yes)) == -1) {
-        anetSetError(err, "setsockopt SO_KEEPALIVE: %s\n", strerror(errno));
+        anetSetError(err, "setsockopt SO_KEEPALIVE: %s", strerror(errno));
         return ANET_ERR;
     }
     return ANET_OK;
@@ -115,7 +115,7 @@ int anetResolve(char *err, char *host, char *ipbuf)
 
         he = gethostbyname(host);
         if (he == NULL) {
-            anetSetError(err, "can't resolve: %s\n", host);
+            anetSetError(err, "can't resolve: %s", host);
             return ANET_ERR;
         }
         memcpy(&sa.sin_addr, he->h_addr, sizeof(struct in_addr));
@@ -127,14 +127,14 @@ int anetResolve(char *err, char *host, char *ipbuf)
 static int anetCreateSocket(char *err, int domain) {
     int s, on = 1;
     if ((s = socket(domain, SOCK_STREAM, 0)) == -1) {
-        anetSetError(err, "creating socket: %s\n", strerror(errno));
+        anetSetError(err, "creating socket: %s", strerror(errno));
         return ANET_ERR;
     }
 
     /* Make sure connection-intensive things like the redis benckmark
      * will be able to close/open sockets a zillion of times */
     if (setsockopt(s, SOL_SOCKET, SO_REUSEADDR, &on, sizeof(on)) == -1) {
-        anetSetError(err, "setsockopt SO_REUSEADDR: %s\n", strerror(errno));
+        anetSetError(err, "setsockopt SO_REUSEADDR: %s", strerror(errno));
         return ANET_ERR;
     }
     return s;
@@ -157,7 +157,7 @@ static int anetTcpGenericConnect(char *err, char *addr, int port, int flags)
 
         he = gethostbyname(addr);
         if (he == NULL) {
-            anetSetError(err, "can't resolve: %s\n", addr);
+            anetSetError(err, "can't resolve: %s", addr);
             close(s);
             return ANET_ERR;
         }
@@ -172,7 +172,7 @@ static int anetTcpGenericConnect(char *err, char *addr, int port, int flags)
             flags & ANET_CONNECT_NONBLOCK)
             return s;
 
-        anetSetError(err, "connect: %s\n", strerror(errno));
+        anetSetError(err, "connect: %s", strerror(errno));
         close(s);
         return ANET_ERR;
     }
@@ -208,7 +208,7 @@ int anetUnixGenericConnect(char *err, char *path, int flags)
             flags & ANET_CONNECT_NONBLOCK)
             return s;
 
-        anetSetError(err, "connect: %s\n", strerror(errno));
+        anetSetError(err, "connect: %s", strerror(errno));
         close(s);
         return ANET_ERR;
     }
@@ -257,12 +257,12 @@ int anetWrite(int fd, char *buf, int count)
 
 static int anetListen(char *err, int s, struct sockaddr *sa, socklen_t len) {
     if (bind(s,sa,len) == -1) {
-        anetSetError(err, "bind: %s\n", strerror(errno));
+        anetSetError(err, "bind: %s", strerror(errno));
         close(s);
         return ANET_ERR;
     }
     if (listen(s, 511) == -1) { /* the magic 511 constant is from nginx */
-        anetSetError(err, "listen: %s\n", strerror(errno));
+        anetSetError(err, "listen: %s", strerror(errno));
         close(s);
         return ANET_ERR;
     }
@@ -282,7 +282,7 @@ int anetTcpServer(char *err, int port, char *bindaddr)
     sa.sin_port = htons(port);
     sa.sin_addr.s_addr = htonl(INADDR_ANY);
     if (bindaddr && inet_aton(bindaddr, &sa.sin_addr) == 0) {
-        anetSetError(err, "Invalid bind address\n");
+        anetSetError(err, "invalid bind address");
         close(s);
         return ANET_ERR;
     }
@@ -315,7 +315,7 @@ static int anetGenericAccept(char *err, int s, struct sockaddr *sa, socklen_t *l
             if (errno == EINTR)
                 continue;
             else {
-                anetSetError(err, "accept: %s\n", strerror(errno));
+                anetSetError(err, "accept: %s", strerror(errno));
                 return ANET_ERR;
             }
         }

src/anet.h

@@ -35,6 +35,10 @@
 #define ANET_ERR -1
 #define ANET_ERR_LEN 256
 
+#if defined(__sun)
+#define AF_LOCAL AF_UNIX
+#endif
+
 int anetTcpConnect(char *err, char *addr, int port);
 int anetTcpNonBlockConnect(char *err, char *addr, int port);
 int anetUnixConnect(char *err, char *path);

src/config.c

@@ -194,10 +194,8 @@ void loadServerConfig(char *filename) {
             if ((server.repl_serve_stale_data = yesnotoi(argv[1])) == -1) {
                 err = "argument must be 'yes' or 'no'"; goto loaderr;
             }
-        } else if (!strcasecmp(argv[0],"glueoutputbuf") && argc == 2) {
-            if ((server.glueoutputbuf = yesnotoi(argv[1])) == -1) {
-                err = "argument must be 'yes' or 'no'"; goto loaderr;
-            }
+        } else if (!strcasecmp(argv[0],"glueoutputbuf")) {
+            redisLog(REDIS_WARNING, "Deprecated configuration directive: \"%s\"", argv[0]);
         } else if (!strcasecmp(argv[0],"rdbcompression") && argc == 2) {
             if ((server.rdbcompression = yesnotoi(argv[1])) == -1) {
                 err = "argument must be 'yes' or 'no'"; goto loaderr;

src/db.c

@@ -453,6 +453,8 @@ void propagateExpire(redisDb *db, robj *key) {
 int expireIfNeeded(redisDb *db, robj *key) {
     time_t when = getExpire(db,key);
 
+    if (when < 0) return 0; /* No expire for this key */
+
     /* If we are running in the context of a slave, return ASAP:
      * the slave key expiration is controlled by the master that will
      * send us synthesized DEL operations for expired keys.
@@ -464,8 +466,6 @@ int expireIfNeeded(redisDb *db, robj *key) {
         return time(NULL) > when;
     }
 
-    if (when < 0) return 0;
-
     /* Return when this key has not expired */
     if (time(NULL) <= when) return 0;
 

src/help.h

@@ -1,4 +1,4 @@
-/* Automatically generated by utils/generate-command-help.rb, do not edit. */
+/* Automatically generated by generate-command-help.rb, do not edit. */
 
 #ifndef __REDIS_HELP_H
 #define __REDIS_HELP_H
@@ -53,11 +53,21 @@ struct commandHelp {
     "Remove and get the last element in a list, or block until one is available",
     2,
     "1.3.1" },
+    { "BRPOPLPUSH",
+    "source destination timeout",
+    "Pop a value from a list, push it to another list and return it; or block until one is available",
+    2,
+    "2.1.7" },
     { "CONFIG GET",
     "parameter",
     "Get the value of a configuration parameter",
     9,
     "2.0" },
+    { "CONFIG RESETSTAT",
+    "-",
+    "Reset the stats returned by INFO",
+    9,
+    "2.0" },
     { "CONFIG SET",
     "parameter value",
     "Set a configuration parameter to the given value",
@@ -79,7 +89,7 @@ struct commandHelp {
     9,
     "0.101" },
     { "DECR",
-    "key decrement",
+    "key",
     "Decrement the integer value of a key by one",
     1,
     "0.07" },
@@ -138,6 +148,11 @@ struct commandHelp {
     "Get the value of a key",
     1,
     "0.07" },
+    { "GETBIT",
+    "key offset",
+    "Returns the bit value at offset in the string value stored at key",
+    1,
+    "2.1.8" },
     { "GETSET",
     "key value",
     "Set the string value of a key and return its old value",
@@ -344,12 +359,12 @@ struct commandHelp {
     0,
     "0.07" },
     { "RENAME",
-    "old new",
+    "key newkey",
     "Rename a key",
     0,
     "0.07" },
     { "RENAMENX",
-    "old new",
+    "key newkey",
     "Rename a key, only if the new key does not exist",
     0,
     "0.07" },
@@ -408,8 +423,13 @@ struct commandHelp {
     "Set the string value of a key",
     1,
     "0.07" },
+    { "SETBIT",
+    "key offset value",
+    "Sets or clears the bit at offset in the string value stored at key",
+    1,
+    "2.1.8" },
     { "SETEX",
-    "key timestamp value",
+    "key seconds value",
     "Set the value and expiration of a key",
     1,
     "1.3.10" },
@@ -418,6 +438,11 @@ struct commandHelp {
     "Set the value of a key, only if the key does not exist",
     1,
     "0.07" },
+    { "SETRANGE",
+    "key offset value",
+    "Overwrite part of a string at key starting at the specified offset",
+    1,
+    "2.1.8" },
     { "SHUTDOWN",
     "-",
     "Synchronously save the dataset to disk and then shut down the server",
@@ -454,7 +479,7 @@ struct commandHelp {
     3,
     "0.091" },
     { "SORT",
-    "key [BY pattern] [LIMIT start count] [GET pattern [GET pattern ...]] [ASC|DESC] [ALPHA] [STORE destination]",
+    "key [BY pattern] [LIMIT offset count] [GET pattern [GET pattern ...]] [ASC|DESC] [ALPHA] [STORE destination]",
     "Sort the elements in a list, set or sorted set",
     0,
     "0.07" },
@@ -484,7 +509,7 @@ struct commandHelp {
     6,
     "1.3.8" },
     { "SUBSTR",
-    "key start stop",
+    "key start end",
     "Get a substring of the string stored at a key",
     1,
     "1.3.4" },
@@ -549,17 +574,17 @@ struct commandHelp {
     4,
     "1.1" },
     { "ZINTERSTORE",
-    "destination key [key ...] [WEIGHTS weight] [AGGREGATE SUM|MIN|MAX]",
+    "destination numkeys key [key ...] [WEIGHTS weight] [AGGREGATE SUM|MIN|MAX]",
     "Intersect multiple sorted sets and store the resulting sorted set in a new key",
     4,
     "1.3.10" },
     { "ZRANGE",
-    "key start stop",
+    "key start stop [WITHSCORES]",
     "Return a range of members in a sorted set, by index",
     4,
     "1.1" },
     { "ZRANGEBYSCORE",
-    "key min max",
+    "key min max [WITHSCORES] [LIMIT offset count]",
     "Return a range of members in a sorted set, by score",
     4,
     "1.050" },
@@ -584,10 +609,15 @@ struct commandHelp {
     4,
     "1.1" },
     { "ZREVRANGE",
-    "key start stop",
+    "key start stop [WITHSCORES]",
     "Return a range of members in a sorted set, by index, with scores ordered from high to low",
     4,
     "1.1" },
+    { "ZREVRANGEBYSCORE",
+    "key max min [WITHSCORES] [LIMIT offset count]",
+    "Return a range of members in a sorted set, by score, with scores ordered from high to low",
+    4,
+    "2.1.6" },
     { "ZREVRANK",
     "key member",
     "Determine the index of a member in a sorted set, with scores ordered from high to low",
@@ -599,7 +629,7 @@ struct commandHelp {
     4,
     "1.1" },
     { "ZUNIONSTORE",
-    "destination key [key ...] [WEIGHTS weight] [AGGREGATE SUM|MIN|MAX]",
+    "destination numkeys key [key ...] [WEIGHTS weight] [AGGREGATE SUM|MIN|MAX]",
     "Add multiple sorted sets and store the resulting sorted set in a new key",
     4,
     "1.3.10" }

src/networking.c

@@ -457,6 +457,13 @@ void freeClient(redisClient *c) {
     ln = listSearchKey(server.clients,c);
     redisAssert(ln != NULL);
     listDelNode(server.clients,ln);
+    /* When client was just unblocked because of a blocking operation,
+     * remove it from the list with unblocked clients. */
+    if (c->flags & REDIS_UNBLOCKED) {
+        ln = listSearchKey(server.unblocked_clients,c);
+        redisAssert(ln != NULL);
+        listDelNode(server.unblocked_clients,ln);
+    }
     /* Remove from the list of clients waiting for swapped keys, or ready
      * to be restarted, but not yet woken up again. */
     if (c->flags & REDIS_IO_WAIT) {
@@ -516,15 +523,6 @@ void sendReplyToClient(aeEventLoop *el, int fd, void *privdata, int mask) {
     REDIS_NOTUSED(el);
     REDIS_NOTUSED(mask);
 
-    /* Use writev() if we have enough buffers to send */
-    if (!server.glueoutputbuf &&
-        listLength(c->reply) > REDIS_WRITEV_THRESHOLD &&
-        !(c->flags & REDIS_MASTER))
-    {
-        sendReplyToClientWritev(el, fd, privdata, mask);
-        return;
-    }
-
     while(c->bufpos > 0 || listLength(c->reply)) {
         if (c->bufpos > 0) {
             if (c->flags & REDIS_MASTER) {
@@ -595,84 +593,6 @@ void sendReplyToClient(aeEventLoop *el, int fd, void *privdata, int mask) {
     }
 }
 
-void sendReplyToClientWritev(aeEventLoop *el, int fd, void *privdata, int mask)
-{
-    redisClient *c = privdata;
-    int nwritten = 0, totwritten = 0, objlen, willwrite;
-    robj *o;
-    struct iovec iov[REDIS_WRITEV_IOVEC_COUNT];
-    int offset, ion = 0;
-    REDIS_NOTUSED(el);
-    REDIS_NOTUSED(mask);
-
-    listNode *node;
-    while (listLength(c->reply)) {
-        offset = c->sentlen;
-        ion = 0;
-        willwrite = 0;
-
-        /* fill-in the iov[] array */
-        for(node = listFirst(c->reply); node; node = listNextNode(node)) {
-            o = listNodeValue(node);
-            objlen = sdslen(o->ptr);
-
-            if (totwritten + objlen - offset > REDIS_MAX_WRITE_PER_EVENT)
-                break;
-
-            if(ion == REDIS_WRITEV_IOVEC_COUNT)
-                break; /* no more iovecs */
-
-            iov[ion].iov_base = ((char*)o->ptr) + offset;
-            iov[ion].iov_len = objlen - offset;
-            willwrite += objlen - offset;
-            offset = 0; /* just for the first item */
-            ion++;
-        }
-
-        if(willwrite == 0)
-            break;
-
-        /* write all collected blocks at once */
-        if((nwritten = writev(fd, iov, ion)) < 0) {
-            if (errno != EAGAIN) {
-                redisLog(REDIS_VERBOSE,
-                         "Error writing to client: %s", strerror(errno));
-                freeClient(c);
-                return;
-            }
-            break;
-        }
-
-        totwritten += nwritten;
-        offset = c->sentlen;
-
-        /* remove written robjs from c->reply */
-        while (nwritten && listLength(c->reply)) {
-            o = listNodeValue(listFirst(c->reply));
-            objlen = sdslen(o->ptr);
-
-            if(nwritten >= objlen - offset) {
-                listDelNode(c->reply, listFirst(c->reply));
-                nwritten -= objlen - offset;
-                c->sentlen = 0;
-            } else {
-                /* partial write */
-                c->sentlen += nwritten;
-                break;
-            }
-            offset = 0;
-        }
-    }
-
-    if (totwritten > 0)
-        c->lastinteraction = time(NULL);
-
-    if (listLength(c->reply) == 0) {
-        c->sentlen = 0;
-        aeDeleteFileEvent(server.el,c->fd,AE_WRITABLE);
-    }
-}
-
 /* resetClient prepare the client to process the next command */
 void resetClient(redisClient *c) {
     freeClientArgv(c);
@@ -908,3 +828,22 @@ void readQueryFromClient(aeEventLoop *el, int fd, void *privdata, int mask) {
     }
     processInputBuffer(c);
 }
+
+void getClientsMaxBuffers(unsigned long *longest_output_list,
+                          unsigned long *biggest_input_buffer) {
+    redisClient *c;
+    listNode *ln;
+    listIter li;
+    unsigned long lol = 0, bib = 0;
+
+    listRewind(server.clients,&li);
+    while ((ln = listNext(&li)) != NULL) {
+        c = listNodeValue(ln);
+
+        if (listLength(c->reply) > lol) lol = listLength(c->reply);
+        if (sdslen(c->querybuf) > bib) bib = sdslen(c->querybuf);
+    }
+    *longest_output_list = lol;
+    *biggest_input_buffer = bib;
+}
+

src/redis-cli.c

@@ -414,10 +414,11 @@ static sds cliFormatReplyRaw(redisReply *r) {
 }
 
 static int cliReadReply(int output_raw_strings) {
+    void *_reply;
     redisReply *reply;
     sds out;
 
-    if (redisGetReply(context,(void**)&reply) != REDIS_OK) {
+    if (redisGetReply(context,&_reply) != REDIS_OK) {
         if (config.shutdown)
             return REDIS_OK;
         if (config.interactive) {
@@ -431,6 +432,7 @@ static int cliReadReply(int output_raw_strings) {
         return REDIS_ERR; /* avoid compiler warning */
     }
 
+    reply = (redisReply*)_reply;
     if (output_raw_strings) {
         out = cliFormatReplyRaw(reply);
     } else {

src/redis.c

@@ -687,6 +687,7 @@ void beforeSleep(struct aeEventLoop *eventLoop) {
         redisAssert(ln != NULL);
         c = ln->value;
         listDelNode(server.unblocked_clients,ln);
+        c->flags &= ~REDIS_UNBLOCKED;
 
         /* Process remaining data in the input buffer. */
         if (c->querybuf && sdslen(c->querybuf) > 0)
@@ -768,7 +769,6 @@ void initServerConfig() {
     server.syslog_enabled = 0;
     server.syslog_ident = zstrdup("redis");
     server.syslog_facility = LOG_LOCAL0;
-    server.glueoutputbuf = 1;
     server.daemonize = 0;
     server.appendonly = 0;
     server.appendfsync = APPENDFSYNC_EVERYSEC;
@@ -891,6 +891,7 @@ void initServer() {
     server.stat_numcommands = 0;
     server.stat_numconnections = 0;
     server.stat_expiredkeys = 0;
+    server.stat_evictedkeys = 0;
     server.stat_starttime = time(NULL);
     server.stat_keyspace_misses = 0;
     server.stat_keyspace_hits = 0;
@@ -1141,9 +1142,11 @@ sds genRedisInfoString(void) {
     int j;
     char hmem[64];
     struct rusage self_ru, c_ru;
+    unsigned long lol, bib;
 
     getrusage(RUSAGE_SELF, &self_ru);
     getrusage(RUSAGE_CHILDREN, &c_ru);
+    getClientsMaxBuffers(&lol,&bib);
 
     bytesToHuman(hmem,zmalloc_used_memory());
     info = sdscatprintf(sdsempty(),
@@ -1162,6 +1165,8 @@ sds genRedisInfoString(void) {
         "used_cpu_user_childrens:%.2f\r\n"
         "connected_clients:%d\r\n"
         "connected_slaves:%d\r\n"
+        "client_longest_output_list:%lu\r\n"
+        "client_biggest_input_buf:%lu\r\n"
         "blocked_clients:%d\r\n"
         "used_memory:%zu\r\n"
         "used_memory_human:%s\r\n"
@@ -1177,6 +1182,7 @@ sds genRedisInfoString(void) {
         "total_connections_received:%lld\r\n"
         "total_commands_processed:%lld\r\n"
         "expired_keys:%lld\r\n"
+        "evicted_keys:%lld\r\n"
         "keyspace_hits:%lld\r\n"
         "keyspace_misses:%lld\r\n"
         "hash_max_zipmap_entries:%zu\r\n"
@@ -1200,6 +1206,7 @@ sds genRedisInfoString(void) {
         (float)c_ru.ru_stime.tv_sec+(float)c_ru.ru_stime.tv_usec/1000000,
         listLength(server.clients)-listLength(server.slaves),
         listLength(server.slaves),
+        lol, bib,
         server.bpop_blocked_clients,
         zmalloc_used_memory(),
         hmem,
@@ -1219,6 +1226,7 @@ sds genRedisInfoString(void) {
         server.stat_numconnections,
         server.stat_numcommands,
         server.stat_expiredkeys,
+        server.stat_evictedkeys,
         server.stat_keyspace_hits,
         server.stat_keyspace_misses,
         server.hash_max_zipmap_entries,
@@ -1311,6 +1319,19 @@ sds genRedisInfoString(void) {
             eta
         );
     }
+
+    info = sdscat(info,"allocation_stats:");
+    for (j = 0; j <= ZMALLOC_MAX_ALLOC_STAT; j++) {
+        size_t count = zmalloc_allocations_for_size(j);
+        if (count) {
+            if (info[sdslen(info)-1] != ':') info = sdscatlen(info,",",1);
+            info = sdscatprintf(info,"%s%d=%zu",
+                (j == ZMALLOC_MAX_ALLOC_STAT) ? ">=" : "",
+                j,count);
+        }
+    }
+    info = sdscat(info,"\r\n");
+
     for (j = 0; j < server.dbnum; j++) {
         long long keys, vkeys;
 
@@ -1436,44 +1457,13 @@ void freeMemoryIfNeeded(void) {
             if (bestkey) {
                 robj *keyobj = createStringObject(bestkey,sdslen(bestkey));
                 dbDelete(db,keyobj);
-                server.stat_expiredkeys++;
+                server.stat_evictedkeys++;
                 decrRefCount(keyobj);
                 freed++;
             }
         }
         if (!freed) return; /* nothing to free... */
     }
-
-    while(0) {
-        int j, k, freed = 0;
-        for (j = 0; j < server.dbnum; j++) {
-            int minttl = -1;
-            sds minkey = NULL;
-            robj *keyobj = NULL;
-            struct dictEntry *de;
-
-            if (dictSize(server.db[j].expires)) {
-                freed = 1;
-                /* From a sample of three keys drop the one nearest to
-                 * the natural expire */
-                for (k = 0; k < 3; k++) {
-                    time_t t;
-
-                    de = dictGetRandomKey(server.db[j].expires);
-                    t = (time_t) dictGetEntryVal(de);
-                    if (minttl == -1 || t < minttl) {
-                        minkey = dictGetEntryKey(de);
-                        minttl = t;
-                    }
-                }
-                keyobj = createStringObject(minkey,sdslen(minkey));
-                dbDelete(server.db+j,keyobj);
-                server.stat_expiredkeys++;
-                decrRefCount(keyobj);
-            }
-        }
-        if (!freed) return; /* nothing to free... */
-    }
 }
 
 /* =================================== Main! ================================ */
@@ -1504,7 +1494,7 @@ void createPidFile(void) {
     /* Try to write the pid file in a best-effort way. */
     FILE *fp = fopen(server.pidfile,"w");
     if (fp) {
-        fprintf(fp,"%d\n",getpid());
+        fprintf(fp,"%d\n",(int)getpid());
         fclose(fp);
     }
 }

src/redis.h

@@ -50,11 +50,6 @@
 #define REDIS_REPLY_CHUNK_BYTES (5*1500) /* 5 TCP packets with default MTU */
 #define REDIS_MAX_LOGMSG_LEN    1024 /* Default maximum length of syslog messages */
 
-/* If more then REDIS_WRITEV_THRESHOLD write packets are pending use writev */
-#define REDIS_WRITEV_THRESHOLD      3
-/* Max number of iovecs used for each writev call */
-#define REDIS_WRITEV_IOVEC_COUNT    256
-
 /* Hash table parameters */
 #define REDIS_HT_MINFILL        10      /* Minimal hash table fill 10% */
 
@@ -146,6 +141,8 @@
 #define REDIS_IO_WAIT 32    /* The client is waiting for Virtual Memory I/O */
 #define REDIS_DIRTY_CAS 64  /* Watched keys modified. EXEC will fail. */
 #define REDIS_CLOSE_AFTER_REPLY 128 /* Close after writing entire reply. */
+#define REDIS_UNBLOCKED 256 /* This client was unblocked and is stored in
+                               server.unblocked_clients */
 
 /* Client request types */
 #define REDIS_REQ_INLINE 1
@@ -194,8 +191,8 @@
 #define APPENDFSYNC_EVERYSEC 2
 
 /* Zip structure related defaults */
-#define REDIS_HASH_MAX_ZIPMAP_ENTRIES 64
-#define REDIS_HASH_MAX_ZIPMAP_VALUE 512
+#define REDIS_HASH_MAX_ZIPMAP_ENTRIES 512
+#define REDIS_HASH_MAX_ZIPMAP_VALUE 64
 #define REDIS_LIST_MAX_ZIPLIST_ENTRIES 512
 #define REDIS_LIST_MAX_ZIPLIST_VALUE 64
 #define REDIS_SET_MAX_INTSET_ENTRIES 512
@@ -387,11 +384,11 @@ struct redisServer {
     long long stat_numcommands;     /* number of processed commands */
     long long stat_numconnections;  /* number of connections received */
     long long stat_expiredkeys;     /* number of expired keys */
+    long long stat_evictedkeys;     /* number of evicted keys (maxmemory) */
     long long stat_keyspace_hits;   /* number of successful lookups of keys */
     long long stat_keyspace_misses; /* number of failed lookups of keys */
     /* Configuration */
     int verbosity;
-    int glueoutputbuf;
     int maxidletime;
     int dbnum;
     int daemonize;
@@ -640,7 +637,6 @@ void closeTimedoutClients(void);
 void freeClient(redisClient *c);
 void resetClient(redisClient *c);
 void sendReplyToClient(aeEventLoop *el, int fd, void *privdata, int mask);
-void sendReplyToClientWritev(aeEventLoop *el, int fd, void *privdata, int mask);
 void addReply(redisClient *c, robj *obj);
 void *addDeferredMultiBulkLength(redisClient *c);
 void setDeferredMultiBulkLength(redisClient *c, void *node, long length);
@@ -662,6 +658,8 @@ void addReplyDouble(redisClient *c, double d);
 void addReplyLongLong(redisClient *c, long long ll);
 void addReplyMultiBulkLen(redisClient *c, long length);
 void *dupClientReplyValue(void *o);
+void getClientsMaxBuffers(unsigned long *longest_output_list,
+                          unsigned long *biggest_input_buffer);
 
 #ifdef __GNUC__
 void addReplyErrorFormat(redisClient *c, const char *fmt, ...)

src/t_list.c

@@ -773,7 +773,8 @@ void unblockClientWaitingData(redisClient *c) {
     zfree(c->bpop.keys);
     c->bpop.keys = NULL;
     c->bpop.target = NULL;
-    c->flags &= (~REDIS_BLOCKED);
+    c->flags &= ~REDIS_BLOCKED;
+    c->flags |= REDIS_UNBLOCKED;
     server.bpop_blocked_clients--;
     listAddNodeTail(server.unblocked_clients,c);
 }

src/t_string.c

@@ -1,4 +1,3 @@
-#include <limits.h>
 #include "redis.h"
 
 /*-----------------------------------------------------------------------------
@@ -346,14 +345,19 @@ void msetnxCommand(redisClient *c) {
 }
 
 void incrDecrCommand(redisClient *c, long long incr) {
-    long long value;
+    long long value, oldvalue;
     robj *o;
 
     o = lookupKeyWrite(c->db,c->argv[1]);
     if (o != NULL && checkType(c,o,REDIS_STRING)) return;
     if (getLongLongFromObjectOrReply(c,o,&value,NULL) != REDIS_OK) return;
 
+    oldvalue = value;
     value += incr;
+    if ((incr < 0 && value > oldvalue) || (incr > 0 && value < oldvalue)) {
+        addReplyError(c,"increment or decrement would overflow");
+        return;
+    }
     o = createStringObjectFromLongLong(value);
     dbReplace(c->db,c->argv[1],o);
     touchWatchedKey(c->db,c->argv[1]);

src/t_zset.c

@@ -263,7 +263,7 @@ zskiplistNode *zslFirstWithScore(zskiplist *zsl, double score) {
  * Returns 0 when the element cannot be found, rank otherwise.
  * Note that the rank is 1-based due to the span of zsl->header to the
  * first element. */
-unsigned long zslistTypeGetRank(zskiplist *zsl, double score, robj *o) {
+unsigned long zslGetRank(zskiplist *zsl, double score, robj *o) {
     zskiplistNode *x;
     unsigned long rank = 0;
     int i;
@@ -287,7 +287,7 @@ unsigned long zslistTypeGetRank(zskiplist *zsl, double score, robj *o) {
 }
 
 /* Finds an element by its rank. The rank argument needs to be 1-based. */
-zskiplistNode* zslistTypeGetElementByRank(zskiplist *zsl, unsigned long rank) {
+zskiplistNode* zslGetElementByRank(zskiplist *zsl, unsigned long rank) {
     zskiplistNode *x;
     unsigned long traversed = 0;
     int i;
@@ -810,10 +810,10 @@ void zrangeGenericCommand(redisClient *c, int reverse) {
     /* check if starting point is trivial, before searching
      * the element in log(N) time */
     if (reverse) {
-        ln = start == 0 ? zsl->tail : zslistTypeGetElementByRank(zsl, llen-start);
+        ln = start == 0 ? zsl->tail : zslGetElementByRank(zsl, llen-start);
     } else {
         ln = start == 0 ?
-            zsl->header->level[0].forward : zslistTypeGetElementByRank(zsl, start+1);
+            zsl->header->level[0].forward : zslGetElementByRank(zsl, start+1);
     }
 
     /* Return the result in form of a multi-bulk reply */
@@ -1039,7 +1039,7 @@ void zrankGenericCommand(redisClient *c, int reverse) {
     }
 
     score = dictGetEntryVal(de);
-    rank = zslistTypeGetRank(zsl, *score, c->argv[2]);
+    rank = zslGetRank(zsl, *score, c->argv[2]);
     if (rank) {
         if (reverse) {
             addReplyLongLong(c, zsl->length - rank);

src/version.h

@@ -1 +1 @@
-#define REDIS_VERSION "2.1.8"
+#define REDIS_VERSION "2.1.11"

src/ziplist.c

@@ -119,6 +119,7 @@ static unsigned int zipEntryEncoding(unsigned char *p) {
         return p[0] & 0xf0;
     }
     assert(NULL);
+    return 0;
 }
 
 /* Return bytes needed to store integer encoded by 'encoding' */
@@ -129,13 +130,14 @@ static unsigned int zipIntSize(unsigned char encoding) {
     case ZIP_INT_64B: return sizeof(int64_t);
     }
     assert(NULL);
+    return 0;
 }
 
 /* Decode the encoded length pointed by 'p'. If a pointer to 'lensize' is
  * provided, it is set to the number of bytes required to encode the length. */
 static unsigned int zipDecodeLength(unsigned char *p, unsigned int *lensize) {
     unsigned char encoding = zipEntryEncoding(p);
-    unsigned int len;
+    unsigned int len = 0;
 
     if (ZIP_IS_STR(encoding)) {
         switch(encoding) {
@@ -300,7 +302,7 @@ static void zipSaveInteger(unsigned char *p, int64_t value, unsigned char encodi
 static int64_t zipLoadInteger(unsigned char *p, unsigned char encoding) {
     int16_t i16;
     int32_t i32;
-    int64_t i64, ret;
+    int64_t i64, ret = 0;
     if (encoding == ZIP_INT_16B) {
         memcpy(&i16,p,sizeof(i16));
         ret = i16;

src/zmalloc.c

@@ -33,6 +33,7 @@
 #include <string.h>
 #include <pthread.h>
 #include "config.h"
+#include "zmalloc.h"
 
 #ifdef HAVE_MALLOC_SIZE
 #define PREFIX_SIZE (0)
@@ -52,19 +53,22 @@
 #define free(ptr) tc_free(ptr)
 #endif
 
-#define increment_used_memory(__n) do { \
+#define update_zmalloc_stat_alloc(__n,__size) do { \
     size_t _n = (__n); \
+    size_t _stat_slot = (__size < ZMALLOC_MAX_ALLOC_STAT) ? __size : ZMALLOC_MAX_ALLOC_STAT; \
     if (_n&(sizeof(long)-1)) _n += sizeof(long)-(_n&(sizeof(long)-1)); \
     if (zmalloc_thread_safe) { \
         pthread_mutex_lock(&used_memory_mutex);  \
         used_memory += _n; \
+        zmalloc_allocations[_stat_slot]++; \
         pthread_mutex_unlock(&used_memory_mutex); \
     } else { \
         used_memory += _n; \
+        zmalloc_allocations[_stat_slot]++; \
     } \
 } while(0)
 
-#define decrement_used_memory(__n) do { \
+#define update_zmalloc_stat_free(__n) do { \
     size_t _n = (__n); \
     if (_n&(sizeof(long)-1)) _n += sizeof(long)-(_n&(sizeof(long)-1)); \
     if (zmalloc_thread_safe) { \
@@ -79,6 +83,8 @@
 static size_t used_memory = 0;
 static int zmalloc_thread_safe = 0;
 pthread_mutex_t used_memory_mutex = PTHREAD_MUTEX_INITIALIZER;
+/* Note that malloc_allocations elements are initialized to zero by C */
+size_t zmalloc_allocations[ZMALLOC_MAX_ALLOC_STAT+1];
 
 static void zmalloc_oom(size_t size) {
     fprintf(stderr, "zmalloc: Out of memory trying to allocate %zu bytes\n",
@@ -92,11 +98,11 @@ void *zmalloc(size_t size) {
 
     if (!ptr) zmalloc_oom(size);
 #ifdef HAVE_MALLOC_SIZE
-    increment_used_memory(redis_malloc_size(ptr));
+    update_zmalloc_stat_alloc(redis_malloc_size(ptr),size);
     return ptr;
 #else
     *((size_t*)ptr) = size;
-    increment_used_memory(size+PREFIX_SIZE);
+    update_zmalloc_stat_alloc(size+PREFIX_SIZE,size);
     return (char*)ptr+PREFIX_SIZE;
 #endif
 }
@@ -106,11 +112,11 @@ void *zcalloc(size_t size) {
 
     if (!ptr) zmalloc_oom(size);
 #ifdef HAVE_MALLOC_SIZE
-    increment_used_memory(redis_malloc_size(ptr));
+    update_zmalloc_stat_alloc(redis_malloc_size(ptr),size);
     return ptr;
 #else
     *((size_t*)ptr) = size;
-    increment_used_memory(size+PREFIX_SIZE);
+    update_zmalloc_stat_alloc(size+PREFIX_SIZE,size);
     return (char*)ptr+PREFIX_SIZE;
 #endif
 }
@@ -128,8 +134,8 @@ void *zrealloc(void *ptr, size_t size) {
     newptr = realloc(ptr,size);
     if (!newptr) zmalloc_oom(size);
 
-    decrement_used_memory(oldsize);
-    increment_used_memory(redis_malloc_size(newptr));
+    update_zmalloc_stat_free(oldsize);
+    update_zmalloc_stat_alloc(redis_malloc_size(newptr),size);
     return newptr;
 #else
     realptr = (char*)ptr-PREFIX_SIZE;
@@ -138,8 +144,8 @@ void *zrealloc(void *ptr, size_t size) {
     if (!newptr) zmalloc_oom(size);
 
     *((size_t*)newptr) = size;
-    decrement_used_memory(oldsize);
-    increment_used_memory(size);
+    update_zmalloc_stat_free(oldsize);
+    update_zmalloc_stat_alloc(size,size);
     return (char*)newptr+PREFIX_SIZE;
 #endif
 }
@@ -152,12 +158,12 @@ void zfree(void *ptr) {
 
     if (ptr == NULL) return;
 #ifdef HAVE_MALLOC_SIZE
-    decrement_used_memory(redis_malloc_size(ptr));
+    update_zmalloc_stat_free(redis_malloc_size(ptr));
     free(ptr);
 #else
     realptr = (char*)ptr-PREFIX_SIZE;
     oldsize = *((size_t*)realptr);
-    decrement_used_memory(oldsize+PREFIX_SIZE);
+    update_zmalloc_stat_free(oldsize+PREFIX_SIZE);
     free(realptr);
 #endif
 }
@@ -179,6 +185,11 @@ size_t zmalloc_used_memory(void) {
     return um;
 }
 
+size_t zmalloc_allocations_for_size(size_t size) {
+    if (size > ZMALLOC_MAX_ALLOC_STAT) return 0;
+    return zmalloc_allocations[size];
+}
+
 void zmalloc_enable_thread_safeness(void) {
     zmalloc_thread_safe = 1;
 }

src/zmalloc.h

@@ -40,5 +40,8 @@ size_t zmalloc_used_memory(void);
 void zmalloc_enable_thread_safeness(void);
 float zmalloc_get_fragmentation_ratio(void);
 size_t zmalloc_get_rss(void);
+size_t zmalloc_allocations_for_size(size_t size);
+
+#define ZMALLOC_MAX_ALLOC_STAT 256
 
 #endif /* _ZMALLOC_H */

utils/generate-command-help.rb

@@ -42,6 +42,7 @@ end
 def commands
   return @commands if @commands
 
+  require "rubygems"
   require "net/http"
   require "net/https"
   require "json"