version bumped to 2.0.4, release note for this version added

Replaying an AOF with blocking pop commands would lead to a state where
the fake AOF client would register itself (multiple times) as blocking
for a series of keys. Subsequent pushes against these keys could result
in a crash. Reported by and traced with the help from Jamie Turner.

.gitignore

@@ -1,4 +1,3 @@
-00-RELEASENOTES
 *.o
 *.rdb
 *.log

00-RELEASENOTES

@@ -0,0 +1,186 @@
+Welcome to Redis 2.0.4
+
+This release fixes two non critical bugs:
+
+    - HMGET used to crash when called against a key that was not holding
+      an hash. Fixed by Pieter Noordhuis.
+    
+    - Redis will now not try to save the DB if no save points for RDB are
+      configured, when used as a non persistent cache. Thanks to Robey
+      Pointer from Twitter for providing this patch.
+
+Waiting 2.2 ...
+Salvatore
+
+--------------------------------------------------------------------------------
+
+Welcome to Redis 2.0.3
+
+This release fixes two important bugs:
+
+    - The maxmemory directive was broken in Redis <= 2.0.2, since from time to
+      time it replayed with an error about memory limit reached even when
+      it was possible to expire some volatile key to make room for new data.
+
+      The new behaviour is the correct one of always allowing write operations
+      to succeed as long as there are other volatile keys to remove.
+
+    - An AOF bug related to blocking POP could crash Redis on AOF reload.
+      This is now fixed thanks to Pieter Noordhuis and a kind user that
+      helped us on IRC.
+
+Enjoy!
+Salvatore
+
+--------------------------------------------------------------------------------
+
+Welcome to Redis 2.0.2
+
+This is a bugfix release, with the followign changes:
+
+    - Fixed a bug that may slow down significantly (from a few milliseconds
+      to many seconds) server side intersections when a background write is
+      in progress. This was due to the hash table resize policy, prevented when
+      there was a saving child. Now it's prevented only up to 5 times
+      overbooking, so we try hard to prevent a lot of copy on write, but
+      avoiding to trigger pathological hash table performances of O(N) instead
+      of O(1).
+    - Fixed expired keys counter in INFO output. It was not counting keys
+      force-expired due to max-memory limit reached.
+
+Ciao,
+Salvatore
+
+--------------------------------------------------------------------------------
+
+Welcome to Redis 2.0.1
+
+This is a bugfix release, with the following changes:
+
+    - Fixed a not critical but important VM bug: from time to time a race
+      condition may happen and a client may never get the reply from a given
+      request.
+    - Now we have a make install target as stated in the README
+    - redis-cli no longer tries to understand if it's part of a pipe
+      so there is to use the -x option to read the last arg from stdin.
+      This prevents it from creating problem when running inside cron scripts.
+    - Fixed the init script provided in the tar.gz. It was totally broken.
+    - Fixed a bug related to connecting more than 10k clients to Redis.
+
+Good hacking,
+Salvatore
+
+--------------------------------------------------------------------------------
+
+Welcome to Redis 2.0.0
+
+Finally we reached 2.0.0 stable!
+
+This is just RC4 with (many) bug fixes applied.
+Also this release features an updated redis-cli, backported from master.
+
+Many thanks to VMware, Pieter Noordhuis and the Redis community for making
+this possible.
+
+For a list of new / improved things in 2.0.0 please check this web page:
+
+    http://code.google.com/p/redis/wiki/Redis_2_0_0_Changelog
+
+Enjoy!
+Salvatore
+
+--------------------------------------------------------------------------------
+
+Welcome to Redis 1.3.17 (2.0.0 RC4)
+
+Compared to RC3 this one contains only an important replication related bugfix
+about calls to ZINTERSTORE. Replication users are strongly encouraged to
+upgrade ASAP!
+
+If no new bugs will be found in a few weeks this is likely to become
+2.0.0 stable in short time (one or two weeks).
+
+Regards,
+Salvatore
+
+--------------------------------------------------------------------------------
+
+Welcome to Redis 1.3.16 (2.0.0 RC3)
+
+This is the third Release Candidate of Redis 2.0.0, fixing a few important
+Virtual Memory bugs. If you use VM it is a very good idea to upgrade ASAP.
+
+List of chages:
+
+2010-07-22 don't open/close log file if log level is not matched (antirez)
+2010-07-22 recent VM fixes backported to 2.0.0 branch (antirez)
+2010-07-22 fix rare condition where 'key' would already be destroyed while is was needed later on (Pieter Noordhuis)
+2010-07-21 vm_blocked_clients count fixed in INFO, thanks to Pietern Noordhuis (antirez)
+
+--------------------------------------------------------------------------------
+
+Welcome to Redis 1.3.15 (2.0.0 RC2)
+
+This is the second Release Candidate of Redis 2.0.0.
+
+List of changes:
+
+* Faster DB and AOF loading
+* Fixed swappability score computation
+* New version of the test suite
+* Fixed ZINCR Nan bugs leading to server crash and added tests
+* Require the flag "-c" to be used for redis-cli to read the last argument
+
+--------------------------------------------------------------------------------
+
+Welcome to Redis 1.3.14 (2.0.0 RC1)
+
+This is the first Release Candidate of Redis 2.0.0
+2.0.0 is a big release with many interesting news, the following is the
+list of the major improvements.
+
+WHAT'S NEW IN REDIS 2.0.x
+
+* Redis transactions: MULTI/EXEC (antirez)
+* Blocking POP: BLPOP and BRPOP (antirez)
+* Virtual Memory (antirez with contributions from Pieter Noordhuis)
+* APPEND command (antirez)
+* WITHSCORES in ZRANGEBYSCORE thanks to Sam Hendley (antirez)
+* Much faster .rdb loading/saving times (antirez)
+* Add DISCARD command to discard queued MULTI commands. (djanowski, antirez)
+* Interactive mode to redis-cli. (Michel Martens & Damian Janowski)
+* Countless improvements to skip list implementation (Pieter Noordhuis)
+* ZINTERSTORE, ZUNIONSTORE commands (Pieter Noordhuis)
+* ZRANK command (Pieter Noordhuis)
+* SUBSTR command (antirez)
+* Now list push commands return the length of the new list (Gustavo Picon, antirez)
+* ZREMBYRANK command (Pieter Noordhuis)
+* Hash data type with all the associated commands (antirez & Pieter Noordhuis)
+* Added ZREVRANK (Pieter Noordhuis)
+* Fixed hard to track replication bug (Jeremy Zawodny)
+* Tool to check rdb files for unprocessable opcodes (Pieter Noordhuis)
+* Random internals refactoring / cleanups (antirez & Pieter Noordhuis)
+* Fixed issues 193, 194, 191 (Antonio Ognio)
+* Support for include directive in config parser (Jeremy Zawodny)
+* Fix authentication for redis-cli on non-interactive mode. (Michel Martens)
+* Added linenoise.c for zero deps line editing support. (antirez)
+* Use linenoise for line editing on redis-cli. (Michel Martens)
+* More incremental expiration of keys. (antirez)
+* CONFIG command (antirez)
+* Publis/Subscribe and associated commands (antirez)
+* New form of shared object, sharing all the small integers (antirez)
+* Validate numeric inputs. (Alex McHale)
+* Non blocking hash table resize, thanks to Derek Collison and Pieter Noordhuis for design ideas and help. (antirez)
+* Enable hash dereference in SORT on BY and GET (Pieter Noordhuis)
+* Now all commands support binary safe keys (antirez)
+* SETEX (Set + EXPIRE) implemented (antirez & Pieter Noordhuis)
+* redis-cli now accepts and outputs quoted strings in interactive mode (antirez)
+* New MONITOR output format with timestamp, 1 command per line. (antirez)
+* Tool to check if AOF is valid, and fix it if needed (Pieter Noordhuis)
+* Major speedup to AOF with fsync always, thanks to Derek Collison for the algorithm (antirez)
+* Append only filename configurable (Pieter Noordhuis)
+* Add SIGTERM shutdown handling. (Ashley Martens)
+* Many improvements in Redis test suite (Pieter Noordhuis)
+* Add git commit information to INFO listing (Pieter Noordhuis & antirez)
+* Many more minor things mainly from me and Pieter Noordhuis, but this changelog is already too big ;)
+

Makefile

@@ -15,6 +15,10 @@ endif
 CCOPT= $(CFLAGS) $(CCLINK) $(ARCH) $(PROF)
 DEBUG?= -g -rdynamic -ggdb 
 
+INSTALL_TOP= /usr/local
+INSTALL_BIN= $(INSTALL_TOP)/bin
+INSTALL= cp -p
+
 OBJ = adlist.o ae.o anet.o dict.o redis.o sds.o zmalloc.o lzf_c.o lzf_d.o pqsort.o zipmap.o sha1.o
 BENCHOBJ = ae.o anet.o redis-benchmark.o sds.o adlist.o zmalloc.o
 CLIOBJ = anet.o sds.o adlist.o redis-cli.o zmalloc.o linenoise.o
@@ -113,3 +117,10 @@ noopt:
 
 32bitgprof:
 	make PROF="-pg" ARCH="-arch i386"
+
+install: all
+	$(INSTALL) $(PRGNAME) $(INSTALL_BIN)
+	$(INSTALL) $(BENCHPRGNAME) $(INSTALL_BIN)
+	$(INSTALL) $(CLIPRGNAME) $(INSTALL_BIN)
+	$(INSTALL) $(CHECKDUMPPRGNAME) $(INSTALL_BIN)
+	$(INSTALL) $(CHECKAOFPRGNAME) $(INSTALL_BIN)

dict.c

@@ -49,8 +49,13 @@
 /* Using dictEnableResize() / dictDisableResize() we make possible to
  * enable/disable resizing of the hash table as needed. This is very important
  * for Redis, as we use copy-on-write and don't want to move too much memory
- * around when there is a child performing saving operations. */
+ * around when there is a child performing saving operations.
+ *
+ * Note that even when dict_can_resize is set to 0, not all resizes are
+ * prevented: an hash table is still allowed to grow if the ratio between
+ * the number of elements and the buckets > dict_force_resize_ratio. */
 static int dict_can_resize = 1;
+static unsigned int dict_force_resize_ratio = 5;
 
 /* ---------------------------- Utility funcitons --------------------------- */
 
@@ -522,14 +527,23 @@ dictEntry *dictGetRandomKey(dict *d)
 /* Expand the hash table if needed */
 static int _dictExpandIfNeeded(dict *d)
 {
-    /* If the hash table is empty expand it to the intial size,
-     * if the table is "full" dobule its size. */
+    /* Incremental rehashing already in progress. Return. */
     if (dictIsRehashing(d)) return DICT_OK;
-    if (d->ht[0].size == 0)
-        return dictExpand(d, DICT_HT_INITIAL_SIZE);
-    if (d->ht[0].used >= d->ht[0].size && dict_can_resize)
+
+    /* If the hash table is empty expand it to the intial size. */
+    if (d->ht[0].size == 0) return dictExpand(d, DICT_HT_INITIAL_SIZE);
+
+    /* If we reached the 1:1 ratio, and we are allowed to resize the hash
+     * table (global setting) or we should avoid it but the ratio between
+     * elements/buckets is over the "safe" threshold, we resize doubling
+     * the number of buckets. */
+    if (d->ht[0].used >= d->ht[0].size &&
+        (dict_can_resize ||
+         d->ht[0].used/d->ht[0].size > dict_force_resize_ratio))
+    {
         return dictExpand(d, ((d->ht[0].size > d->ht[0].used) ?
                                     d->ht[0].size : d->ht[0].used)*2);
+    }
     return DICT_OK;
 }
 

redis-cli.c

@@ -61,6 +61,7 @@ static struct config {
     int pubsub_mode;
     int raw_output; /* output mode per command */
     int tty; /* flag for default output format */
+    int stdinarg; /* get last arg from stdin. (-x option) */
     char mb_sep;
     char *auth;
     char *historyfile;
@@ -336,6 +337,8 @@ static int parseOptions(int argc, char **argv) {
             i++;
         } else if (!strcmp(argv[i],"-h") && lastarg) {
             usage();
+        } else if (!strcmp(argv[i],"-x")) {
+            config.stdinarg = 1;
         } else if (!strcmp(argv[i],"-p") && !lastarg) {
             config.hostport = atoi(argv[i+1]);
             i++;
@@ -388,9 +391,8 @@ static sds readArgFromStdin(void) {
 
 static void usage() {
     fprintf(stderr, "usage: redis-cli [-iv] [-h host] [-p port] [-a authpw] [-r repeat_times] [-n db_num] cmd arg1 arg2 arg3 ... argN\n");
-    fprintf(stderr, "usage: echo \"argN\" | redis-cli [-h host] [-p port] [-a authpw] [-r repeat_times] [-n db_num] cmd arg1 arg2 ... arg(N-1)\n");
-    fprintf(stderr, "\nIf a pipe from standard input is detected this data is used as last argument.\n\n");
-    fprintf(stderr, "example: cat /etc/passwd | redis-cli set my_passwd\n");
+    fprintf(stderr, "usage: echo \"argN\" | redis-cli -x [options] cmd arg1 arg2 ... arg(N-1)\n\n");
+    fprintf(stderr, "example: cat /etc/passwd | redis-cli -x set my_passwd\n");
     fprintf(stderr, "example: redis-cli get my_passwd\n");
     fprintf(stderr, "example: redis-cli -r 100 lpush mylist x\n");
     fprintf(stderr, "\nRun in interactive mode: redis-cli -i or just don't pass any command\n");
@@ -455,9 +457,7 @@ static void repl() {
 
 static int noninteractive(int argc, char **argv) {
     int retval = 0;
-    struct stat s;
-    fstat(fileno(stdin), &s);
-    if (S_ISFIFO(s.st_mode) || S_ISREG(s.st_mode)) { /* pipe, regular file */
+    if (config.stdinarg) {
         argv = zrealloc(argv, (argc+1)*sizeof(char*));
         argv[argc] = readArgFromStdin();
         retval = cliSendCommand(argc+1, argv, config.repeat);
@@ -480,6 +480,7 @@ int main(int argc, char **argv) {
     config.monitor_mode = 0;
     config.pubsub_mode = 0;
     config.raw_output = 0;
+    config.stdinarg = 0;
     config.auth = NULL;
     config.historyfile = NULL;
     config.tty = isatty(fileno(stdout)) || (getenv("FAKETTY") != NULL);

redis.c

@@ -27,7 +27,7 @@
  * POSSIBILITY OF SUCH DAMAGE.
  */
 
-#define REDIS_VERSION "2.0.0"
+#define REDIS_VERSION "2.0.4"
 
 #include "fmacros.h"
 #include "config.h"
@@ -1515,10 +1515,21 @@ static int serverCron(struct aeEventLoop *eventLoop, long long id, void *clientD
                 num = REDIS_EXPIRELOOKUPS_PER_CRON;
             while (num--) {
                 dictEntry *de;
+                robj *key;
                 time_t t;
 
                 if ((de = dictGetRandomKey(db->expires)) == NULL) break;
                 t = (time_t) dictGetEntryVal(de);
+                key = dictGetEntryKey(de);
+                /* Don't expire keys that are in the contest of I/O jobs.
+                 * Otherwise decrRefCount will kill the I/O thread and
+                 * clients waiting for this keys will wait forever.
+                 *
+                 * In general this change will not have any impact on the
+                 * performance of the expiring algorithm but it's much safer. */
+                if (server.vm_enabled &&
+                    (key->storage == REDIS_VM_SWAPPING ||
+                     key->storage == REDIS_VM_LOADING)) continue;
                 if (now > t) {
                     deleteKey(db,dictGetEntryKey(de));
                     expired++;
@@ -2285,9 +2296,6 @@ static void call(redisClient *c, struct redisCommand *cmd) {
 static int processCommand(redisClient *c) {
     struct redisCommand *cmd;
 
-    /* Free some memory if needed (maxmemory setting) */
-    if (server.maxmemory) freeMemoryIfNeeded();
-
     /* Handle the multi bulk command type. This is an alternative protocol
      * supported by Redis in order to receive commands that are composed of
      * multiple binary-safe "bulk" arguments. The latency of processing is
@@ -2419,7 +2427,12 @@ static int processCommand(redisClient *c) {
         return 1;
     }
 
-    /* Handle the maxmemory directive */
+    /* Handle the maxmemory directive.
+     *
+     * First we try to free some memory if possible (if there are volatile
+     * keys in the dataset). If there are not the only thing we can do
+     * is returning an error. */
+    if (server.maxmemory) freeMemoryIfNeeded();
     if (server.maxmemory && (cmd->flags & REDIS_CMD_DENYOOM) &&
         zmalloc_used_memory() > server.maxmemory)
     {
@@ -2717,6 +2730,14 @@ static redisClient *createClient(int fd) {
     anetNonBlock(NULL,fd);
     anetTcpNoDelay(NULL,fd);
     if (!c) return NULL;
+    if (aeCreateFileEvent(server.el,fd,AE_READABLE,
+        readQueryFromClient, c) == AE_ERR)
+    {
+        close(fd);
+        zfree(c);
+        return NULL;
+    }
+
     selectDb(c,0);
     c->fd = fd;
     c->querybuf = sdsempty();
@@ -2742,11 +2763,6 @@ static redisClient *createClient(int fd) {
     c->pubsub_patterns = listCreate();
     listSetFreeMethod(c->pubsub_patterns,decrRefCount);
     listSetMatchMethod(c->pubsub_patterns,listMatchObjects);
-    if (aeCreateFileEvent(server.el, c->fd, AE_READABLE,
-        readQueryFromClient, c) == AE_ERR) {
-        freeClient(c);
-        return NULL;
-    }
     listAddNodeTail(server.clients,c);
     initClientMultiState(c);
     return c;
@@ -3137,6 +3153,7 @@ static int deleteKey(redisDb *db, robj *key) {
      * it's count. This may happen when we get the object reference directly
      * from the hash table with dictRandomKey() or dict iterators */
     incrRefCount(key);
+    if (server.vm_enabled) handleClientsBlockedOnSwappedKey(db,key);
     if (dictSize(db->expires)) dictDelete(db->expires,key);
     retval = dictDelete(db->dict,key);
     decrRefCount(key);
@@ -4204,7 +4221,7 @@ static int prepareForShutdown() {
         /* Append only file: fsync() the AOF and exit */
         fsync(server.appendfd);
         if (server.vm_enabled) unlink(server.vm_swap_file);
-    } else {
+    } else if (server.saveparamslen > 0) {
         /* Snapshotting. Perform a SYNC SAVE and exit */
         if (rdbSave(server.dbfilename) == REDIS_OK) {
             if (server.daemonize)
@@ -4219,6 +4236,8 @@ static int prepareForShutdown() {
             redisLog(REDIS_WARNING,"Error trying to save the DB, can't exit");
             return REDIS_ERR;
         }
+    } else {
+        redisLog(REDIS_WARNING,"Not saving DB.");
     }
     redisLog(REDIS_WARNING,"Server exit now, bye bye...");
     return REDIS_OK;
@@ -6690,6 +6709,7 @@ static void hmgetCommand(redisClient *c) {
     o = lookupKeyRead(c->db,c->argv[1]);
     if (o != NULL && o->type != REDIS_HASH) {
         addReply(c,shared.wrongtypeerr);
+        return;
     }
 
     /* Note the check for o != NULL happens inside the loop. This is
@@ -7619,6 +7639,22 @@ static void blockForKeys(redisClient *c, robj **keys, int numkeys, time_t timeou
     list *l;
     int j;
 
+    /* Never block for keys when the AOF is being replayed.
+     *
+     * When a BPOP is issued against an expiring list, the list is expired
+     * by means of the delete-on-write semantic, which causes the BPOP
+     * command to be written to the AOF. Then, the BPOP ends up in a blocking
+     * state and waits for a PUSH on any given key from another client.
+     *
+     * On replay, the expiring list will also be expired (if it isn't already),
+     * and the fake AOF client will block for a push. When multiple BPOPs
+     * (issued by multiple clients) are written to the AOF, this can cause the
+     * same blocking code to be executed against the single fake AOF client,
+     * which in turn can place the client in the list(s) of blocking clients
+     * *multiple times*. This state should be prevented, so simply skip
+     * blocking for the fake AOF client. */
+    if (c->fd < 0) return;
+
     c->blockingkeys = zmalloc(sizeof(robj*)*numkeys);
     c->blockingkeysnum = numkeys;
     c->blockingto = timeout;
@@ -8212,6 +8248,7 @@ static void freeMemoryIfNeeded(void) {
                     }
                 }
                 deleteKey(server.db+j,minkey);
+                server.stat_expiredkeys++;
             }
         }
         if (!freed) return; /* nothing to free... */

tests/unit/type/hash.tcl

@@ -140,6 +140,11 @@ start_server {tags {"hash"}} {
         set _ $rv
     } {{{} {}} {{} {}} {{} {}}}
 
+    test {HMGET against wrong type} {
+        r set wrongtype somevalue
+        assert_error "*wrong*" {r hmget wrongtype field1 field2}
+    }
+
     test {HMGET - small hash} {
         set keys {}
         set vals {}

utils/redis_init_script

@@ -21,15 +21,14 @@ case "$1" in
         then
                 echo -n "$PIDFILE does not exist, process is not running\n"
         else
+		PID=$(cat $PIDFILE)
                 echo -n "Stopping ...\n"
-                echo -n "Sending SHUTDOWN\r\n" | nc localhost $REDISPORT &
-                PID=$(cat $PIDFILE)
+                echo -n "SHUTDOWN\r\n" | nc localhost $REDISPORT &
                 while [ -x /proc/${PIDFILE} ]
                 do
                     echo "Waiting for Redis to shutdown ..."
                     sleep 1
                 done
-                rm $PIDFILE
                 echo "Redis stopped"
         fi
         ;;