diff --git a/00-RELEASENOTES b/00-RELEASENOTES index dca07237..37c0e546 100644 --- a/00-RELEASENOTES +++ b/00-RELEASENOTES @@ -14,189 +14,303 @@ HIGH: There is a critical bug that may affect a subset of users. Upgrade! CRITICAL: There is a critical bug affecting MOST USERS. Upgrade ASAP. -------------------------------------------------------------------------------- ----[ Redis 2.6.0 ] +--[ Redis 2.8.9 ] Release date: 22 Apr 2014 -* [BUGFIX] Allow AUTH when server is in -BUSY state because of a slow script. -* [BUGFIX] MULTI/EXEC flow now makes sense when observed in MONITOR -* [BUGFIX] SCRIPT KILL now uses different error prefixes for different errors. -* [BUGFIX] Default memory limit for 32bit archs lowered from 3.5 to 3 GB. -* [BUGFIX] redis-check-dump is now compatible with RDB files generated by 2.6. -* [IMPROVED] New field in INFO: slave_read_only. +# UPGRADE URGENCY: LOW, only new features introduced, no bugs fixed. ----[ Redis 2.5.14 (2.6 Release Candidate 8) ] +* [NEW] The HyperLogLog data structure. You can read more about it + in this blog post. http://antirez.com/news/75 +* [NEW] The Sorted Set data type has now support for lexicographic range + queries, check the new commands ZRANGEBYLEX, ZLEXCOUNT and + ZREMRANGEBYLEX, which are documented at http://redis.io. -* [BUGFIX] Fixed compilation on FreeBSD. -* [IMPROVED] SRANDMEMBER that returns multiple random elements. -* [IMPROVED] Sentinel backported to 2.6. It will be taken in sync with 2.8. -* [IMPROVED] Helper function for scripting to return errors and status replies. -* [IMPROVED] SORT by nosort [ASC|DESC] uses sorted set elements ordering. -* [BUGFIX] Better resistence to system clock skew. -* [IMPROVED] Warn the user when the configured maxmemory seems odd. -* [BUGFIX] Hashing function is now murmurhash2 for security purposes. -* [IMPROVED] Install script no longer uses a template but redis.conf itself. +--[ Redis 2.8.8 ] Release date: 25 Mar 2014 ----[ Redis 2.5.13 (2.6 Release Candidate 7) ] +# UPGRADE URGENCY: HIGH for Redis, LOW for Sentinel. There is a potentially + critical bug fix causing data loss in Redis but it requires + a combination of disk full and the use of the + SHUTDOWN command. -UPGRADE URGENCY: HIGH +* [FIX] Fixed data loss when SHUTDOWN was used with a disk full condition. +* [FIX] Fixed a memory leak in the SORT syntax error processing. +* [FIX] When Sentinel down-after-milliseconds parameter is modified at runtime + now it gets propagated to all the slaves and sentinel instances + of the master. +* [FIX] `install_server.sh` script finally fixed. +* [FIX] Different fixes to maxclients handling. -* [BUGFIX] Theoretical bug in ziplist fixed. -* [BUGFIX] Better out of memory handling (Log produced in log file). -* [BUGFIX] Incrementally flush RDB file on slave side while performing the - first synchronization with the master. This makes Redis less - blocking in environments where disk I/O is slow. -* [BUGFIX] Don't crash with Lua's redis.call() without arguments. -* [BUGFIX] Don't crash after a big number of Lua calls on 32 bit systems - because of a failed assertion. -* [BUGFIX] Fix SORT behaviour when called from scripting. -* [BUGFIX] Adjust slave PING period accordingly to REDIS_HZ define. -* [BUGFIX] BITCOUNT: fix crash on overflowing arguments. -* [BUGFIX] Return an error when SELECT argument is not an integer. -* [BUGFIX] Blocking operations on lists were completely reimplemented for - correctness. Now blocking list ops and pushes originated from - Lua scripts will play well together and will be replicated - and transmitted to the AOF correctly. -* [IMPROVED] Send async PING before starting replication to avoid blocking if - master allows us to connect but it is actually not able to reply. -* [IMPROVED] Support slave-priority for Redis Sentinel. -* [IMPROVED] Hiredis library updated. +* [NEW] Sentinels are now able to send update messages in a peer-to-peer + fashion even if no Redis instances are available. Now the Sentinel + liveness property that the most updated configuration in a given + partition is propagated to all the Sentinels is extended to partitions + without reachable instances. +* [NEW] Sentinel safety properties are now ensured in a crash-recovery system + model since some state is persisted on disk before replying to other + nodes, and reloaded at startup. +* [NEW] Sentinel now uses CLIENT SETNAME so that it is easy to identify + Sentinels using CLIENT LIST among other clients. +* [NEW] Sentinel failure detection and reconnection code improved. +* [NEW] Use all 24 bits (instead of 22) for the Redis objects LRU field. + Note that the new LRU algorithm using eviction pools was not backported + from unstable for safery / code maturity concerns. +* [NEW] Majory speedup for the INFO command (it is now 6 times faster). +* [NEW] More Sentinel unit tests. +* [NEW] New command DEBUG ERROR returns the specified error. Example: + DEBUG ERROR "LOADING database". This is handy to write Redis client + libraries unit tests. +* [NEW] redis-cli now supports multi-line editing via updated linenoise lib. ----[ Redis 2.5.12 (2.6 Release Candidate 6) ] +Thanks to Matt Stancliff and Jan-Erik Rediger for the work done in the context +of this release. -UPGRADE URGENCY: MODERATE. +--[ Redis 2.8.7 ] Release date: 5 Mar 2014 -* [BUGFIX] Fixed a timing attack on AUTH (Issue #560). -* [BUGFIX] Don't assume that "char" is signed. -* [BUGFIX] Check that we have connection before enabling pipe mode. -* [BUGFIX] Use the optimized version of the function to convert a double to - its string representation. Compilation was disabled because of - a typo in the #if statement. -* [IMPROVED} REPLCONF internal command introduced, now INFO shows slaves with - correct port numbers. This makes 2.5.12 Redis Sentinel compatible. -* [IMPROVED] Truncate short write from the AOF for a cleaner restart. On short - writes (for instance out of space) Redis will now try to remove - the half-written data so that the next restart will work without - the need for the "redis-check-aof" utility. -* [IMPROVED] New in INFO: aof_last_bgrewrite_status -* [IMPROVED] Allow Pub/Sub in contexts where other commands are blocked. -* [BUGFIX] mark fd as writable when EPOLLERR or EPOLLHUP is returned by - epoll_wait. +# UPGRADE URGENCY: LOW for Redis, LOW for Sentinel. However this release adds + new features so users may want to upgrade in order to + exploit the new functionalities. ----[ Redis 2.5.11 (2.6 Release Candidate 5) ] +* [FIX] Sometimes the absolute config file path was obtained in a wrong way. + This happened when there was a "dir" directive inside the config file + and at the same time the configuration file was given as a relative + path to redis-server or redis-sentinel executables. +* [FIX] redis-cli: Automatically enter --slave mode when SYNC or PSYNC are + called during an interactive session. +* [FIX] Sentinel "IDONTKNOW" error removed as it does not made sense with the + new Sentinel design. This error was actually a fix for a design error + in the first implementation of Sentinel. +* [FIX] Sentinel: added a missing exit() call to abort after config file + checks at startup. This error was introduced with an improvement in + a previous 2.8 release. +* [FIX] BITCOUNT: fixed unaligned access causing issues in sparc and other + archs not capable of dealing with unaligned accesses. This also makes + the code faster in archs where unaligned accesses are allowed. +* [FIX] Sentinel: better nodes fail over start time desynchronization to avoid + split-brain during the voting process needed to get authorization to + fail over. This means the system is less likely to need to retry + and will fail over faster. No changes in behavior / correctness. +* [FIX] Force INFO used_memory_peak to match peak memory. This generated some + confusion among users even if it was not an actual bug. -UPGRADE URGENCY: HIGH. +* [NEW] Sentinel unit tests and framework. More tests needed and units must + be improved in order to have less false positives, but it is a start + and features a debugging console that is useful to fix tests or to + inspect bugs causing tests failures. +* [NEW] New Sentinel events: +/-monitor and +set used to monitor when an + instance to monitor is added or removed, or when a configuration + is modified via SENTINEL SET. +* [NEW] Redis-cli updated to use SCAN instead of random sampling via + RANDOMKEY in order to implement --bigkeys feature. Moreover the + implementation now supports pipelining and reports more information + at the end of the scan. Much faster, much better. A special thank + you to Michael Grunder for this improvement. +* [NEW] redis-cli now supports a new --intrinsic-latency mode that is able + to meter the latency of a system due to kernel / hypervisor. + How to use it is explained at http://redis.io/topics/latency. +* [NEW] New command BITPOS: find first bit set or clear in a bitmap. +* [NEW] CONFIG REWRITE calls are now logged. -* [BUGFIX] Fixed Hash corruption when loading an RDB file generated by - previous versions of Redis that encoded hashes using - a different ziplist encoding format for small integers. - All the fileds that are integers in the range 0-255 may not - be recognized, or duplicated un updates, causing a crash - when the ziplist is converted to a real hash. (Issue #547). -* [BUGFIX] Fixed the count of memory used by output buffers in the - setDeferredMultiBulkLength() function. +--[ Redis 2.8.6 ] Release date: 13 Feb 2014 ----[ Redis 2.5.10 (2.6 Release Candidate 4) ] +# UPGRADE URGENCY: HIGH for Redis, LOW for Sentinel. Redis users using Lua + scripts with EVALSHA and attached slaves and/or AOF + persistence should consider upgrading ASAP. -UPGRADE URGENCY: HIGH. +* [FIX] Fixed an critical EVALSHA script cache bug: scripts executed may not + propagate to AOF / Slaves correctly under certain conditions. + See issue #1549 at Github for more information. +* [FIX] Fixed multiple bugs resulting into closing the link with master or slave + during replication without good reasons. This will result in useless + resynchronizations, or infinite loops where the replication link can't + be established. +* [FIX] Don't count the time needed to populate the buffers of clients waiting + in MONITOR mode when populating the Slow Log entries. -* [BUGFIX] Allow PREFIX to be overwritten on "make install". -* [BUGFIX] Run the test with just one client if the computer is slow. -* [BUGFIX] Event port support in our event driven libray. -* [BUGFIX] Jemalloc updated to 3.0.0. This fixes a possibly AOF rewrite issue. - See https://github.com/antirez/redis/issues/504 for info. -* [BUGFIX] Fixed issue #516: ZINTERSTORE / ZUNIONSTORE with mixed sets/zsets. -* [BUGFIX] Set fd to writable when poll(2) detects POLLERR or POLLHUP event. -* [BUGFIX] Fixed RESTORE hash failure (Issue #532). -* [IMPROVED] Allow an AOF rewrite buffer > 2GB (Related to issue #504). -* [IMPROVED] Server cron function frequency is now configurable (REDIS_HZ). -* [IMPROVED] Better, less blocking expired keys collection algorithm. -* [FEATURE] New commands: BITOP and BITCOUNT. -* [FEATURE] redis-cli --pipe for mass import. +* [NEW] AOF write errors (like no space on device) no longer abort Redis if the + fsync policy is none or every second. The database enters a read-only + mode where every write is refused with an error. Normal operations are + restored as soon as Redis is able to append again data to the AOF file. +* [NEW] Sentinel now accepts SHUTDOWN command. -What's new in Redis 2.5.9 (aka 2.6 Release Candidate 3) -======================================================= +--[ Redis 2.8.5 ] Release date: 4 Feb 2014 -UPGRADE URGENCY: critical, upgrade ASAP. +# UPGRADE URGENCY: HIGH for Redis, LOW for Sentinel. Redis users using Lua + scripts with expires, and Redis users relying on the + ability of Redis to block writes on RDB saving errors + should plan to upgrade ASAP. -* [BUGFIX] Fix for issue #500 (https://github.com/antirez/redis/pull/500). - Redis 2.6-RC1 and RC2 may corrupt ziplist-encoded sorted sets - produced by Redis 2.4.x. -* [BUGFIX] Fixed several bugs in init.d script. -* [BUGFIX] syncio.c functions modified for speed and correctness. On osx - (and possibly other BSD-based systems) the slave would block on - replication to send the SYNC command when the master was not - available. This is fixed now, but was not affecting Linux installs. -* Now when slave-serve-stale-data is set to yes and the master is down, instead - of reporting a generic error Redis replies with -MASTERDOWN. +* [FIX] Fixed a replication bug caused by Lua scripts + expired keys: keys could + expire in the middle of scripts causing non-deterministic behavior. +* [FIX] MISCONFIG error if condition fixed, the server was no longer able + to stop writes on RDB misconfiguration after this error was introduced. +* [FIX] REDIS_AOF_REWRITE_MIN_SIZE is now 64mb like example redis.conf default. +* [FIX] Perform fflush() before fsync() in rio.c (bug without actual effects). +* [FIX] Don't log MONITOR clients as disconnecting slaves. +* [FIX] SENTINEL MASTER arity check fixed. Crashed the Sentinel instance when + the command was given without arguments. -What's new in Redis 2.5.8 (aka 2.6 Release Candidate 2) -======================================================= +* [NEW] Allow CONFIG and SHUTDOWN while in stale-slave state. +* [NEW] Support for configurable TCP listen(2) backlog size. +* [NEW] redis-cli supports SCAN via the --scan and --pattern options. +* [NEW] SENTINEL SET master quorum via runtime API implemented. -UPGRADE URGENCY: high for all the users of the KEYS command, otherwise low. +--[ Redis 2.8.4 ] Release date: 13 Jan 2014 -* [BUGFIX] Fix for KEYS command: if the DB contains keys with expires the KEYS - command may return the wrong output, having duplicated or missing - keys. See issue #487 and #488 on github for details. +# UPGRADE URGENCY: MODERATE for Redis and Sentinel. -What's new in Redis 2.5.7 (aka 2.6 Release Candidate 1) -======================================================= +* [FIX] Makefile compatibility with non common make variants improved. +* [FIX] SDIFF crash in very unlikely to trigger state fixed. +* [FIX] Config rewriting fixed: don't wipe options unknown to the rewrite + process. +* [FIX] Set TCP port to 0 works again to disable TCP networking. +* [FIX] Fixed replication with old Redis instances as masters by not + sending REPLCONF ACK to them. +* [FIX] Fix keyspace notifications rewrite and CONFIG GET output. +* [FIX] Fix RESTORE TTL handling in 32 bit systems (32 bit overflow). -UPGRADE URGENCY: upgrade not recommended because this is an RC release. +* [NEW] Sentinel now has a run time configuration API. +* [NEW] Log when we lost connection with master or slave. +* [NEW] When instance is turned from slave to master now inherits the + old master replication offset when possible. This improves the + Sentinel failover procedure. -* This is the first release candidate for Redis 2.6. We are not aware of - bugs, but part of this code is young and was never tested in production - environments, so handle with care. +--[ Redis 2.8.3 ] Release date: 11 Dec 2013 -An overview of new features and changes in Redis 2.6.x -====================================================== +# UPGRADE URGENCY: MODERATE for Redis, HIGH for Sentinel. -* Server side Lua scripting, see http://redis.io/commands/eval -* Virtual Memory removed (was deprecated in 2.4) -* Hardcoded limits about max number of clients removed. -* AOF low level semantics is generally more sane, and especially when used - in slaves. -* Milliseconds resolution expires, also added new commands with milliseconds - precision (PEXPIRE, PTTL, ...). -* Better memory usage for "small" lists, ziplists and hashes when fields or - values contain small integers. -* Read only slaves. -* New bit opeations: BITCOUNT and BITOP commands. -* Clients max output buffer soft and hard limits. You can specifiy different - limits for different classes of clients (normal,pubsub,slave). -* More incremental (less blocking) expired keys collection algorithm, in - practical terms this means that Redis is more responsive when a very - big number of keys expire about at the same time. -* AOF is now able to rewrite aggregate data types using variadic commands, - often producing an AOF that is faster to save, load, and is smaller in size. -* Every redis.conf directive is now accepted as a command line option for the - redis-server binary, with the same name and number of arguments. -* Hash table seed randomization for protection against collisions attacks. -* Performances improved when writing large objects to Redis. -* Integrated memory test, see redis-server --test-memory. -* INCRBYFLOAT and HINCRBYFLOAT commands. -* New DUMP, RESTORE, MIGRATE commands (back ported from Redis Cluster to 2.6). -* CRC64 checksump in RDB files. -* Better MONITOR output and behavior (now commands are logged before execution). -* "Software Watchdog" feature to debug latency issues. -* Significant parts of the core refactored or rewritten. New internal APIs - and core changes allowed to develop Redis Cluster on top of the new code, - however for 2.6 all the cluster code was removed, and will be released with - Redis 3.0 when it is more complete and stable. -* Redis ASCII art logo added at startup. -* Crash report on memory violation or failed asserts improved significantly - to make debugging of hard to catch bugs simpler. -* redis-benchmark improvements: ability to run selected tests, - CSV output, faster, better help. -* redis-cli improvements: --eval for comfortable development of Lua scripts. -* SHUTDOWN now supports two optional arguments: "SAVE" and "NOSAVE". -* INFO output split into sections, the command is now able to just show - pecific sections. -* New statistics about how many time a command was called, and how much - execution time it used (INFO commandstats). -* More predictable SORT behavior in edge cases. -* Better support for big endian and *BSD systems. -* Build system improved. +* [FIX] Sentinel instance role sampling fixed, the system is now more + reliable during failover and when reconfiguring instances with + non matching configuration. +* [FIX] Inline requests are now handled even when terminated with just LF. +* [FIX] Replication timeout handling greatly improved, now the slave is able + to ping the master while removing the old data from memory, and while + loading the new RDB file. This avoid false timeouts sensed by + masters. +* [FIX] Fixed a replication bug involving 32 bit instances and big datasets + hard to compress that resulted into more than 2GB of RDB file sent. +* [FIX] Return error for inline requests with unbalanced quotes. +* [FIX] Publish the slave replication offset even when disconnected from the + master if there is still a cached master instance. -Migrating from 2.4 to 2.6 +--[ Redis 2.8.2 ] Release date: 2 Dec 2013 + +# UPGRADE URGENCY: MODERATE for both Redis and Sentinel. + +* [FIX] Sentinel better desynchronization to avoid split-brain elections + where no Sentinel managed to get elected. +* [FIX] Stop accepting writes on "MISCONF" error only if master, not slave. +* [FIX] Reply to PING with an error on "MISCONF" errors. + +--[ Redis 2.8.1 ] Release date: 25 Nov 2013 + +# UPGRADE URGENCY: LOW for Redis, CRITICAL for Senitnel. You don't need to + upgrade your Redis instances but it is highly recommended + to upgrade and restart all the Sentinel processes. + +* [FIX] Fixed a bug in "new Sentinel" config propagation. +* [FIX] Fixed a false positive in Redis tests. + +--[ Redis 2.8.0 ] Release date: 22 Nov 2013 + +# UPGRADE URGENCY: LOW, unless you want to upgrade to new Sentinel code. + +* [FIX] Fixed an error in rdbWriteRaw() that should have no practical impact. +* [NEW] Log the new master when SLAVEOF command is used. +* [NEW] Sentinel code synchronized with the unstable branch, the new Sentinel + is a reimplementation that uses more reliable algorithms. + +--[ Redis 2.8 Release Candidate 6 (2.7.106) ] Release date: 6 Nov 2013 + +This is the 6th release candidate of Redis 2.8 (official version is 2.7.106). + +# UPGRADE URGENCY: LOW, only new features back ported, no fixes. + +* [NEW] SCAN, SSCAN, HSCAN, ZSCAN commands. + +--[ Redis 2.8 Release Candidate 5 (2.7.105) ] Release date: 9 Oct 2013 + +This is the 5th release candidate of Redis 2.8 (official version is 2.7.105). +Important bugs fixed inside. + +# UPGRADE URGENCY: HIGH because of many non critical replication bugs fixed. + +* [FIX] redis-cli: don't crash with --bigkeys when the key no longer exist. +* [FIX] Allow AUTH / PING when disconnected from slave and serve-stale-data is no. +* [FIX] PSYNC: safer handling of PSYNC requests with offsets in the future. +* [FIX] Replication: Fix master timeout detection. +* [FIX] Replication: Correctly install the write handler after successful PSYNC. + +--[ Redis 2.8 Release Candidate 4 (2.7.104) ] Release date: 30 Aug 2013 + +This is the fourth release candidate of Redis 2.8 (official version is 2.7.104). +Important bugs fixed inside. + +# UPGRADE URGENCY: HIGH because of the EVAL memory leak. + +* [FIX] Fixed a serious EVAL memory leak in the Lua stack. +* [FIX] Fixed server startup when no IPv6 address exists in any interface. +* [FIX] Send MISCONFIG error when BGSAVE fails because can't fork. +* [FIX] Memory efficiency with large (> a few kbytes) values improved considerably. +* [NEW] DEBUG SDSLEN for sds memory debugging. + +--[ Redis 2.8 Release Candidate 3 (2.7.103) ] Release date: 19 Aug 2013 + +This is the third release candidate of Redis 2.8 (official version is 2.7.103). +Important bugs fixed inside. + +# UPGRADE URGENCY: HIGH + +* [FIX] Improved expired keys collection algorithm. Even under heavy load keys + to be expired can't accumulate because of lack of CPU time. +* [FIX] Replication speed regression fixed (issue #1238). +* [FIX] Fixed an hard to trigger PSYNC bug. +* [FIX] Fixed Issue #1240, ZUNIONSTORE could lead to wrong result. +* [NEW] Add per-db average TTL information in INFO output. +* [NEW] redis-benchmark improvements. +* [NEW] dict.c API wrong usage detection. + +--[ Redis 2.8 Release Candidate 2 (2.7.102) ] Release date: 30 Jul 2013 + +This is the second release candidate of Redis 2.8 (official version is 2.7.102). +Important bugs fixed inside. + +# UPGRADE URGENCY: HIGH + +* [FIX] Fixed a critical replication bug, see issue #1221. +* [NEW] The new inline protocol now accepts quoted strings like, for example + you can now type in a telnet session: set 'foo bar' "hello world\n". + +--[ Redis 2.8 Release Candidate 1 (2.7.101) ] Release date: 18 Jul 2013 + +This is the first release candidate of Redis 2.8 (official version is 2.7.101). + +The following is a list of improvements in Redis 2.8, compared to Redis 2.6. + +* [NEW] Slaves are now able to partially resynchronize with the master, so most + of the times a full resynchronization with the RDB creation in the master + side is not needed when the master-slave link is disconnected for a short + amount of time. +* [NEW] Experimental IPv6 support. +* [NEW] Slaves explicitly ping masters now, a master is able to detect a timed out + slave independently. +* [NEW] Masters can stop accepting writes if not enough slaves with a given + maximum latency are connected. +* [NEW] Keyspace changes notifications via Pub/Sub. +* [NEW] CONFIG SET maxclients is now available. +* [NEW] Ability to bind multiple IP addresses. +* [NEW] Set process names so that you can recognize, in the "ps" command output, + the listening port of an instance, or if it is a saving child. +* [NEW] Automatic memory check on crash. +* [NEW] CONFIG REWRITE is able to materialize the changes in the configuration + operated using CONFIG SET into the redis.conf file. +* [NEW] More NetBSD friendly code base. +* [NEW] PUBSUB command for Pub/Sub introspection capabilities. +* [NEW] EVALSHA can now be replicated as such, without requiring to be expanded + to a full EVAL for the replication link. +* [NEW] Better Lua scripts error reporting. +* [NEW] SDIFF performance improved. +* [FIX] A number of bugfixes. + +Migrating from 2.6 to 2.8 ========================= Redis 2.6 is mostly a strict subset of 2.8. However there are a few things @@ -204,6 +318,9 @@ that you should be aware of: The following commands changed behavior: + * The TTL and PTTL commands now return -2 if the key does not exist and + -1 if it exists but has no associated expire. Redis 2.6 and previous + versions used to return -1 for both the conditions. * SORT with ALPHA now sorts according to local collation locale if no STORE option is used. * ZADD/ZINCRBY are now able to accept a bigger range of values as valid diff --git a/COPYING b/COPYING index c8665ba6..a58de44d 100644 --- a/COPYING +++ b/COPYING @@ -1,4 +1,4 @@ -Copyright (c) 2006-2012, Salvatore Sanfilippo +Copyright (c) 2006-2014, Salvatore Sanfilippo All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: diff --git a/README b/README index 329eb1cb..b7a12b82 100644 --- a/README +++ b/README @@ -130,7 +130,7 @@ it the proper way for a production system, we have a script doing this for Ubuntu and Debian systems: % cd utils - % ./install_server + % ./install_server.sh The script will ask you a few questions and will setup everything you need to run Redis properly as a background daemon that will start again on diff --git a/README.md b/README.md index 89da3976..0490c28a 100644 --- a/README.md +++ b/README.md @@ -1,8 +1,8 @@ -Redis on Windows 2.8.4 +Redis on Windows 2.8.9 === ## What's new in this release -- This is a port for Windows based on Redis 2.8. The latest version merged in 2.8.4. +- This is a port for Windows based on Redis 2.8. The latest version merged in 2.8.9. - There is support for the x64 version. We have dropped support for the 32-bit version. - The binaries (unsigned) have been moved to a zip file in the \bin folder to make them easier to find. The Release build automatically updates the zip file. @@ -12,7 +12,7 @@ Redis on Windows 2.8.4 - We are moving towards moving all Windows-specific changes into the Win32_Interop library. ## Repo branches -- 2.8.4.msopen: This is the branch for the Windows Redis port based on Redis 2.8 +- 2.8.4_msopen: This is the branch for the Windows Redis port based on Redis 2.8 - 2.6: This is the branch for the Windows Redis port based on Redis 2.6. - 2.4: This branch has the Windows Redis port based on Redis 2.4. diff --git a/bin/release/redis-2.8.4.zip b/bin/release/redis-2.8.4.zip index deb8a1e4..d9baa709 100644 Binary files a/bin/release/redis-2.8.4.zip and b/bin/release/redis-2.8.4.zip differ diff --git a/deps/linenoise/linenoise.c b/deps/linenoise/linenoise.c index 393862a7..2b0df430 100644 --- a/deps/linenoise/linenoise.c +++ b/deps/linenoise/linenoise.c @@ -10,8 +10,8 @@ * * ------------------------------------------------------------------------ * - * Copyright (c) 2010, Salvatore Sanfilippo - * Copyright (c) 2010, Pieter Noordhuis + * Copyright (c) 2010-2013, Salvatore Sanfilippo + * Copyright (c) 2010-2013, Pieter Noordhuis * * All rights reserved. * @@ -45,12 +45,10 @@ * - http://www.3waylabs.com/nw/WWW/products/wizcon/vt220.html * * Todo list: - * - Switch to gets() if $TERM is something we can't support. * - Filter bogus Ctrl+ combinations. * - Win32 support * * Bloat: - * - Completion? * - History search like Ctrl+r in readline? * * List of escape sequences used by this program, we do everything just @@ -72,6 +70,17 @@ * Sequence: ESC [ n C * Effect: moves cursor forward of n chars * + * When multi line mode is enabled, we also use an additional escape + * sequence. However multi line editing is disabled by default. + * + * CUU (Cursor Up) + * Sequence: ESC [ n A + * Effect: moves cursor up of n chars. + * + * CUD (Cursor Down) + * Sequence: ESC [ n B + * Effect: moves cursor down of n chars. + * * The following are used to clear the screen: ESC [ H ESC [ 2 J * This is actually composed of two sequences: * @@ -95,6 +104,7 @@ #include #include #include +#include #include #include "linenoise.h" #ifdef _WIN32 @@ -104,24 +114,66 @@ #define LINENOISE_DEFAULT_HISTORY_MAX_LEN 100 #define LINENOISE_MAX_LINE 4096 -static char *unsupported_term[] = {"dumb","cons25",NULL}; +static char *unsupported_term[] = {"dumb","cons25","emacs",NULL}; static linenoiseCompletionCallback *completionCallback = NULL; #ifndef _WIN32 -static struct termios orig_termios; /* in order to restore at exit */ +static struct termios orig_termios; /* In order to restore at exit.*/ #endif -static int rawmode = 0; /* for atexit() function to check if restore is needed*/ -static int atexit_registered = 0; /* register atexit just 1 time */ +static int rawmode = 0; /* For atexit() function to check if restore is needed*/ +static int mlmode = 0; /* Multi line mode. Default is single line. */ +static int atexit_registered = 0; /* Register atexit just 1 time. */ static int history_max_len = LINENOISE_DEFAULT_HISTORY_MAX_LEN; static int history_len = 0; -char **history = NULL; +static char **history = NULL; + +/* The linenoiseState structure represents the state during line editing. + * We pass this state to functions implementing specific editing + * functionalities. */ +struct linenoiseState { + int ifd; /* Terminal stdin file descriptor. */ + int ofd; /* Terminal stdout file descriptor. */ + char *buf; /* Edited line buffer. */ + size_t buflen; /* Edited line buffer size. */ + const char *prompt; /* Prompt to display. */ + size_t plen; /* Prompt length. */ + size_t pos; /* Current cursor position. */ + size_t oldpos; /* Previous refresh cursor position. */ + size_t len; /* Current edited line length. */ + size_t cols; /* Number of columns in terminal. */ + size_t maxrows; /* Maximum num of rows used so far (multiline mode) */ + int history_index; /* The history index we are currently editing. */ +}; + +enum KEY_ACTION{ + KEY_NULL = 0, /* NULL */ + CTRL_A = 1, /* Ctrl+a */ + CTRL_B = 2, /* Ctrl-b */ + CTRL_C = 3, /* Ctrl-c */ + CTRL_D = 4, /* Ctrl-d */ + CTRL_E = 5, /* Ctrl-e */ + CTRL_F = 6, /* Ctrl-f */ + CTRL_H = 8, /* Ctrl-h */ + TAB = 9, /* Tab */ + CTRL_K = 11, /* Ctrl+k */ + CTRL_L = 12, /* Ctrl+l */ + ENTER = 13, /* Enter */ + CTRL_N = 14, /* Ctrl-n */ + CTRL_P = 16, /* Ctrl-p */ + CTRL_T = 20, /* Ctrl-t */ + CTRL_U = 21, /* Ctrl+u */ + CTRL_W = 23, /* Ctrl+w */ + ESC = 27, /* Escape */ + BACKSPACE = 127 /* Backspace */ +}; static void linenoiseAtExit(void); int linenoiseHistoryAdd(const char *line); +static void refreshLine(struct linenoiseState *l); #ifdef _WIN32 #ifndef STDIN_FILENO - #define STDIN_FILENO (_fileno(stdin)) + #define STDIN_FILENO (_fileno(stdin)) #endif HANDLE hOut; @@ -144,7 +196,7 @@ static int win32read(char *c) { *c = b.Event.KeyEvent.uChar.AsciiChar; //if (e.dwControlKeyState & (LEFT_ALT_PRESSED | RIGHT_ALT_PRESSED)) { - /* Alt+key ignored */ + /* Alt+key ignored */ //} else if (e.dwControlKeyState & (LEFT_CTRL_PRESSED | RIGHT_CTRL_PRESSED)) { @@ -232,19 +284,36 @@ static int win32read(char *c) { return -1; /* Makes compiler happy */ } +#endif -#ifdef __STRICT_ANSI__ -char *strdup(const char *s) { - size_t l = strlen(s)+1; - char *p = malloc(l); +/* Debugging macro. */ +#if 0 +FILE *lndebug_fp = NULL; +#define lndebug(...) \ + do { \ + if (lndebug_fp == NULL) { \ + lndebug_fp = fopen("/tmp/lndebug.txt","a"); \ + fprintf(lndebug_fp, \ + "[%d %d %d] p: %d, rows: %d, rpos: %d, max: %d, oldmax: %d\n", \ + (int)l->len,(int)l->pos,(int)l->oldpos,plen,rows,rpos, \ + (int)l->maxrows,old_rows); \ + } \ + fprintf(lndebug_fp, ", " __VA_ARGS__); \ + fflush(lndebug_fp); \ + } while (0) +#else +#define lndebug(fmt, ...) +#endif - memcpy(p,s,l); - return p; +/* ======================= Low level terminal handling ====================== */ + +/* Set if to use or not the multi line mode. */ +void linenoiseSetMultiLine(int ml) { + mlmode = ml; } -#endif /* __STRICT_ANSI__ */ - -#endif /* _WIN32 */ +/* Return true if the terminal name is in the list of terminals we know are + * not able to understand basic escape sequences. */ static int isUnsupportedTerm(void) { #ifndef _WIN32 char *term = getenv("TERM"); @@ -257,16 +326,7 @@ static int isUnsupportedTerm(void) { return 0; } -static void freeHistory(void) { - if (history) { - int j; - - for (j = 0; j < history_len; j++) - free(history[j]); - free(history); - } -} - +/* Raw mode: 1960 magic shit. */ static int enableRawMode(int fd) { #ifndef _WIN32 struct termios raw; @@ -345,19 +405,34 @@ static void disableRawMode(int fd) { #endif } -/* At exit we'll try to fix the terminal to the initial conditions. */ -static void linenoiseAtExit(void) { -#ifdef _WIN32 - SetConsoleMode(hIn, consolemode); - CloseHandle(hOut); - CloseHandle(hIn); -#else - disableRawMode(STDIN_FILENO); -#endif - freeHistory(); +/* Use the ESC [6n escape sequence to query the horizontal cursor position + * and return it. On error -1 is returned, on success the position of the + * cursor. */ +static int getCursorPosition(int ifd, int ofd) { + char buf[32]; + int cols, rows; + unsigned int i = 0; + + /* Report cursor location */ + if (write(ofd, "\x1b[6n", 4) != 4) return -1; + + /* Read the response: ESC [ rows ; cols R */ + while (i < sizeof(buf)-1) { + if (read(ifd,buf+i,1) != 1) break; + if (buf[i] == 'R') break; + i++; + } + buf[i] = '\0'; + + /* Parse it. */ + if (buf[0] != ESC || buf[1] != '[') return -1; + if (sscanf(buf+2,"%d;%d",&rows,&cols) != 2) return -1; + return cols; } -static int getColumns(void) { +/* Try to get the number of columns in the current terminal, or assume 80 + * if it fails. */ +static int getColumns(int ifd, int ofd) { #ifdef _WIN32 CONSOLE_SCREEN_BUFFER_INFO b; @@ -366,70 +441,54 @@ static int getColumns(void) { #else struct winsize ws; - if (ioctl(1, TIOCGWINSZ, &ws) == -1) return 80; - return ws.ws_col; + if (ioctl(1, TIOCGWINSZ, &ws) == -1 || ws.ws_col == 0) { + /* ioctl() failed. Try to query the terminal itself. */ + int start, cols; + + /* Get the initial position so we can restore it later. */ + start = getCursorPosition(ifd,ofd); + if (start == -1) goto failed; + + /* Go to right margin and get position. */ + if (write(ofd,"\x1b[999C",6) != 6) goto failed; + cols = getCursorPosition(ifd,ofd); + if (cols == -1) goto failed; + + /* Restore position. */ + if (cols > start) { + char seq[32]; + snprintf(seq,32,"\x1b[%dD",cols-start); + if (write(ofd,seq,strlen(seq)) == -1) { + /* Can't recover... */ + } + } + return cols; + } else { + return ws.ws_col; + } + +failed: + return 80; #endif } -static void refreshLine(int fd, const char *prompt, char *buf, size_t len, size_t pos, size_t cols) { - char seq[64]; -#ifdef _WIN32 - DWORD pl, bl, w; - CONSOLE_SCREEN_BUFFER_INFO b; - COORD coord; -#endif - size_t plen = strlen(prompt); - - while((plen+pos) >= cols) { - buf++; - len--; - pos--; +/* Clear the screen. Used to handle ctrl+l */ +void linenoiseClearScreen(void) { + if (write(STDOUT_FILENO,"\x1b[H\x1b[2J",7) <= 0) { + /* nothing to do, just to avoid warning. */ } - while (plen+len > cols) { - len--; - } - -#ifndef _WIN32 - /* Cursor to left edge */ - snprintf(seq,64,"\x1b[0G"); - if (write(fd,seq,strlen(seq)) == -1) return; - /* Write the prompt and the current buffer content */ - if (write(fd,prompt,strlen(prompt)) == -1) return; - if (write(fd,buf,len) == -1) return; - /* Erase to right */ - snprintf(seq,64,"\x1b[0K"); - if (write(fd,seq,strlen(seq)) == -1) return; - /* Move cursor to original position. */ - snprintf(seq,64,"\x1b[0G\x1b[%dC", (int)(pos+plen)); - if (write(fd,seq,strlen(seq)) == -1) return; -#else - - REDIS_NOTUSED(seq); - REDIS_NOTUSED(fd); - - /* Get buffer console info */ - if (!GetConsoleScreenBufferInfo(hOut, &b)) return; - /* Erase Line */ - coord.X = 0; - coord.Y = b.dwCursorPosition.Y; - FillConsoleOutputCharacterA(hOut, ' ', b.dwSize.X, coord, &w); - /* Cursor to the left edge */ - SetConsoleCursorPosition(hOut, coord); - /* Write the prompt and the current buffer content */ - WriteConsole(hOut, prompt, (DWORD)plen, &pl, NULL); - WriteConsole(hOut, buf, (DWORD)len, &bl, NULL); - /* Move cursor to original position. */ - coord.X = (int)(pos+plen); - coord.Y = b.dwCursorPosition.Y; - SetConsoleCursorPosition(hOut, coord); -#endif } -static void beep() { +/* Beep, used for completion when there is nothing to complete or when all + * the choices were already shown. */ +static void linenoiseBeep(void) { fprintf(stderr, "\x7"); fflush(stderr); } +/* ============================== Completion ================================ */ + +/* Free a list of completion option populated by linenoiseAddCompletion(). */ static void freeCompletions(linenoiseCompletions *lc) { size_t i; for (i = 0; i < lc->len; i++) @@ -438,28 +497,39 @@ static void freeCompletions(linenoiseCompletions *lc) { free(lc->cvec); } -static int completeLine(int fd, const char *prompt, char *buf, size_t buflen, size_t *len, size_t *pos, size_t cols) { +/* This is an helper function for linenoiseEdit() and is called when the + * user types the key in order to complete the string currently in the + * input. + * + * The state of the editing is encapsulated into the pointed linenoiseState + * structure as described in the structure definition. */ +static int completeLine(struct linenoiseState *ls) { linenoiseCompletions lc = { 0, NULL }; int nread, nwritten; char c = 0; - completionCallback(buf,&lc); + completionCallback(ls->buf,&lc); if (lc.len == 0) { - beep(); + linenoiseBeep(); } else { size_t stop = 0, i = 0; - size_t clen; while(!stop) { /* Show completion or original buffer */ if (i < lc.len) { - clen = strlen(lc.cvec[i]); - refreshLine(fd,prompt,lc.cvec[i],clen,clen,cols); + struct linenoiseState saved = *ls; + + ls->len = ls->pos = strlen(lc.cvec[i]); + ls->buf = lc.cvec[i]; + refreshLine(ls); + ls->len = saved.len; + ls->pos = saved.pos; + ls->buf = saved.buf; } else { - refreshLine(fd,prompt,buf,*len,*pos,cols); + refreshLine(ls); } - nread = read(fd,&c,1); + nread = read(ls->ifd,&c,1); if (nread <= 0) { freeCompletions(&lc); return -1; @@ -468,20 +538,18 @@ static int completeLine(int fd, const char *prompt, char *buf, size_t buflen, si switch(c) { case 9: /* tab */ i = (i+1) % (lc.len+1); - if (i == lc.len) beep(); + if (i == lc.len) linenoiseBeep(); break; case 27: /* escape */ /* Re-show original buffer */ - if (i < lc.len) { - refreshLine(fd,prompt,buf,*len,*pos,cols); - } + if (i < lc.len) refreshLine(ls); stop = 1; break; default: /* Update buffer and return */ if (i < lc.len) { - nwritten = snprintf(buf,buflen,"%s",lc.cvec[i]); - *len = *pos = nwritten; + nwritten = snprintf(ls->buf,ls->buflen,"%s",lc.cvec[i]); + ls->len = ls->pos = nwritten; } stop = 1; break; @@ -493,240 +561,526 @@ static int completeLine(int fd, const char *prompt, char *buf, size_t buflen, si return c; /* Return last read character */ } -void linenoiseClearScreen(void) { - if (write(STDIN_FILENO,"\x1b[H\x1b[2J",7) <= 0) { - /* nothing to do, just to avoid warning. */ +/* Register a callback function to be called for tab-completion. */ +void linenoiseSetCompletionCallback(linenoiseCompletionCallback *fn) { + completionCallback = fn; +} + +/* This function is used by the callback function registered by the user + * in order to add completion options given the input string when the + * user typed . See the example.c source code for a very easy to + * understand example. */ +void linenoiseAddCompletion(linenoiseCompletions *lc, const char *str) { + size_t len = strlen(str); + char *copy, **cvec; + + copy = malloc(len+1); + if (copy == NULL) return; + memcpy(copy,str,len+1); + cvec = realloc(lc->cvec,sizeof(char*)*(lc->len+1)); + if (cvec == NULL) { + free(copy); + return; + } + lc->cvec = cvec; + lc->cvec[lc->len++] = copy; +} + +/* =========================== Line editing ================================= */ + +/* We define a very simple "append buffer" structure, that is an heap + * allocated string where we can append to. This is useful in order to + * write all the escape sequences in a buffer and flush them to the standard + * output in a single call, to avoid flickering effects. */ +struct abuf { + char *b; + int len; +}; + +static void abInit(struct abuf *ab) { + ab->b = NULL; + ab->len = 0; +} + +static void abAppend(struct abuf *ab, const char *s, int len) { + char *new = realloc(ab->b,ab->len+len); + + if (new == NULL) return; + memcpy(new+ab->len,s,len); + ab->b = new; + ab->len += len; +} + +static void abFree(struct abuf *ab) { + free(ab->b); +} + +/* Single line low level line refresh. + * + * Rewrite the currently edited line accordingly to the buffer content, + * cursor position, and number of columns of the terminal. */ +static void refreshSingleLine(struct linenoiseState *l) { + char seq[64]; + size_t plen = strlen(l->prompt); + int fd = l->ofd; + char *buf = l->buf; + size_t len = l->len; + size_t pos = l->pos; + struct abuf ab; + + while((plen+pos) >= l->cols) { + buf++; + len--; + pos--; + } + while (plen+len > l->cols) { + len--; + } + + abInit(&ab); + /* Cursor to left edge */ + snprintf(seq,64,"\x1b[0G"); + abAppend(&ab,seq,(int)strlen(seq)); + /* Write the prompt and the current buffer content */ + abAppend(&ab,l->prompt,(int)strlen(l->prompt)); + abAppend(&ab,buf,(int)len); + /* Erase to right */ + snprintf(seq,64,"\x1b[0K"); + abAppend(&ab,seq,(int)strlen(seq)); + /* Move cursor to original position. */ + snprintf(seq,64,"\x1b[0G\x1b[%dC", (int)(pos+plen)); + abAppend(&ab,seq,(int)strlen(seq)); + if (write(fd,ab.b,ab.len) == -1) {} /* Can't recover from write error. */ + abFree(&ab); +} + +/* Multi line low level line refresh. + * + * Rewrite the currently edited line accordingly to the buffer content, + * cursor position, and number of columns of the terminal. */ +static void refreshMultiLine(struct linenoiseState *l) { + char seq[64]; + int plen = (int)strlen(l->prompt); + int rows = (int)((plen+l->len+l->cols-1)/l->cols); /* rows used by current buf. */ + int rpos = (int)((plen+l->oldpos+l->cols)/l->cols); /* cursor relative row. */ + int rpos2; /* rpos after refresh. */ + int old_rows = (int)l->maxrows; + int fd = l->ofd, j; + struct abuf ab; + + /* Update maxrows if needed. */ + if (rows > (int)l->maxrows) l->maxrows = rows; + + /* First step: clear all the lines used before. To do so start by + * going to the last row. */ + abInit(&ab); + if (old_rows-rpos > 0) { + lndebug("go down %d", old_rows-rpos); + snprintf(seq,64,"\x1b[%dB", old_rows-rpos); + abAppend(&ab,seq,(int)strlen(seq)); + } + + /* Now for every row clear it, go up. */ + for (j = 0; j < old_rows-1; j++) { + lndebug("clear+up"); + snprintf(seq,64,"\x1b[0G\x1b[0K\x1b[1A"); + abAppend(&ab,seq,(int)strlen(seq)); + } + + /* Clean the top line. */ + lndebug("clear"); + snprintf(seq,64,"\x1b[0G\x1b[0K"); + abAppend(&ab,seq,(int)strlen(seq)); + + /* Write the prompt and the current buffer content */ + abAppend(&ab,l->prompt,(int)strlen(l->prompt)); + abAppend(&ab,l->buf,(int)l->len); + + /* If we are at the very end of the screen with our prompt, we need to + * emit a newline and move the prompt to the first column. */ + if (l->pos && + l->pos == l->len && + (l->pos+plen) % l->cols == 0) + { + lndebug(""); + abAppend(&ab,"\n",1); + snprintf(seq,64,"\x1b[0G"); + abAppend(&ab,seq,(int)strlen(seq)); + rows++; + if (rows > (int)l->maxrows) l->maxrows = rows; + } + + /* Move cursor to right position. */ + rpos2 = (int)((plen+l->pos+l->cols)/l->cols); /* current cursor relative row. */ + lndebug("rpos2 %d", rpos2); + + /* Go up till we reach the expected positon. */ + if (rows-rpos2 > 0) { + lndebug("go-up %d", rows-rpos2); + snprintf(seq,64,"\x1b[%dA", rows-rpos2); + abAppend(&ab,seq,(int)strlen(seq)); + } + + /* Set column. */ + lndebug("set col %d", 1+((plen+(int)l->pos) % (int)l->cols)); + snprintf(seq,64,"\x1b[%dG", 1+((plen+(int)l->pos) % (int)l->cols)); + abAppend(&ab,seq,(int)strlen(seq)); + + lndebug("\n"); + l->oldpos = l->pos; + + if (write(fd,ab.b,ab.len) == -1) {} /* Can't recover from write error. */ + abFree(&ab); +} + +/* Calls the two low level functions refreshSingleLine() or + * refreshMultiLine() according to the selected mode. */ +static void refreshLine(struct linenoiseState *l) { + if (mlmode) + refreshMultiLine(l); + else + refreshSingleLine(l); +} + +/* Insert the character 'c' at cursor current position. + * + * On error writing to the terminal -1 is returned, otherwise 0. */ +int linenoiseEditInsert(struct linenoiseState *l, char c) { + if (l->len < l->buflen) { + if (l->len == l->pos) { + l->buf[l->pos] = c; + l->pos++; + l->len++; + l->buf[l->len] = '\0'; + if ((!mlmode && l->plen+l->len < l->cols) /* || mlmode */) { + /* Avoid a full update of the line in the + * trivial case. */ + if (write(l->ofd,&c,1) == -1) return -1; + } else { + refreshLine(l); + } + } else { + memmove(l->buf+l->pos+1,l->buf+l->pos,l->len-l->pos); + l->buf[l->pos] = c; + l->len++; + l->pos++; + l->buf[l->len] = '\0'; + refreshLine(l); + } + } + return 0; +} + +/* Move cursor on the left. */ +void linenoiseEditMoveLeft(struct linenoiseState *l) { + if (l->pos > 0) { + l->pos--; + refreshLine(l); } } -static int linenoisePrompt(int fd, char *buf, size_t buflen, const char *prompt) { - size_t plen = strlen(prompt); - size_t pos = 0; - size_t len = 0; - size_t cols = getColumns(); - int history_index = 0; - size_t old_pos; - size_t diff; -#ifdef _WIN32 - DWORD foo; -#endif +/* Move cursor on the right. */ +void linenoiseEditMoveRight(struct linenoiseState *l) { + if (l->pos != l->len) { + l->pos++; + refreshLine(l); + } +} - buf[0] = '\0'; - buflen--; /* Make sure there is always space for the nulterm */ +/* Move cursor to the start of the line. */ +void linenoiseEditMoveHome(struct linenoiseState *l) { + if (l->pos != 0) { + l->pos = 0; + refreshLine(l); + } +} + +/* Move cursor to the end of the line. */ +void linenoiseEditMoveEnd(struct linenoiseState *l) { + if (l->pos != l->len) { + l->pos = l->len; + refreshLine(l); + } +} + +/* Substitute the currently edited line with the next or previous history + * entry as specified by 'dir'. */ +#define LINENOISE_HISTORY_NEXT 0 +#define LINENOISE_HISTORY_PREV 1 +void linenoiseEditHistoryNext(struct linenoiseState *l, int dir) { + if (history_len > 1) { + /* Update the current history entry before to + * overwrite it with the next one. */ + free(history[history_len - 1 - l->history_index]); + history[history_len - 1 - l->history_index] = strdup(l->buf); + /* Show the new entry */ + l->history_index += (dir == LINENOISE_HISTORY_PREV) ? 1 : -1; + if (l->history_index < 0) { + l->history_index = 0; + return; + } else if (l->history_index >= history_len) { + l->history_index = history_len-1; + return; + } + strncpy(l->buf,history[history_len - 1 - l->history_index],l->buflen); + l->buf[l->buflen-1] = '\0'; + l->len = l->pos = strlen(l->buf); + refreshLine(l); + } +} + +/* Delete the character at the right of the cursor without altering the cursor + * position. Basically this is what happens with the "Delete" keyboard key. */ +void linenoiseEditDelete(struct linenoiseState *l) { + if (l->len > 0 && l->pos < l->len) { + memmove(l->buf+l->pos,l->buf+l->pos+1,l->len-l->pos-1); + l->len--; + l->buf[l->len] = '\0'; + refreshLine(l); + } +} + +/* Backspace implementation. */ +void linenoiseEditBackspace(struct linenoiseState *l) { + if (l->pos > 0 && l->len > 0) { + memmove(l->buf+l->pos-1,l->buf+l->pos,l->len-l->pos); + l->pos--; + l->len--; + l->buf[l->len] = '\0'; + refreshLine(l); + } +} + +/* Delete the previosu word, maintaining the cursor at the start of the + * current word. */ +void linenoiseEditDeletePrevWord(struct linenoiseState *l) { + size_t old_pos = l->pos; + size_t diff; + + while (l->pos > 0 && l->buf[l->pos-1] == ' ') + l->pos--; + while (l->pos > 0 && l->buf[l->pos-1] != ' ') + l->pos--; + diff = old_pos - l->pos; + memmove(l->buf+l->pos,l->buf+old_pos,l->len-old_pos+1); + l->len -= diff; + refreshLine(l); +} + +/* This function is the core of the line editing capability of linenoise. + * It expects 'fd' to be already in "raw mode" so that every key pressed + * will be returned ASAP to read(). + * + * The resulting string is put into 'buf' when the user type enter, or + * when ctrl+d is typed. + * + * The function returns the length of the current buffer. */ +static int linenoiseEdit(int stdin_fd, int stdout_fd, char *buf, size_t buflen, const char *prompt) +{ + struct linenoiseState l; + + /* Populate the linenoise state that we pass to functions implementing + * specific editing functionalities. */ + l.ifd = stdin_fd; + l.ofd = stdout_fd; + l.buf = buf; + l.buflen = buflen; + l.prompt = prompt; + l.plen = strlen(prompt); + l.oldpos = l.pos = 0; + l.len = 0; + l.cols = getColumns(stdin_fd, stdout_fd); + l.maxrows = 0; + l.history_index = 0; + + /* Buffer starts empty. */ + l.buf[0] = '\0'; + l.buflen--; /* Make sure there is always space for the nulterm */ /* The latest history entry is always our current buffer, that * initially is just an empty string. */ linenoiseHistoryAdd(""); -#ifdef _WIN32 - if (!WriteConsole(hOut, prompt, (DWORD)plen, &foo, NULL)) return -1; -#else - if (write(fd,prompt,plen) == -1) return -1; -#endif + if (write(l.ofd,prompt,l.plen) == -1) return -1; while(1) { char c; int nread; - char seq[2], seq2[2]; + char seq[3]; -#ifdef _WIN32 - nread = win32read(&c); -#else - nread = read(fd,&c,1); -#endif - if (nread <= 0) return (int)len; + nread = read(l.ifd,&c,1); + if (nread <= 0) return (int)l.len; /* Only autocomplete when the callback is set. It returns < 0 when * there was an error reading from fd. Otherwise it will return the * character that should be handled next. */ if (c == 9 && completionCallback != NULL) { - c = completeLine(fd,prompt,buf,buflen,&len,&pos,cols); + c = completeLine(&l); /* Return on errors */ - if (c < 0) return (int)len; + if (c < 0) return (int)l.len; /* Read next character when 0 */ if (c == 0) continue; } switch(c) { - case 13: /* enter */ + case ENTER: /* enter */ history_len--; free(history[history_len]); - return (int)len; - case 3: /* ctrl-c */ + return (int)l.len; + case CTRL_C: /* ctrl-c */ errno = EAGAIN; return -1; - case 127: /* backspace */ -#ifdef _WIN32 - /* delete in _WIN32*/ - /* win32read() will send 127 for DEL and 8 for BS and Ctrl-H */ - if (pos < len && len > 0) { - memmove(buf+pos,buf+pos+1,len-pos); - len--; - buf[len] = '\0'; - refreshLine(fd,prompt,buf,len,pos,cols); - } - break; -#endif + case BACKSPACE: /* backspace */ case 8: /* ctrl-h */ - if (pos > 0 && len > 0) { - memmove(buf+pos-1,buf+pos,len-pos); - pos--; - len--; - buf[len] = '\0'; - refreshLine(fd,prompt,buf,len,pos,cols); - } + linenoiseEditBackspace(&l); break; - case 4: /* ctrl-d, remove char at right of cursor */ - if (len > 1 && pos < (len-1)) { - memmove(buf+pos,buf+pos+1,len-pos); - len--; - buf[len] = '\0'; - refreshLine(fd,prompt,buf,len,pos,cols); - } else if (len == 0) { + case CTRL_D: /* ctrl-d, remove char at right of cursor, or of the + line is empty, act as end-of-file. */ + if (l.len > 0) { + linenoiseEditDelete(&l); + } else { history_len--; free(history[history_len]); return -1; } break; - case 20: /* ctrl-t */ - if (pos > 0 && pos < len) { - int aux = buf[pos-1]; - buf[pos-1] = buf[pos]; - buf[pos] = aux; - if (pos != len-1) pos++; - refreshLine(fd,prompt,buf,len,pos,cols); + case CTRL_T: /* ctrl-t, swaps current character with previous. */ + if (l.pos > 0 && l.pos < l.len) { + int aux = buf[l.pos-1]; + buf[l.pos-1] = buf[l.pos]; + buf[l.pos] = aux; + if (l.pos != l.len-1) l.pos++; + refreshLine(&l); } break; - case 2: /* ctrl-b */ - goto left_arrow; - case 6: /* ctrl-f */ - goto right_arrow; - case 16: /* ctrl-p */ - seq[1] = 65; - goto up_down_arrow; - case 14: /* ctrl-n */ - seq[1] = 66; - goto up_down_arrow; + case CTRL_B: /* ctrl-b */ + linenoiseEditMoveLeft(&l); break; - case 27: /* escape sequence */ - if (read(fd,seq,2) == -1) break; - if (seq[0] == 91 && seq[1] == 68) { -left_arrow: - /* left arrow */ - if (pos > 0) { - pos--; - refreshLine(fd,prompt,buf,len,pos,cols); - } - } else if (seq[0] == 91 && seq[1] == 67) { -right_arrow: - /* right arrow */ - if (pos != len) { - pos++; - refreshLine(fd,prompt,buf,len,pos,cols); - } - } else if (seq[0] == 91 && (seq[1] == 65 || seq[1] == 66)) { -up_down_arrow: - /* up and down arrow: history */ - if (history_len > 1) { - /* Update the current history entry before to - * overwrite it with tne next one. */ - free(history[history_len-1-history_index]); - history[history_len-1-history_index] = strdup(buf); - /* Show the new entry */ - history_index += (seq[1] == 65) ? 1 : -1; - if (history_index < 0) { - history_index = 0; + case CTRL_F: /* ctrl-f */ + linenoiseEditMoveRight(&l); + break; + case CTRL_P: /* ctrl-p */ + linenoiseEditHistoryNext(&l, LINENOISE_HISTORY_PREV); + break; + case CTRL_N: /* ctrl-n */ + linenoiseEditHistoryNext(&l, LINENOISE_HISTORY_NEXT); + break; + case ESC: /* escape sequence */ + /* Read the next two bytes representing the escape sequence. + * Use two calls to handle slow terminals returning the two + * chars at different times. */ + if (read(l.ifd,seq,1) == -1) break; + if (read(l.ifd,seq+1,1) == -1) break; + + /* ESC [ sequences. */ + if (seq[0] == '[') { + if (seq[1] >= '0' && seq[1] <= '9') { + /* Extended escape, read additional byte. */ + if (read(l.ifd,seq+2,1) == -1) break; + if (seq[2] == '~') { + switch(seq[1]) { + case '3': /* Delete key. */ + linenoiseEditDelete(&l); + break; + } + } + } else { + switch(seq[1]) { + case 'A': /* Up */ + linenoiseEditHistoryNext(&l, LINENOISE_HISTORY_PREV); break; - } else if (history_index >= history_len) { - history_index = history_len-1; + case 'B': /* Down */ + linenoiseEditHistoryNext(&l, LINENOISE_HISTORY_NEXT); + break; + case 'C': /* Right */ + linenoiseEditMoveRight(&l); + break; + case 'D': /* Left */ + linenoiseEditMoveLeft(&l); + break; + case 'H': /* Home */ + linenoiseEditMoveHome(&l); + break; + case 'F': /* End*/ + linenoiseEditMoveEnd(&l); break; } - strncpy(buf,history[history_len-1-history_index],buflen); - buf[buflen] = '\0'; - len = pos = strlen(buf); - refreshLine(fd,prompt,buf,len,pos,cols); } - } else if (seq[0] == 91 && seq[1] > 48 && seq[1] < 55) { - /* extended escape */ - if (read(fd,seq2,2) == -1) break; - if (seq[1] == 51 && seq2[0] == 126) { - /* delete */ - if (len > 0 && pos < len) { - memmove(buf+pos,buf+pos+1,len-pos-1); - len--; - buf[len] = '\0'; - refreshLine(fd,prompt,buf,len,pos,cols); - } + } + + /* ESC O sequences. */ + else if (seq[0] == 'O') { + switch(seq[1]) { + case 'H': /* Home */ + linenoiseEditMoveHome(&l); + break; + case 'F': /* End*/ + linenoiseEditMoveEnd(&l); + break; } } break; default: - if (len < buflen) { - if (len == pos) { - buf[pos] = c; - pos++; - len++; - buf[len] = '\0'; - if (plen+len < cols) { - /* Avoid a full update of the line in the - * trivial case. */ -#ifdef _WIN32 - if (!WriteConsole(hOut, &c, 1, &foo, NULL)) return -1; -#else - if (write(fd,&c,1) == -1) return -1; -#endif - } else { - refreshLine(fd,prompt,buf,len,pos,cols); - } - } else { - memmove(buf+pos+1,buf+pos,len-pos); - buf[pos] = c; - len++; - pos++; - buf[len] = '\0'; - refreshLine(fd,prompt,buf,len,pos,cols); - } - } + if (linenoiseEditInsert(&l,c)) return -1; break; - case 21: /* Ctrl+u, delete the whole line. */ + case CTRL_U: /* Ctrl+u, delete the whole line. */ buf[0] = '\0'; - pos = len = 0; - refreshLine(fd,prompt,buf,len,pos,cols); + l.pos = l.len = 0; + refreshLine(&l); break; - case 11: /* Ctrl+k, delete from current to end of line. */ - buf[pos] = '\0'; - len = pos; - refreshLine(fd,prompt,buf,len,pos,cols); + case CTRL_K: /* Ctrl+k, delete from current to end of line. */ + buf[l.pos] = '\0'; + l.len = l.pos; + refreshLine(&l); break; - case 1: /* Ctrl+a, go to the start of the line */ - pos = 0; - refreshLine(fd,prompt,buf,len,pos,cols); + case CTRL_A: /* Ctrl+a, go to the start of the line */ + linenoiseEditMoveHome(&l); break; - case 5: /* ctrl+e, go to the end of the line */ - pos = len; - refreshLine(fd,prompt,buf,len,pos,cols); + case CTRL_E: /* ctrl+e, go to the end of the line */ + linenoiseEditMoveEnd(&l); break; - case 12: /* ctrl+l, clear screen */ + case CTRL_L: /* ctrl+l, clear screen */ linenoiseClearScreen(); - refreshLine(fd,prompt,buf,len,pos,cols); + refreshLine(&l); break; - case 23: /* ctrl+w, delete previous word */ - old_pos = pos; - while (pos > 0 && buf[pos-1] == ' ') - pos--; - while (pos > 0 && buf[pos-1] != ' ') - pos--; - diff = old_pos - pos; - memmove(&buf[pos], &buf[old_pos], len-old_pos+1); - len -= diff; - refreshLine(fd,prompt,buf,len,pos,cols); + case CTRL_W: /* ctrl+w, delete previous word */ + linenoiseEditDeletePrevWord(&l); break; } } - return (int)len; + return (int)l.len; } +/* This special mode is used by linenoise in order to print scan codes + * on screen for debugging / development purposes. It is implemented + * by the linenoise_example program using the --keycodes option. */ +void linenoisePrintKeyCodes(void) { + char quit[4]; + + printf("Linenoise key codes debugging mode.\n" + "Press keys to see scan codes. Type 'quit' at any time to exit.\n"); + if (enableRawMode(STDIN_FILENO) == -1) return; + memset(quit,' ',4); + while(1) { + char c; + int nread; + + nread = read(STDIN_FILENO,&c,1); + if (nread <= 0) continue; + memmove(quit,quit+1,sizeof(quit)-1); /* shift string to left. */ + quit[sizeof(quit)-1] = c; /* Insert current char on the right. */ + if (memcmp(quit,"quit",sizeof(quit)) == 0) break; + + printf("'%c' %02x (%d) (type quit to exit)\n", + isprint(c) ? c : '?', (int)c, (int)c); + printf("\x1b[0G"); /* Go left edge manually, we are in raw mode. */ + fflush(stdout); + } + disableRawMode(STDIN_FILENO); +} + +/* This function calls the line editing function linenoiseEdit() using + * the STDIN file descriptor set in raw mode. */ static int linenoiseRaw(char *buf, size_t buflen, const char *prompt) { - int fd = STDIN_FILENO; int count; if (buflen == 0) { @@ -734,6 +1088,7 @@ static int linenoiseRaw(char *buf, size_t buflen, const char *prompt) { return -1; } if (!isatty(STDIN_FILENO)) { + /* Not a tty: read from file / pipe. */ if (fgets(buf, (int)buflen, stdin) == NULL) return -1; count = (int)strlen(buf); if (count && buf[count-1] == '\n') { @@ -741,14 +1096,20 @@ static int linenoiseRaw(char *buf, size_t buflen, const char *prompt) { buf[count] = '\0'; } } else { - if (enableRawMode(fd) == -1) return -1; - count = linenoisePrompt(fd, buf, buflen, prompt); - disableRawMode(fd); + /* Interactive editing. */ + if (enableRawMode(STDIN_FILENO) == -1) return -1; + count = linenoiseEdit(STDIN_FILENO, STDOUT_FILENO, buf, buflen, prompt); + disableRawMode(STDIN_FILENO); printf("\n"); } return count; } +/* The high level function that is the main API of the linenoise library. + * This function checks if the terminal has basic capabilities, just checking + * for a blacklist of stupid terminals, and later either calls the line + * editing function or uses dummy fgets() so that you will be able to type + * something even in the most desperate of the conditions. */ char *linenoise(const char *prompt) { char buf[LINENOISE_MAX_LINE]; int count; @@ -772,29 +1133,50 @@ char *linenoise(const char *prompt) { } } -/* Register a callback function to be called for tab-completion. */ -void linenoiseSetCompletionCallback(linenoiseCompletionCallback *fn) { - completionCallback = fn; +/* ================================ History ================================= */ + +/* Free the history, but does not reset it. Only used when we have to + * exit() to avoid memory leaks are reported by valgrind & co. */ +static void freeHistory(void) { + if (history) { + int j; + + for (j = 0; j < history_len; j++) + free(history[j]); + free(history); + } } -void linenoiseAddCompletion(linenoiseCompletions *lc, char *str) { - size_t len = strlen(str); - char *copy = malloc(len+1); - memcpy(copy,str,len+1); - lc->cvec = realloc(lc->cvec,sizeof(char*)*(lc->len+1)); - lc->cvec[lc->len++] = copy; +/* At exit we'll try to fix the terminal to the initial conditions. */ +static void linenoiseAtExit(void) { + disableRawMode(STDIN_FILENO); + freeHistory(); } -/* Using a circular buffer is smarter, but a bit more complex to handle. */ +/* This is the API call to add a new entry in the linenoise history. + * It uses a fixed array of char pointers that are shifted (memmoved) + * when the history max length is reached in order to remove the older + * entry and make room for the new one, so it is not exactly suitable for huge + * histories, but will work well for a few hundred of entries. + * + * Using a circular buffer is smarter, but a bit more complex to handle. */ int linenoiseHistoryAdd(const char *line) { char *linecopy; if (history_max_len == 0) return 0; + + /* Initialization on first call. */ if (history == NULL) { history = malloc(sizeof(char*)*history_max_len); if (history == NULL) return 0; memset(history,0,(sizeof(char*)*history_max_len)); } + + /* Don't add duplicated lines. */ + if (history_len && !strcmp(history[history_len-1], line)) return 0; + + /* Add an heap allocated copy of the line in the history. + * If we reached the max length, remove the older line. */ linecopy = strdup(line); if (!linecopy) return 0; if (history_len == history_max_len) { @@ -807,6 +1189,10 @@ int linenoiseHistoryAdd(const char *line) { return 1; } +/* Set the maximum length for the history. This function can be called even + * if there is already some history, the function will make sure to retain + * just the latest 'len' elements if the new history length value is smaller + * than the amount of items already inside the history. */ int linenoiseHistorySetMaxLen(int len) { char **new; @@ -816,8 +1202,16 @@ int linenoiseHistorySetMaxLen(int len) { new = malloc(sizeof(char*)*len); if (new == NULL) return 0; - if (len < tocopy) tocopy = len; - memcpy(new,history+(history_max_len-tocopy), sizeof(char*)*tocopy); + + /* If we can't copy everything, free the elements we'll not use. */ + if (len < tocopy) { + int j; + + for (j = 0; j < tocopy-len; j++) free(history[j]); + tocopy = len; + } + memset(new,0,sizeof(char*)*len); + memcpy(new,history+(history_len-tocopy), sizeof(char*)*tocopy); free(history); history = new; } @@ -829,7 +1223,7 @@ int linenoiseHistorySetMaxLen(int len) { /* Save the history in the specified file. On success 0 is returned * otherwise -1 is returned. */ -int linenoiseHistorySave(char *filename) { +int linenoiseHistorySave(const char *filename) { #ifdef _WIN32 FILE *fp = fopen(filename,"wb"); #else @@ -849,7 +1243,7 @@ int linenoiseHistorySave(char *filename) { * * If the file exists and the operation succeeded 0 is returned, otherwise * on error -1 is returned. */ -int linenoiseHistoryLoad(char *filename) { +int linenoiseHistoryLoad(const char *filename) { FILE *fp = fopen(filename,"r"); char buf[LINENOISE_MAX_LINE]; diff --git a/deps/linenoise/linenoise.h b/deps/linenoise/linenoise.h index 76a703c2..e22ebd3f 100644 --- a/deps/linenoise/linenoise.h +++ b/deps/linenoise/linenoise.h @@ -3,39 +3,44 @@ * * See linenoise.c for more information. * + * ------------------------------------------------------------------------ + * * Copyright (c) 2010, Salvatore Sanfilippo * Copyright (c) 2010, Pieter Noordhuis * * All rights reserved. - * + * * Redistribution and use in source and binary forms, with or without - * modification, are permitted provided that the following conditions are met: + * modification, are permitted provided that the following conditions are + * met: + * + * * Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. * - * * Redistributions of source code must retain the above copyright notice, - * this list of conditions and the following disclaimer. - * * Redistributions in binary form must reproduce the above copyright + * * Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in the * documentation and/or other materials provided with the distribution. - * * Neither the name of Redis nor the names of its contributors may be used - * to endorse or promote products derived from this software without - * specific prior written permission. - * - * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" - * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE - * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE - * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE - * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR - * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF - * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS - * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN - * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) - * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE - * POSSIBILITY OF SUCH DAMAGE. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS + * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR + * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT + * HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, + * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY + * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT + * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE + * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. */ #ifndef __LINENOISE_H #define __LINENOISE_H +#ifdef __cplusplus +extern "C" { +#endif + typedef struct linenoiseCompletions { size_t len; char **cvec; @@ -43,13 +48,19 @@ typedef struct linenoiseCompletions { typedef void(linenoiseCompletionCallback)(const char *, linenoiseCompletions *); void linenoiseSetCompletionCallback(linenoiseCompletionCallback *); -void linenoiseAddCompletion(linenoiseCompletions *, char *); +void linenoiseAddCompletion(linenoiseCompletions *, const char *); char *linenoise(const char *prompt); int linenoiseHistoryAdd(const char *line); int linenoiseHistorySetMaxLen(int len); -int linenoiseHistorySave(char *filename); -int linenoiseHistoryLoad(char *filename); +int linenoiseHistorySave(const char *filename); +int linenoiseHistoryLoad(const char *filename); void linenoiseClearScreen(void); +void linenoiseSetMultiLine(int ml); +void linenoisePrintKeyCodes(void); + +#ifdef __cplusplus +} +#endif #endif /* __LINENOISE_H */ diff --git a/msvs/RedisServer.vcxproj b/msvs/RedisServer.vcxproj index 1010dd80..f9900141 100644 --- a/msvs/RedisServer.vcxproj +++ b/msvs/RedisServer.vcxproj @@ -104,6 +104,7 @@ + diff --git a/msvs/setups/documentation/Redis Release Notes.docx b/msvs/setups/documentation/Redis Release Notes.docx index ac68a3e0..493e8b26 100644 Binary files a/msvs/setups/documentation/Redis Release Notes.docx and b/msvs/setups/documentation/Redis Release Notes.docx differ diff --git a/redis.conf b/redis.conf index 574141b3..ff49faa3 100644 --- a/redis.conf +++ b/redis.conf @@ -44,6 +44,15 @@ pidfile /var/run/redis.pid # If port 0 is specified Redis will not listen on a TCP socket. port 6379 +# TCP listen() backlog. +# +# In high requests-per-second environments you need an high backlog in order +# to avoid slow clients connections issues. Note that the Linux kernel +# will silently truncate it to the value of /proc/sys/net/core/somaxconn so +# make sure to raise both the value of somaxconn and tcp_max_syn_backlog +# in order to get the desired effect. +tcp-backlog 511 + # By default Redis listens for connections from all the network interfaces # available on the server. It is possible to listen to just one or multiple # interfaces using the "bind" configuration directive, followed by one or @@ -626,6 +635,20 @@ set-max-intset-entries 512 zset-max-ziplist-entries 128 zset-max-ziplist-value 64 +# HyperLogLog sparse representation bytes limit. The limit includes the +# 16 bytes header. When an HyperLogLog using the sparse representation crosses +# this limit, it is convereted into the dense representation. +# +# A value greater than 16000 is totally useless, since at that point the +# dense representation is more memory efficient. +# +# The suggested value is ~ 3000 in order to have the benefits of +# the space efficient encoding without slowing down too much PFADD, +# which is O(N) with the sparse encoding. Thev value can be raised to +# ~ 10000 when CPU is not a concern, but space is, and the data set is +# composed of many HyperLogLogs with cardinality in the 0 - 15000 range. +hll-sparse-max-bytes 3000 + # Active rehashing uses 1 millisecond every 100 milliseconds of CPU time in # order to help rehashing the main Redis hash table (the one mapping top-level # keys to values). The hash table implementation Redis uses (see dict.c) diff --git a/runtest-sentinel b/runtest-sentinel new file mode 100755 index 00000000..1650eea7 --- /dev/null +++ b/runtest-sentinel @@ -0,0 +1,14 @@ +#!/bin/sh +TCL_VERSIONS="8.5 8.6" +TCLSH="" + +for VERSION in $TCL_VERSIONS; do + TCL=`which tclsh$VERSION 2>/dev/null` && TCLSH=$TCL +done + +if [ -z $TCLSH ] +then + echo "You need tcl 8.5 or newer in order to run the Redis Sentinel test" + exit 1 +fi +$TCLSH tests/sentinel.tcl $* diff --git a/sentinel.conf b/sentinel.conf index e4434222..ff255f87 100644 --- a/sentinel.conf +++ b/sentinel.conf @@ -86,10 +86,10 @@ sentinel failover-timeout mymaster 180000 # or to reconfigure clients after a failover. The scripts are executed # with the following rules for error handling: # -# If script exists with "1" the execution is retried later (up to a maximum +# If script exits with "1" the execution is retried later (up to a maximum # number of times currently set to 10). # -# If script exists with "2" (or an higher value) the script execution is +# If script exits with "2" (or an higher value) the script execution is # not retried. # # If script terminates because it receives a signal the behavior is the same diff --git a/src/Makefile b/src/Makefile index a8631e02..995e4914 100644 --- a/src/Makefile +++ b/src/Makefile @@ -107,7 +107,7 @@ endif REDIS_SERVER_NAME=redis-server REDIS_SENTINEL_NAME=redis-sentinel -REDIS_SERVER_OBJ=adlist.o ae.o anet.o dict.o redis.o sds.o zmalloc.o lzf_c.o lzf_d.o pqsort.o zipmap.o sha1.o ziplist.o release.o networking.o util.o object.o db.o replication.o rdb.o t_string.o t_list.o t_set.o t_zset.o t_hash.o config.o aof.o pubsub.o multi.o debug.o sort.o intset.o syncio.o migrate.o endianconv.o slowlog.o scripting.o bio.o rio.o rand.o memtest.o crc64.o bitops.o sentinel.o notify.o setproctitle.o +REDIS_SERVER_OBJ=adlist.o ae.o anet.o dict.o redis.o sds.o zmalloc.o lzf_c.o lzf_d.o pqsort.o zipmap.o sha1.o ziplist.o release.o networking.o util.o object.o db.o replication.o rdb.o t_string.o t_list.o t_set.o t_zset.o t_hash.o config.o aof.o pubsub.o multi.o debug.o sort.o intset.o syncio.o migrate.o endianconv.o slowlog.o scripting.o bio.o rio.o rand.o memtest.o crc64.o bitops.o sentinel.o notify.o setproctitle.o hyperloglog.o REDIS_CLI_NAME=redis-cli REDIS_CLI_OBJ=anet.o sds.o adlist.o redis-cli.o zmalloc.o release.o anet.o ae.o crc64.o REDIS_BENCHMARK_NAME=redis-benchmark @@ -204,6 +204,9 @@ distclean: clean test: $(REDIS_SERVER_NAME) $(REDIS_CHECK_AOF_NAME) @(cd ..; ./runtest) +test-sentinel: $(REDIS_SENTINEL_NAME) + @(cd ..; ./runtest-sentinel) + check: test lcov: diff --git a/src/Makefile.dep b/src/Makefile.dep index 7c1fea0d..c354781c 100644 --- a/src/Makefile.dep +++ b/src/Makefile.dep @@ -26,6 +26,10 @@ debug.o: debug.c redis.h fmacros.h config.h ../deps/lua/src/lua.h \ ziplist.h intset.h version.h util.h rdb.h rio.h sha1.h crc64.h bio.h dict.o: dict.c fmacros.h dict.h zmalloc.h redisassert.h endianconv.o: endianconv.c +hyperloglog.o: hyperloglog.c redis.h fmacros.h config.h \ + ../deps/lua/src/lua.h ../deps/lua/src/luaconf.h ae.h sds.h dict.h \ + adlist.h zmalloc.h anet.h ziplist.h intset.h version.h util.h rdb.h \ + rio.h intset.o: intset.c intset.h zmalloc.h endianconv.h config.h lzf_c.o: lzf_c.c lzfP.h lzf_d.o: lzf_d.c lzfP.h diff --git a/src/Win32_Interop/Win32_FDAPI.cpp b/src/Win32_Interop/Win32_FDAPI.cpp index be69fd82..f51a3bc5 100644 --- a/src/Win32_Interop/Win32_FDAPI.cpp +++ b/src/Win32_Interop/Win32_FDAPI.cpp @@ -178,7 +178,6 @@ bool IsWindowsVersionAtLeast(WORD wMajorVersion, WORD wMinorVersion, WORD wServi void EnableFastLoopback(SOCKET s) { // if Win8+, use fast path option on loopback -// if ( false ) { if (IsWindowsVersionAtLeast(HIBYTE(_WIN32_WINNT_WIN8), LOBYTE(_WIN32_WINNT_WIN8), 0)) { #ifndef SIO_LOOPBACK_FAST_PATH const DWORD SIO_LOOPBACK_FAST_PATH = 0x98000010; // from Win8 SDK diff --git a/src/Win32_Interop/Win32_QFork.cpp b/src/Win32_Interop/Win32_QFork.cpp index 0a854411..03b7931c 100644 --- a/src/Win32_Interop/Win32_QFork.cpp +++ b/src/Win32_Interop/Win32_QFork.cpp @@ -41,6 +41,9 @@ #include using namespace std; +const long long cSentinelHeapSize = 30 * 1024 * 1024; +extern "C" int checkForSentinelMode(int argc, char **argv); + extern "C" { // forward def from util.h. @@ -656,10 +659,14 @@ StartupStatus QForkStartup(int argc, char** argv) { } if( maxheapBytes == -1 ) { - maxheapBytes = perfinfo.PhysicalTotal * pageSize; + if (checkForSentinelMode(argc, argv)) { + // Sentinel mode does not need a large heap. This conserves disk space and page file reservation requirements. + maxheapBytes = cSentinelHeapSize; + } else { + maxheapBytes = perfinfo.PhysicalTotal * pageSize; + } } - if (foundSlaveFlag) { LPVOID exceptionHandler = AddVectoredExceptionHandler( 1, VectoredHeapMapper ); StartupStatus retVal = StartupStatus::ssFAILED; diff --git a/src/anet.c b/src/anet.c index a9f0b885..63da2180 100644 --- a/src/anet.c +++ b/src/anet.c @@ -452,18 +452,15 @@ int anetWrite(int fd, char *buf, int count) return totlen; } -static int anetListen(char *err, int s, struct sockaddr *sa, socklen_t len) { +static int anetListen(char *err, int s, struct sockaddr *sa, socklen_t len, int backlog) { if (bind(s,sa,len) == -1) { anetSetError(err, "bind: %s", strerror(errno)); close(s); return ANET_ERR; } - /* Use a backlog of 512 entries. We pass 511 to the listen() call because - * the kernel does: backlogsize = roundup_pow_of_two(backlogsize + 1); - * which will thus give us a backlog of 512 entries */ #ifdef _WIN32 - if (aeWinListen(s, 512) == SOCKET_ERROR) { + if (aeWinListen(s, 512) == SOCKET_ERROR) { #else if (listen(s, 511) == -1) { #endif @@ -484,7 +481,7 @@ static int anetV6Only(char *err, int s) { return ANET_OK; } -static int _anetTcpServer(char *err, int port, char *bindaddr, int af) +static int _anetTcpServer(char *err, int port, char *bindaddr, int af, int backlog) { int s, rv; char _port[6]; /* strlen("65535") */ @@ -510,7 +507,7 @@ static int _anetTcpServer(char *err, int port, char *bindaddr, int af) #else if (anetSetReuseAddr(err,s) == ANET_ERR) goto error; #endif - if (anetListen(err,s,p->ai_addr,p->ai_addrlen) == ANET_ERR) goto error; + if (anetListen(err,s,p->ai_addr,p->ai_addrlen,backlog) == ANET_ERR) goto error; goto end; } if (p == NULL) { @@ -525,17 +522,17 @@ end: return s; } -int anetTcpServer(char *err, int port, char *bindaddr) +int anetTcpServer(char *err, int port, char *bindaddr, int backlog) { - return _anetTcpServer(err, port, bindaddr, AF_INET); + return _anetTcpServer(err, port, bindaddr, AF_INET, backlog); } -int anetTcp6Server(char *err, int port, char *bindaddr) +int anetTcp6Server(char *err, int port, char *bindaddr, int backlog) { - return _anetTcpServer(err, port, bindaddr, AF_INET6); + return _anetTcpServer(err, port, bindaddr, AF_INET6, backlog); } -int anetUnixServer(char *err, char *path, mode_t perm) +int anetUnixServer(char *err, char *path, mode_t perm, int backlog) { #ifdef _WIN32 ANET_NOTUSED(err); @@ -552,7 +549,7 @@ int anetUnixServer(char *err, char *path, mode_t perm) memset(&sa,0,sizeof(sa)); sa.sun_family = AF_LOCAL; strncpy(sa.sun_path,path,sizeof(sa.sun_path)-1); - if (anetListen(err,s,(struct sockaddr*)&sa,sizeof(sa)) == ANET_ERR) + if (anetListen(err,s,(struct sockaddr*)&sa,sizeof(sa),backlog) == ANET_ERR) return ANET_ERR; if (perm) chmod(sa.sun_path, perm); diff --git a/src/anet.h b/src/anet.h index 2ab9398a..3f893be2 100644 --- a/src/anet.h +++ b/src/anet.h @@ -50,9 +50,9 @@ int anetUnixNonBlockConnect(char *err, char *path); int anetRead(int fd, char *buf, int count); int anetResolve(char *err, char *host, char *ipbuf, size_t ipbuf_len); int anetResolveIP(char *err, char *host, char *ipbuf, size_t ipbuf_len); -int anetTcpServer(char *err, int port, char *bindaddr); -int anetTcp6Server(char *err, int port, char *bindaddr); -int anetUnixServer(char *err, char *path, mode_t perm); +int anetTcpServer(char *err, int port, char *bindaddr, int backlog); +int anetTcp6Server(char *err, int port, char *bindaddr, int backlog); +int anetUnixServer(char *err, char *path, mode_t perm, int backlog); int anetTcpAccept(char *err, int serversock, char *ip, size_t ip_len, int *port); int anetUnixAccept(char *err, int serversock); int anetWrite(int fd, char *buf, int count); diff --git a/src/aof.c b/src/aof.c index 95234491..fc9acbc1 100644 --- a/src/aof.c +++ b/src/aof.c @@ -129,7 +129,7 @@ void aofRewriteBufferAppend(unsigned char *s, unsigned long len) { } /* Write the buffer (possibly composed of multiple blocks) into the specified - * fd. If no short write or any other error happens -1 is returned, + * fd. If a short write or any other error happens -1 is returned, * otherwise the number of bytes written is returned. */ ssize_t aofRewriteBufferWrite(int fd) { listNode *ln; @@ -239,6 +239,7 @@ int startAppendOnly(void) { * * However if force is set to 1 we'll write regardless of the background * fsync. */ +#define AOF_WRITE_LOG_ERROR_RATE 30 /* Seconds between errors logging. */ void flushAppendOnlyFile(int force) { ssize_t nwritten; int sync_in_progress = 0; @@ -280,27 +281,76 @@ void flushAppendOnlyFile(int force) { * or alike */ nwritten = write(server.aof_fd,server.aof_buf,(unsigned int)sdslen(server.aof_buf)); if (nwritten != (signed)sdslen(server.aof_buf)) { - /* Ooops, we are in troubles. The best thing to do for now is - * aborting instead of giving the illusion that everything is - * working as expected. */ + static time_t last_write_error_log = 0; + int can_log = 0; + + /* Limit logging rate to 1 line per AOF_WRITE_LOG_ERROR_RATE seconds. */ + if ((server.unixtime - last_write_error_log) > AOF_WRITE_LOG_ERROR_RATE) { + can_log = 1; + last_write_error_log = server.unixtime; + } + + /* Lof the AOF write error and record the error code. */ if (nwritten == -1) { - redisLog(REDIS_WARNING,"Exiting on error writing to the append-only file: %s",strerror(errno)); + if (can_log) { + redisLog(REDIS_WARNING,"Error writing to the AOF file: %s", + strerror(errno)); + server.aof_last_write_errno = errno; + } } else { - redisLog(REDIS_WARNING,"Exiting on short write while writing to " - "the append-only file: %s (nwritten=%ld, " - "expected=%ld)", - strerror(errno), - (long)nwritten, - (long)sdslen(server.aof_buf)); + if (can_log) { + redisLog(REDIS_WARNING,"Short write while writing to " + "the AOF file: (nwritten=%lld, " + "expected=%lld)", + (long long)nwritten, + (long long)sdslen(server.aof_buf)); + } if (ftruncate(server.aof_fd, server.aof_current_size) == -1) { - redisLog(REDIS_WARNING, "Could not remove short write " - "from the append-only file. Redis may refuse " - "to load the AOF the next time it starts. " - "ftruncate: %s", strerror(errno)); + if (can_log) { + redisLog(REDIS_WARNING, "Could not remove short write " + "from the append-only file. Redis may refuse " + "to load the AOF the next time it starts. " + "ftruncate: %s", strerror(errno)); + } + } else { + /* If the ftrunacate() succeeded we can set nwritten to + * -1 since there is no longer partial data into the AOF. */ + nwritten = -1; } + server.aof_last_write_errno = ENOSPC; + } + + /* Handle the AOF write error. */ + if (server.aof_fsync == AOF_FSYNC_ALWAYS) { + /* We can't recover when the fsync policy is ALWAYS since the + * reply for the client is already in the output buffers, and we + * have the contract with the user that on acknowledged write data + * is synched on disk. */ + redisLog(REDIS_WARNING,"Can't recover from AOF write error when the AOF fsync policy is 'always'. Exiting..."); + exit(1); + } else { + /* Recover from failed write leaving data into the buffer. However + * set an error to stop accepting writes as long as the error + * condition is not cleared. */ + server.aof_last_write_status = REDIS_ERR; + + /* Trim the sds buffer if there was a partial write, and there + * was no way to undo it with ftruncate(2). */ + if (nwritten > 0) { + server.aof_current_size += nwritten; + sdsrange(server.aof_buf,nwritten,-1); + } + return; /* We'll try again on the next call... */ + } + } else { + /* Successful write(2). If AOF was in error state, restore the + * OK state and log the event. */ + if (server.aof_last_write_status == REDIS_ERR) { + redisLog(REDIS_WARNING, + "AOF write error looks solved, Redis can write again."); + server.aof_last_write_status = REDIS_OK; } - exit(1); } server.aof_current_size += nwritten; @@ -946,9 +996,9 @@ int rewriteAppendOnlyFile(char *filename) { } /* Make sure data will not remain on the OS's output buffers */ - fflush(fp); - aof_fsync(fileno(fp)); - fclose(fp); + if (fflush(fp) == EOF) goto werr; + if (aof_fsync(fileno(fp)) == -1) goto werr; + if (fclose(fp) == EOF) goto werr; /* Use RENAME to make sure the DB file is changed atomically only * if the generate DB file is ok. */ diff --git a/src/bitops.c b/src/bitops.c index 9179194b..aed666f8 100644 --- a/src/bitops.c +++ b/src/bitops.c @@ -60,11 +60,18 @@ static int getBitOffsetFromArgument(redisClient *c, robj *o, size_t *offset) { * work with a input string length up to 512 MB. */ size_t redisPopcount(void *s, long count) { size_t bits = 0; - unsigned char *p; - uint32_t *p4 = s; + unsigned char *p = s; + uint32_t *p4; static const unsigned char bitsinbyte[256] = {0,1,1,2,1,2,2,3,1,2,2,3,2,3,3,4,1,2,2,3,2,3,3,4,2,3,3,4,3,4,4,5,1,2,2,3,2,3,3,4,2,3,3,4,3,4,4,5,2,3,3,4,3,4,4,5,3,4,4,5,4,5,5,6,1,2,2,3,2,3,3,4,2,3,3,4,3,4,4,5,2,3,3,4,3,4,4,5,3,4,4,5,4,5,5,6,2,3,3,4,3,4,4,5,3,4,4,5,4,5,5,6,3,4,4,5,4,5,5,6,4,5,5,6,5,6,6,7,1,2,2,3,2,3,3,4,2,3,3,4,3,4,4,5,2,3,3,4,3,4,4,5,3,4,4,5,4,5,5,6,2,3,3,4,3,4,4,5,3,4,4,5,4,5,5,6,3,4,4,5,4,5,5,6,4,5,5,6,5,6,6,7,2,3,3,4,3,4,4,5,3,4,4,5,4,5,5,6,3,4,4,5,4,5,5,6,4,5,5,6,5,6,6,7,3,4,4,5,4,5,5,6,4,5,5,6,5,6,6,7,4,5,5,6,5,6,6,7,5,6,6,7,6,7,7,8}; + /* Count initial bytes not aligned to 32 bit. */ + while((unsigned long)p & 3 && count) { + bits += bitsinbyte[*p++]; + count--; + } + /* Count bits 16 bytes at a time */ + p4 = (uint32_t*)p; while(count>=16) { uint32_t aux1, aux2, aux3, aux4; @@ -87,12 +94,99 @@ size_t redisPopcount(void *s, long count) { ((((aux3 + (aux3 >> 4)) & 0x0F0F0F0F) * 0x01010101) >> 24) + ((((aux4 + (aux4 >> 4)) & 0x0F0F0F0F) * 0x01010101) >> 24); } - /* Count the remaining bytes */ + /* Count the remaining bytes. */ p = (unsigned char*)p4; while(count--) bits += bitsinbyte[*p++]; return bits; } +/* Return the position of the first bit set to one (if 'bit' is 1) or + * zero (if 'bit' is 0) in the bitmap starting at 's' and long 'count' bytes. + * + * The function is guaranteed to return a value >= 0 if 'bit' is 0 since if + * no zero bit is found, it returns count*8 assuming the string is zero + * padded on the right. However if 'bit' is 1 it is possible that there is + * not a single set bit in the bitmap. In this special case -1 is returned. */ +long redisBitpos(void *s, long count, int bit) { + unsigned long *l; + unsigned char *c; + unsigned long skipval, word = 0, one; + long pos = 0; /* Position of bit, to return to the caller. */ + int j; + + /* Process whole words first, seeking for first word that is not + * all ones or all zeros respectively if we are lookig for zeros + * or ones. This is much faster with large strings having contiguous + * blocks of 1 or 0 bits compared to the vanilla bit per bit processing. + * + * Note that if we start from an address that is not aligned + * to sizeof(unsigned long) we consume it byte by byte until it is + * aligned. */ + + /* Skip initial bits not aligned to sizeof(unsigned long) byte by byte. */ + skipval = bit ? 0 : UCHAR_MAX; + c = (unsigned char*) s; + while((unsigned long)c & (sizeof(*l)-1) && count) { + if (*c != skipval) break; + c++; + count--; + pos += 8; + } + + /* Skip bits with full word step. */ + skipval = bit ? 0 : ULONG_MAX; + l = (unsigned long*) c; + while (count >= sizeof(*l)) { + if (*l != skipval) break; + l++; + count -= sizeof(*l); + pos += sizeof(*l)*8; + } + + /* Load bytes into "word" considering the first byte as the most significant + * (we basically consider it as written in big endian, since we consider the + * string as a set of bits from left to right, with the first bit at position + * zero. + * + * Note that the loading is designed to work even when the bytes left + * (count) are less than a full word. We pad it with zero on the right. */ + c = (unsigned char*)l; + for (j = 0; j < sizeof(*l); j++) { + word <<= 8; + if (count) { + word |= *c; + c++; + count--; + } + } + + /* Special case: + * If bits in the string are all zero and we are looking for one, + * return -1 to signal that there is not a single "1" in the whole + * string. This can't happen when we are looking for "0" as we assume + * that the right of the string is zero padded. */ + if (bit == 1 && word == 0) return -1; + + /* Last word left, scan bit by bit. The first thing we need is to + * have a single "1" set in the most significant position in an + * unsigned long. We don't know the size of the long so we use a + * simple trick. */ + one = ULONG_MAX; /* All bits set to 1.*/ + one >>= 1; /* All bits set to 1 but the MSB. */ + one = ~one; /* All bits set to 0 but the MSB. */ + + while(one) { + if (((one & word) != 0) == bit) return pos; + pos++; + one >>= 1; + } + + /* If we reached this point, there is a bug in the algorithm, since + * the case of no match is handled as a special case before. */ + redisPanic("End of redisBitpos() reached."); + return 0; /* Just to avoid warnings. */ +} + /* ----------------------------------------------------------------------------- * Bits related string commands: GETBIT, SETBIT, BITCOUNT, BITOP. * -------------------------------------------------------------------------- */ @@ -129,14 +223,7 @@ void setbitCommand(redisClient *c) { dbAdd(c->db,c->argv[1],o); } else { if (checkType(c,o,REDIS_STRING)) return; - - /* Create a copy when the object is shared or encoded. */ - if (o->refcount != 1 || o->encoding != REDIS_ENCODING_RAW) { - robj *decoded = getDecodedObject(o); - o = createStringObject(decoded->ptr, sdslen(decoded->ptr)); - decrRefCount(decoded); - dbOverwrite(c->db,c->argv[1],o); - } + o = dbUnshareStringValue(c->db,c->argv[1],o); } /* Grow sds value to the right length if necessary */ @@ -410,3 +497,90 @@ void bitcountCommand(redisClient *c) { addReplyLongLong(c,redisPopcount(p+start,bytes)); } } + +/* BITPOS key bit [start [end]] */ +void bitposCommand(redisClient *c) { + robj *o; + long bit, start, end, strlen; + unsigned char *p; + char llbuf[32]; + int end_given = 0; + + /* Parse the bit argument to understand what we are looking for, set + * or clear bits. */ + if (getLongFromObjectOrReply(c,c->argv[2],&bit,NULL) != REDIS_OK) + return; + if (bit != 0 && bit != 1) { + addReplyError(c, "The bit argument must be 1 or 0."); + return; + } + + /* If the key does not exist, from our point of view it is an infinite + * array of 0 bits. If the user is looking for the fist clear bit return 0, + * If the user is looking for the first set bit, return -1. */ + if ((o = lookupKeyRead(c->db,c->argv[1])) == NULL) { + addReplyLongLong(c, bit ? -1 : 0); + return; + } + if (checkType(c,o,REDIS_STRING)) return; + + /* Set the 'p' pointer to the string, that can be just a stack allocated + * array if our string was integer encoded. */ + if (o->encoding == REDIS_ENCODING_INT) { + p = (unsigned char*) llbuf; + strlen = ll2string(llbuf,sizeof(llbuf),(long)o->ptr); + } else { + p = (unsigned char*) o->ptr; + strlen = sdslen(o->ptr); + } + + /* Parse start/end range if any. */ + if (c->argc == 4 || c->argc == 5) { + if (getLongFromObjectOrReply(c,c->argv[3],&start,NULL) != REDIS_OK) + return; + if (c->argc == 5) { + if (getLongFromObjectOrReply(c,c->argv[4],&end,NULL) != REDIS_OK) + return; + end_given = 1; + } else { + end = strlen-1; + } + /* Convert negative indexes */ + if (start < 0) start = strlen+start; + if (end < 0) end = strlen+end; + if (start < 0) start = 0; + if (end < 0) end = 0; + if (end >= strlen) end = strlen-1; + } else if (c->argc == 3) { + /* The whole string. */ + start = 0; + end = strlen-1; + } else { + /* Syntax error. */ + addReply(c,shared.syntaxerr); + return; + } + + /* For empty ranges (start > end) we return -1 as an empty range does + * not contain a 0 nor a 1. */ + if (start > end) { + addReplyLongLong(c, -1); + } else { + long bytes = end-start+1; + long pos = redisBitpos(p+start,bytes,bit); + + /* If we are looking for clear bits, and the user specified an exact + * range with start-end, we can't consider the right of the range as + * zero padded (as we do when no explicit end is given). + * + * So if redisBitpos() returns the first bit outside the range, + * we return -1 to the caller, to mean, in the specified range there + * is not a single "0" bit. */ + if (end_given && bit == 0 && pos == bytes*8) { + addReplyLongLong(c,-1); + return; + } + if (pos != -1) pos += start*8; /* Adjust for the bytes we skipped. */ + addReplyLongLong(c,pos); + } +} diff --git a/src/config.c b/src/config.c index f2818cff..d771b355 100644 --- a/src/config.c +++ b/src/config.c @@ -130,6 +130,11 @@ void loadServerConfigFromString(char *config) { if (server.port < 0 || server.port > 65535) { err = "Invalid port"; goto loaderr; } + } else if (!strcasecmp(argv[0],"tcp-backlog") && argc == 2) { + server.tcp_backlog = atoi(argv[1]); + if (server.tcp_backlog < 0) { + err = "Invalid backlog value"; goto loaderr; + } } else if (!strcasecmp(argv[0],"bind") && argc >= 2) { int j, addresses = argc-1; @@ -402,6 +407,8 @@ void loadServerConfigFromString(char *config) { server.zset_max_ziplist_entries = memtoll(argv[1], NULL); } else if (!strcasecmp(argv[0],"zset-max-ziplist-value") && argc == 2) { server.zset_max_ziplist_value = memtoll(argv[1], NULL); + } else if (!strcasecmp(argv[0],"hll-sparse-max-bytes") && argc == 2) { + server.hll_sparse_max_bytes = memtoll(argv[1], NULL); } else if (!strcasecmp(argv[0],"rename-command") && argc == 3) { struct redisCommand *cmd = lookupCommand(argv[1]); int retval; @@ -591,7 +598,7 @@ void configSetCommand(redisClient *c) { } else if (!strcasecmp(c->argv[2]->ptr,"maxclients")) { int orig_value = server.maxclients; - if (getLongLongFromObject(o,&ll) == REDIS_ERR || ll < 0) goto badfmt; + if (getLongLongFromObject(o,&ll) == REDIS_ERR || ll < 1) goto badfmt; /* Try to check if the OS is capable of supporting so many FDs. */ server.maxclients = ll; @@ -756,6 +763,9 @@ void configSetCommand(redisClient *c) { } else if (!strcasecmp(c->argv[2]->ptr,"zset-max-ziplist-value")) { if (getLongLongFromObject(o,&ll) == REDIS_ERR || ll < 0) goto badfmt; server.zset_max_ziplist_value = ll; + } else if (!strcasecmp(c->argv[2]->ptr,"hll-sparse-max-bytes")) { + if (getLongLongFromObject(o,&ll) == REDIS_ERR || ll < 0) goto badfmt; + server.hll_sparse_max_bytes = ll; } else if (!strcasecmp(c->argv[2]->ptr,"lua-time-limit")) { if (getLongLongFromObject(o,&ll) == REDIS_ERR || ll < 0) goto badfmt; server.lua_time_limit = ll; @@ -960,12 +970,15 @@ void configGetCommand(redisClient *c) { server.zset_max_ziplist_entries); config_get_numerical_field("zset-max-ziplist-value", server.zset_max_ziplist_value); + config_get_numerical_field("hll-sparse-max-bytes", + server.hll_sparse_max_bytes); config_get_numerical_field("lua-time-limit",server.lua_time_limit); config_get_numerical_field("slowlog-log-slower-than", server.slowlog_log_slower_than); config_get_numerical_field("slowlog-max-len", server.slowlog_max_len); config_get_numerical_field("port",server.port); + config_get_numerical_field("tcp-backlog",server.tcp_backlog); config_get_numerical_field("databases",server.dbnum); config_get_numerical_field("repl-ping-slave-period",server.repl_ping_slave_period); config_get_numerical_field("repl-timeout",server.repl_timeout); @@ -1446,7 +1459,7 @@ void rewriteConfigSaveOption(struct rewriteConfigState *state) { * resulting into no RDB persistence as expected. */ for (j = 0; j < server.saveparamslen; j++) { line = sdscatprintf(sdsempty(),"save %ld %d", - server.saveparams[j].seconds, server.saveparams[j].changes); + (long) server.saveparams[j].seconds, server.saveparams[j].changes); rewriteConfigRewriteLine(state,"save",line,1); } /* Mark "save" as processed in case server.saveparamslen is zero. */ @@ -1689,6 +1702,7 @@ int rewriteConfig(char *path) { rewriteConfigYesNoOption(state,"daemonize",server.daemonize,0); rewriteConfigStringOption(state,"pidfile",server.pidfile,REDIS_DEFAULT_PID_FILE); rewriteConfigNumericalOption(state,"port",server.port,REDIS_SERVERPORT); + rewriteConfigNumericalOption(state,"tcp-backlog",server.tcp_backlog,REDIS_TCP_BACKLOG); rewriteConfigBindOption(state); rewriteConfigStringOption(state,"unixsocket",server.unixsocket,NULL); rewriteConfigOctalOption(state,"unixsocketperm",server.unixsocketperm,REDIS_DEFAULT_UNIX_SOCKET_PERM); @@ -1758,6 +1772,7 @@ int rewriteConfig(char *path) { rewriteConfigNumericalOption(state,"set-max-intset-entries",server.set_max_intset_entries,REDIS_SET_MAX_INTSET_ENTRIES); rewriteConfigNumericalOption(state,"zset-max-ziplist-entries",server.zset_max_ziplist_entries,REDIS_ZSET_MAX_ZIPLIST_ENTRIES); rewriteConfigNumericalOption(state,"zset-max-ziplist-value",server.zset_max_ziplist_value,REDIS_ZSET_MAX_ZIPLIST_VALUE); + rewriteConfigNumericalOption(state,"hll-sparse-max-bytes",server.hll_sparse_max_bytes,REDIS_DEFAULT_HLL_SPARSE_MAX_BYTES); rewriteConfigYesNoOption(state,"activerehashing",server.activerehashing,REDIS_DEFAULT_ACTIVE_REHASHING); rewriteConfigClientoutputbufferlimitOption(state); rewriteConfigNumericalOption(state,"hz",server.hz,REDIS_DEFAULT_HZ); @@ -1792,14 +1807,7 @@ void configCommand(redisClient *c) { configGetCommand(c); } else if (!strcasecmp(c->argv[1]->ptr,"resetstat")) { if (c->argc != 2) goto badarity; - server.stat_keyspace_hits = 0; - server.stat_keyspace_misses = 0; - server.stat_numcommands = 0; - server.stat_numconnections = 0; - server.stat_expiredkeys = 0; - server.stat_rejected_conn = 0; - server.stat_fork_time = 0; - server.aof_delayed_fsync = 0; + resetServerStats(); resetCommandTableStats(); addReply(c,shared.ok); } else if (!strcasecmp(c->argv[1]->ptr,"rewrite")) { @@ -1809,8 +1817,10 @@ void configCommand(redisClient *c) { return; } if (rewriteConfig(server.configfile) == -1) { + redisLog(REDIS_WARNING,"CONFIG REWRITE failed: %s", strerror(errno)); addReplyErrorFormat(c,"Rewriting config file: %s", strerror(errno)); } else { + redisLog(REDIS_WARNING,"CONFIG REWRITE executed with success."); addReply(c,shared.ok); } } else { diff --git a/src/db.c b/src/db.c index d48904bc..3e25c112 100644 --- a/src/db.c +++ b/src/db.c @@ -166,6 +166,44 @@ int dbDelete(redisDb *db, robj *key) { } } +/* Prepare the string object stored at 'key' to be modified destructively + * to implement commands like SETBIT or APPEND. + * + * An object is usually ready to be modified unless one of the two conditions + * are true: + * + * 1) The object 'o' is shared (refcount > 1), we don't want to affect + * other users. + * 2) The object encoding is not "RAW". + * + * If the object is found in one of the above conditions (or both) by the + * function, an unshared / not-encoded copy of the string object is stored + * at 'key' in the specified 'db'. Otherwise the object 'o' itself is + * returned. + * + * USAGE: + * + * The object 'o' is what the caller already obtained by looking up 'key' + * in 'db', the usage pattern looks like this: + * + * o = lookupKeyWrite(db,key); + * if (checkType(c,o,REDIS_STRING)) return; + * o = dbUnshareStringValue(db,key,o); + * + * At this point the caller is ready to modify the object, for example + * using an sdscat() call to append some data, or anything else. + */ +robj *dbUnshareStringValue(redisDb *db, robj *key, robj *o) { + redisAssert(o->type == REDIS_STRING); + if (o->refcount != 1 || o->encoding != REDIS_ENCODING_RAW) { + robj *decoded = getDecodedObject(o); + o = createStringObject(decoded->ptr, sdslen(decoded->ptr)); + decrRefCount(decoded); + dbOverwrite(db,key,o); + } + return o; +} + long long emptyDb(void(callback)(void*)) { int j; long long removed = 0; @@ -599,11 +637,13 @@ void shutdownCommand(redisClient *c) { return; } } - /* SHUTDOWN can be called even while the server is in "loading" state. - * When this happens we need to make sure no attempt is performed to save + /* When SHUTDOWN is called while the server is loading a dataset in + * memory we need to make sure no attempt is performed to save * the dataset on shutdown (otherwise it could overwrite the current DB - * with half-read data). */ - if (server.loading) + * with half-read data). + * + * Also when in Sentinel mode clear the SAVE flag and force NOSAVE. */ + if (server.loading || server.sentinel_mode) flags = (flags & ~REDIS_SHUTDOWN_SAVE) | REDIS_SHUTDOWN_NOSAVE; if (prepareForShutdown(flags) == REDIS_OK) exit(0); addReplyError(c,"Errors trying to SHUTDOWN. Check logs."); @@ -759,13 +799,21 @@ void propagateExpire(redisDb *db, robj *key) { } int expireIfNeeded(redisDb *db, robj *key) { - long long when = getExpire(db,key); + mstime_t when = getExpire(db,key); + mstime_t now; if (when < 0) return 0; /* No expire for this key */ /* Don't expire anything while loading. It will be done later. */ if (server.loading) return 0; + /* If we are in the context of a Lua script, we claim that time is + * blocked to when the Lua script started. This way a key can expire + * only the first time it is accessed and not in the middle of the + * script execution, making propagation to slaves / AOF consistent. + * See issue #1525 on Github for more information. */ + now = server.lua_caller ? server.lua_time_start : mstime(); + /* If we are running in the context of a slave, return ASAP: * the slave key expiration is controlled by the master that will * send us synthesized DEL operations for expired keys. @@ -773,12 +821,10 @@ int expireIfNeeded(redisDb *db, robj *key) { * Still we try to return the right information to the caller, * that is, 0 if we think the key should be still valid, 1 if * we think the key is expired at this time. */ - if (server.masterhost != NULL) { - return mstime() > when; - } + if (server.masterhost != NULL) return now > when; /* Return when this key has not expired */ - if (mstime() <= when) return 0; + if (now <= when) return 0; /* Delete the key */ server.stat_expiredkeys++; diff --git a/src/debug.c b/src/debug.c index 6b3d30e9..17944afe 100644 --- a/src/debug.c +++ b/src/debug.c @@ -371,21 +371,13 @@ void debugCommand(redisClient *c) { { server.active_expire_enabled = atoi(c->argv[2]->ptr); addReply(c,shared.ok); - } else if (!strcasecmp(c->argv[1]->ptr,"set-active-expire") && - c->argc == 3) - { - server.active_expire_enabled = atoi(c->argv[2]->ptr); - addReply(c,shared.ok); -#ifdef _WIN32 - } else if (!strcasecmp(c->argv[1]->ptr,"flushload")) { - emptyDb(NULL); - if (rdbLoad(server.rdb_filename) != REDIS_OK) { - addReplyError(c,"Error trying to load the RDB dump"); - return; - } - redisLog(REDIS_WARNING,"DB reloaded by DEBUG flushload"); - addReply(c,shared.ok); -#endif + } else if (!strcasecmp(c->argv[1]->ptr,"error") && c->argc == 3) { + sds errstr = sdsnewlen("-",1); + + errstr = sdscatsds(errstr,c->argv[2]->ptr); + errstr = sdsmapchars(errstr,"\n\r"," ",2); /* no newlines in errors. */ + errstr = sdscatlen(errstr,"\r\n",2); + addReplySds(c,errstr); } else { addReplyErrorFormat(c, "Unknown DEBUG subcommand or wrong number of arguments for '%s'", (char*)c->argv[1]->ptr); diff --git a/src/dict.c b/src/dict.c index 4c0e1f34..8e93abbe 100644 --- a/src/dict.c +++ b/src/dict.c @@ -290,7 +290,7 @@ int dictExpand(dict *d, unsigned long size) /* Performs N steps of incremental rehashing. Returns 1 if there are still * keys to move from the old to the new hash table, otherwise 0 is returned. * Note that a rehashing step consists in moving a bucket (that may have more - * thank one key as we use chaining) from the old to the new hash table. */ + * than one key as we use chaining) from the old to the new hash table. */ int dictRehash(dict *d, int n) { if (!dictIsRehashing(d)) return 0; @@ -754,7 +754,7 @@ static unsigned long rev(unsigned long v) { * (where SIZE-1 is always the mask that is equivalent to taking the rest * of the division between the Hash of the key and SIZE). * - * For example if the current hash table size is 64, the mask is + * For example if the current hash table size is 16, the mask is * (in binary) 1111. The position of a key in the hash table will be always * the last four bits of the hash output, and so forth. * diff --git a/src/hyperloglog.c b/src/hyperloglog.c new file mode 100644 index 00000000..ed41d350 --- /dev/null +++ b/src/hyperloglog.c @@ -0,0 +1,1550 @@ +/* hyperloglog.c - Redis HyperLogLog probabilistic cardinality approximation. + * This file implements the algorithm and the exported Redis commands. + * + * Copyright (c) 2014, Salvatore Sanfilippo + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions are met: + * + * * Redistributions of source code must retain the above copyright notice, + * this list of conditions and the following disclaimer. + * * Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * * Neither the name of Redis nor the names of its contributors may be used + * to endorse or promote products derived from this software without + * specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" + * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE + * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR + * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF + * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS + * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN + * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) + * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE + * POSSIBILITY OF SUCH DAMAGE. + */ + +#include "redis.h" + +#include +#include + +/* The Redis HyperLogLog implementation is based on the following ideas: + * + * * The use of a 64 bit hash function as proposed in [1], in order to don't + * limited to cardinalities up to 10^9, at the cost of just 1 additional + * bit per register. + * * The use of 16384 6-bit registers for a great level of accuracy, using + * a total of 12k per key. + * * The use of the Redis string data type. No new type is introduced. + * * No attempt is made to compress the data structure as in [1]. Also the + * algorithm used is the original HyperLogLog Algorithm as in [2], with + * the only difference that a 64 bit hash function is used, so no correction + * is performed for values near 2^32 as in [1]. + * + * [1] Heule, Nunkesser, Hall: HyperLogLog in Practice: Algorithmic + * Engineering of a State of The Art Cardinality Estimation Algorithm. + * + * [2] P. Flajolet, Éric Fusy, O. Gandouet, and F. Meunier. Hyperloglog: The + * analysis of a near-optimal cardinality estimation algorithm. + * + * Redis uses two representations: + * + * 1) A "dense" representation where every entry is represented by + * a 6-bit integer. + * 2) A "sparse" representation using run length compression suitable + * for representing HyperLogLogs with many registers set to 0 in + * a memory efficient way. + * + * + * HLL header + * === + * + * Both the dense and sparse representation have a 16 byte header as follows: + * + * +------+---+-----+----------+ + * | HYLL | E | N/U | Cardin. | + * +------+---+-----+----------+ + * + * The first 4 bytes are a magic string set to the bytes "HYLL". + * "E" is one byte encoding, currently set to HLL_DENSE or + * HLL_SPARSE. N/U are three not used bytes. + * + * The "Cardin." field is a 64 bit integer stored in little endian format + * with the latest cardinality computed that can be reused if the data + * structure was not modified since the last computation (this is useful + * because there are high probabilities that HLLADD operations don't + * modify the actual data structure and hence the approximated cardinality). + * + * When the most significant bit in the most significant byte of the cached + * cardinality is set, it means that the data structure was modified and + * we can't reuse the cached value that must be recomputed. + * + * Dense representation + * === + * + * The dense representation used by Redis is the following: + * + * +--------+--------+--------+------// //--+ + * |11000000|22221111|33333322|55444444 .... | + * +--------+--------+--------+------// //--+ + * + * The 6 bits counters are encoded one after the other starting from the + * LSB to the MSB, and using the next bytes as needed. + * + * Sparse representation + * === + * + * The sparse representation encodes registers using a run length + * encoding composed of three opcodes, two using one byte, and one using + * of two bytes. The opcodes are called ZERO, XZERO and VAL. + * + * ZERO opcode is represented as 00xxxxxx. The 6-bit integer represented + * by the six bits 'xxxxxx', plus 1, means that there are N registers set + * to 0. This opcode can represent from 1 to 64 contiguous registers set + * to the value of 0. + * + * XZERO opcode is represented by two bytes 01xxxxxx yyyyyyyy. The 14-bit + * integer represented by the bits 'xxxxxx' as most significant bits and + * 'yyyyyyyy' as least significant bits, plus 1, means that there are N + * registers set to 0. This opcode can represent from 0 to 16384 contiguous + * registers set to the value of 0. + * + * VAL opcode is represented as 1vvvvvxx. It contains a 5-bit integer + * representing the value of a register, and a 2-bit integer representing + * the number of contiguous registers set to that value 'vvvvv'. + * To obtain the value and run length, the integers vvvvv and xx must be + * incremented by one. This opcode can represent values from 1 to 32, + * repeated from 1 to 4 times. + * + * The sparse representation can't represent registers with a value greater + * than 32, however it is very unlikely that we find such a register in an + * HLL with a cardinality where the sparse representation is still more + * memory efficient than the dense representation. When this happens the + * HLL is converted to the dense representation. + * + * The sparse representation is purely positional. For example a sparse + * representation of an empty HLL is just: XZERO:16384. + * + * An HLL having only 3 non-zero registers at position 1000, 1020, 1021 + * respectively set to 2, 3, 3, is represented by the following three + * opcodes: + * + * XZERO:1000 (Registers 0-999 are set to 0) + * VAL:2,1 (1 register set to value 2, that is register 1000) + * ZERO:19 (Registers 1001-1019 set to 0) + * VAL:3,2 (2 registers set to value 3, that is registers 1020,1021) + * XZERO:15362 (Registers 1022-16383 set to 0) + * + * In the example the sparse representation used just 7 bytes instead + * of 12k in order to represent the HLL registers. In general for low + * cardinality there is a big win in terms of space efficiency, traded + * with CPU time since the sparse representation is slower to access: + * + * The following table shows average cardinality vs bytes used, 100 + * samples per cardinality (when the set was not representable because + * of registers with too big value, the dense representation size was used + * as a sample). + * + * 100 267 + * 200 485 + * 300 678 + * 400 859 + * 500 1033 + * 600 1205 + * 700 1375 + * 800 1544 + * 900 1713 + * 1000 1882 + * 2000 3480 + * 3000 4879 + * 4000 6089 + * 5000 7138 + * 6000 8042 + * 7000 8823 + * 8000 9500 + * 9000 10088 + * 10000 10591 + * + * The dense representation uses 12288 bytes, so there is a big win up to + * a cardinality of ~2000-3000. For bigger cardinalities the constant times + * involved in updating the sparse representation is not justified by the + * memory savings. The exact maximum length of the sparse representation + * when this implementation switches to the dense representation is + * configured via the define server.hll_sparse_max_bytes. + */ + +struct hllhdr { + char magic[4]; /* "HYLL" */ + uint8_t encoding; /* HLL_DENSE or HLL_SPARSE. */ + uint8_t notused[3]; /* Reserved for future use, must be zero. */ + uint8_t card[8]; /* Cached cardinality, little endian. */ + uint8_t registers[]; /* Data bytes. */ +}; + +/* The cached cardinality MSB is used to signal validity of the cached value. */ +#define HLL_INVALIDATE_CACHE(hdr) (hdr)->card[0] |= (1<<7) +#define HLL_VALID_CACHE(hdr) (((hdr)->card[0] & (1<<7)) == 0) + +#define HLL_P 14 /* The greater is P, the smaller the error. */ +#define HLL_REGISTERS (1< 6 + * + * Right shift b0 of 'fb' bits. + * + * +--------+ + * |11000000| <- Initial value of b0 + * |00000011| <- After right shift of 6 pos. + * +--------+ + * + * Left shift b1 of bits 8-fb bits (2 bits) + * + * +--------+ + * |22221111| <- Initial value of b1 + * |22111100| <- After left shift of 2 bits. + * +--------+ + * + * OR the two bits, and finally AND with 111111 (63 in decimal) to + * clean the higher order bits we are not interested in: + * + * +--------+ + * |00000011| <- b0 right shifted + * |22111100| <- b1 left shifted + * |22111111| <- b0 OR b1 + * | 111111| <- (b0 OR b1) AND 63, our value. + * +--------+ + * + * We can try with a different example, like pos = 0. In this case + * the 6-bit counter is actually contained in a single byte. + * + * b0 = 6 * pos / 8 = 0 + * + * +--------+ + * |11000000| <- Our byte at b0 + * +--------+ + * + * fb = 6 * pos % 8 = 0 + * + * So we right shift of 0 bits (no shift in practice) and + * left shift the next byte of 8 bits, even if we don't use it, + * but this has the effect of clearing the bits so the result + * will not be affacted after the OR. + * + * ------------------------------------------------------------------------- + * + * Setting the register is a bit more complex, let's assume that 'val' + * is the value we want to set, already in the right range. + * + * We need two steps, in one we need to clear the bits, and in the other + * we need to bitwise-OR the new bits. + * + * Let's try with 'pos' = 1, so our first byte at 'b' is 0, + * + * "fb" is 6 in this case. + * + * +--------+ + * |11000000| <- Our byte at b0 + * +--------+ + * + * To create a AND-mask to clear the bits about this position, we just + * initialize the mask with the value 63, left shift it of "fs" bits, + * and finally invert the result. + * + * +--------+ + * |00111111| <- "mask" starts at 63 + * |11000000| <- "mask" after left shift of "ls" bits. + * |00111111| <- "mask" after invert. + * +--------+ + * + * Now we can bitwise-AND the byte at "b" with the mask, and bitwise-OR + * it with "val" left-shifted of "ls" bits to set the new bits. + * + * Now let's focus on the next byte b1: + * + * +--------+ + * |22221111| <- Initial value of b1 + * +--------+ + * + * To build the AND mask we start again with the 63 value, right shift + * it by 8-fb bits, and invert it. + * + * +--------+ + * |00111111| <- "mask" set at 2&6-1 + * |00001111| <- "mask" after the right shift by 8-fb = 2 bits + * |11110000| <- "mask" after bitwise not. + * +--------+ + * + * Now we can mask it with b+1 to clear the old bits, and bitwise-OR + * with "val" left-shifted by "rs" bits to set the new value. + */ + +/* Note: if we access the last counter, we will also access the b+1 byte + * that is out of the array, but sds strings always have an implicit null + * term, so the byte exists, and we can skip the conditional (or the need + * to allocate 1 byte more explicitly). */ + +/* Store the value of the register at position 'regnum' into variable 'target'. + * 'p' is an array of unsigned bytes. */ +#define HLL_DENSE_GET_REGISTER(target,p,regnum) do { \ + uint8_t *_p = (uint8_t*) p; \ + unsigned long _byte = regnum*HLL_BITS/8; \ + unsigned long _fb = regnum*HLL_BITS&7; \ + unsigned long _fb8 = 8 - _fb; \ + unsigned long b0 = _p[_byte]; \ + unsigned long b1 = _p[_byte+1]; \ + target = ((b0 >> _fb) | (b1 << _fb8)) & HLL_REGISTER_MAX; \ +} while(0) + +/* Set the value of the register at position 'regnum' to 'val'. + * 'p' is an array of unsigned bytes. */ +#define HLL_DENSE_SET_REGISTER(p,regnum,val) do { \ + uint8_t *_p = (uint8_t*) p; \ + unsigned long _byte = regnum*HLL_BITS/8; \ + unsigned long _fb = regnum*HLL_BITS&7; \ + unsigned long _fb8 = 8 - _fb; \ + unsigned long _v = val; \ + _p[_byte] &= ~(HLL_REGISTER_MAX << _fb); \ + _p[_byte] |= _v << _fb; \ + _p[_byte+1] &= ~(HLL_REGISTER_MAX >> _fb8); \ + _p[_byte+1] |= _v >> _fb8; \ +} while(0) + +/* Macros to access the sparse representation. + * The macros parameter is expected to be an uint8_t pointer. */ +#define HLL_SPARSE_XZERO_BIT 0x40 /* 01xxxxxx */ +#define HLL_SPARSE_VAL_BIT 0x80 /* 1vvvvvxx */ +#define HLL_SPARSE_IS_ZERO(p) (((*(p)) & 0xc0) == 0) /* 00xxxxxx */ +#define HLL_SPARSE_IS_XZERO(p) (((*(p)) & 0xc0) == HLL_SPARSE_XZERO_BIT) +#define HLL_SPARSE_IS_VAL(p) ((*(p)) & HLL_SPARSE_VAL_BIT) +#define HLL_SPARSE_ZERO_LEN(p) (((*(p)) & 0x3f)+1) +#define HLL_SPARSE_XZERO_LEN(p) (((((*(p)) & 0x3f) << 8) | (*((p)+1)))+1) +#define HLL_SPARSE_VAL_VALUE(p) ((((*(p)) >> 2) & 0x1f)+1) +#define HLL_SPARSE_VAL_LEN(p) (((*(p)) & 0x3)+1) +#define HLL_SPARSE_VAL_MAX_VALUE 32 +#define HLL_SPARSE_VAL_MAX_LEN 4 +#define HLL_SPARSE_ZERO_MAX_LEN 64 +#define HLL_SPARSE_XZERO_MAX_LEN 16384 +#define HLL_SPARSE_VAL_SET(p,val,len) do { \ + *(p) = (((val)-1)<<2|((len)-1))|HLL_SPARSE_VAL_BIT; \ +} while(0) +#define HLL_SPARSE_ZERO_SET(p,len) do { \ + *(p) = (len)-1; \ +} while(0) +#define HLL_SPARSE_XZERO_SET(p,len) do { \ + int _l = (len)-1; \ + *(p) = (_l>>8) | HLL_SPARSE_XZERO_BIT; \ + *((p)+1) = (_l&0xff); \ +} while(0) + +/* ========================= HyperLogLog algorithm ========================= */ + +/* Our hash function is MurmurHash2, 64 bit version. + * It was modified for Redis in order to provide the same result in + * big and little endian archs (endian neutral). */ +uint64_t MurmurHash64A (const void * key, int len, unsigned int seed) { + const uint64_t m = 0xc6a4a7935bd1e995; + const int r = 47; + uint64_t h = seed ^ (len * m); + const uint8_t *data = (const uint8_t *)key; + const uint8_t *end = data + (len-(len&7)); + + while(data != end) { + uint64_t k; + +#if (BYTE_ORDER == LITTLE_ENDIAN) + k = *((uint64_t*)data); +#else + k = (uint64_t) data[0]; + k |= (uint64_t) data[1] << 8; + k |= (uint64_t) data[2] << 16; + k |= (uint64_t) data[3] << 24; + k |= (uint64_t) data[4] << 32; + k |= (uint64_t) data[5] << 40; + k |= (uint64_t) data[6] << 48; + k |= (uint64_t) data[7] << 56; +#endif + + k *= m; + k ^= k >> r; + k *= m; + h ^= k; + h *= m; + data += 8; + } + + switch(len & 7) { + case 7: h ^= (uint64_t)data[6] << 48; + case 6: h ^= (uint64_t)data[5] << 40; + case 5: h ^= (uint64_t)data[4] << 32; + case 4: h ^= (uint64_t)data[3] << 24; + case 3: h ^= (uint64_t)data[2] << 16; + case 2: h ^= (uint64_t)data[1] << 8; + case 1: h ^= (uint64_t)data[0]; + h *= m; + }; + + h ^= h >> r; + h *= m; + h ^= h >> r; + return h; +} + +/* Given a string element to add to the HyperLogLog, returns the length + * of the pattern 000..1 of the element hash. As a side effect 'regp' is + * set to the register index this element hashes to. */ +int hllPatLen(unsigned char *ele, size_t elesize, long *regp) { + uint64_t hash, bit, index; + int count; + + /* Count the number of zeroes starting from bit HLL_REGISTERS + * (that is a power of two corresponding to the first bit we don't use + * as index). The max run can be 64-P+1 bits. + * + * Note that the final "1" ending the sequence of zeroes must be + * included in the count, so if we find "001" the count is 3, and + * the smallest count possible is no zeroes at all, just a 1 bit + * at the first position, that is a count of 1. + * + * This may sound like inefficient, but actually in the average case + * there are high probabilities to find a 1 after a few iterations. */ + hash = MurmurHash64A(ele,(int)elesize,0xadc83b19ULL); + index = hash & HLL_P_MASK; /* Register index. */ + hash |= ((uint64_t)1<<63); /* Make sure the loop terminates. */ + bit = HLL_REGISTERS; /* First bit not used to address the register. */ + count = 1; /* Initialized to 1 since we count the "00000...1" pattern. */ + while((hash & bit) == 0) { + count++; + bit <<= 1; + } + *regp = (int) index; + return count; +} + +/* ================== Dense representation implementation ================== */ + +/* "Add" the element in the dense hyperloglog data structure. + * Actually nothing is added, but the max 0 pattern counter of the subset + * the element belongs to is incremented if needed. + * + * 'registers' is expected to have room for HLL_REGISTERS plus an + * additional byte on the right. This requirement is met by sds strings + * automatically since they are implicitly null terminated. + * + * The function always succeed, however if as a result of the operation + * the approximated cardinality changed, 1 is returned. Otherwise 0 + * is returned. */ +int hllDenseAdd(uint8_t *registers, unsigned char *ele, size_t elesize) { + uint8_t oldcount, count; + long index; + + /* Update the register if this element produced a longer run of zeroes. */ + count = hllPatLen(ele,elesize,&index); + HLL_DENSE_GET_REGISTER(oldcount,registers,index); + if (count > oldcount) { + HLL_DENSE_SET_REGISTER(registers,index,count); + return 1; + } else { + return 0; + } +} + +/* Compute SUM(2^-reg) in the dense representation. + * PE is an array with a pre-computer table of values 2^-reg indexed by reg. + * As a side effect the integer pointed by 'ezp' is set to the number + * of zero registers. */ +double hllDenseSum(uint8_t *registers, double *PE, int *ezp) { + double E = 0; + int j, ez = 0; + + /* Redis default is to use 16384 registers 6 bits each. The code works + * with other values by modifying the defines, but for our target value + * we take a faster path with unrolled loops. */ + if (HLL_REGISTERS == 16384 && HLL_BITS == 6) { + uint8_t *r = registers; + unsigned long r0, r1, r2, r3, r4, r5, r6, r7, r8, r9, + r10, r11, r12, r13, r14, r15; + for (j = 0; j < 1024; j++) { + /* Handle 16 registers per iteration. */ + r0 = r[0] & 63; if (r0 == 0) ez++; + r1 = (r[0] >> 6 | r[1] << 2) & 63; if (r1 == 0) ez++; + r2 = (r[1] >> 4 | r[2] << 4) & 63; if (r2 == 0) ez++; + r3 = (r[2] >> 2) & 63; if (r3 == 0) ez++; + r4 = r[3] & 63; if (r4 == 0) ez++; + r5 = (r[3] >> 6 | r[4] << 2) & 63; if (r5 == 0) ez++; + r6 = (r[4] >> 4 | r[5] << 4) & 63; if (r6 == 0) ez++; + r7 = (r[5] >> 2) & 63; if (r7 == 0) ez++; + r8 = r[6] & 63; if (r8 == 0) ez++; + r9 = (r[6] >> 6 | r[7] << 2) & 63; if (r9 == 0) ez++; + r10 = (r[7] >> 4 | r[8] << 4) & 63; if (r10 == 0) ez++; + r11 = (r[8] >> 2) & 63; if (r11 == 0) ez++; + r12 = r[9] & 63; if (r12 == 0) ez++; + r13 = (r[9] >> 6 | r[10] << 2) & 63; if (r13 == 0) ez++; + r14 = (r[10] >> 4 | r[11] << 4) & 63; if (r14 == 0) ez++; + r15 = (r[11] >> 2) & 63; if (r15 == 0) ez++; + + /* Additional parens will allow the compiler to optimize the + * code more with a loss of precision that is not very relevant + * here (floating point math is not commutative!). */ + E += (PE[r0] + PE[r1]) + (PE[r2] + PE[r3]) + (PE[r4] + PE[r5]) + + (PE[r6] + PE[r7]) + (PE[r8] + PE[r9]) + (PE[r10] + PE[r11]) + + (PE[r12] + PE[r13]) + (PE[r14] + PE[r15]); + r += 12; + } + } else { + for (j = 0; j < HLL_REGISTERS; j++) { + unsigned long reg; + + HLL_DENSE_GET_REGISTER(reg,registers,j); + if (reg == 0) { + ez++; + /* Increment E at the end of the loop. */ + } else { + E += PE[reg]; /* Precomputed 2^(-reg[j]). */ + } + } + E += ez; /* Add 2^0 'ez' times. */ + } + *ezp = ez; + return E; +} + +/* ================== Sparse representation implementation ================= */ + +/* Convert the HLL with sparse representation given as input in its dense + * representation. Both representations are represented by SDS strings, and + * the input representation is freed as a side effect. + * + * The function returns REDIS_OK if the sparse representation was valid, + * otherwise REDIS_ERR is returned if the representation was corrupted. */ +int hllSparseToDense(robj *o) { + sds sparse = o->ptr, dense; + struct hllhdr *hdr, *oldhdr = (struct hllhdr*)sparse; + int idx = 0, runlen, regval; + uint8_t *p = (uint8_t*)sparse, *end = p+sdslen(sparse); + + /* If the representation is already the right one return ASAP. */ + hdr = (struct hllhdr*) sparse; + if (hdr->encoding == HLL_DENSE) return REDIS_OK; + + /* Create a string of the right size filled with zero bytes. + * Note that the cached cardinality is set to 0 as a side effect + * that is exactly the cardinality of an empty HLL. */ + dense = sdsnewlen(NULL,HLL_DENSE_SIZE); + hdr = (struct hllhdr*) dense; + *hdr = *oldhdr; /* This will copy the magic and cached cardinality. */ + hdr->encoding = HLL_DENSE; + + /* Now read the sparse representation and set non-zero registers + * accordingly. */ + p += HLL_HDR_SIZE; + while(p < end) { + if (HLL_SPARSE_IS_ZERO(p)) { + runlen = HLL_SPARSE_ZERO_LEN(p); + idx += runlen; + p++; + } else if (HLL_SPARSE_IS_XZERO(p)) { + runlen = HLL_SPARSE_XZERO_LEN(p); + idx += runlen; + p += 2; + } else { + runlen = HLL_SPARSE_VAL_LEN(p); + regval = HLL_SPARSE_VAL_VALUE(p); + while(runlen--) { + HLL_DENSE_SET_REGISTER(hdr->registers,idx,regval); + idx++; + } + p++; + } + } + + /* If the sparse representation was valid, we expect to find idx + * set to HLL_REGISTERS. */ + if (idx != HLL_REGISTERS) { + sdsfree(dense); + return REDIS_ERR; + } + + /* Free the old representation and set the new one. */ + sdsfree(o->ptr); + o->ptr = dense; + return REDIS_OK; +} + +/* "Add" the element in the sparse hyperloglog data structure. + * Actually nothing is added, but the max 0 pattern counter of the subset + * the element belongs to is incremented if needed. + * + * The object 'o' is the String object holding the HLL. The function requires + * a reference to the object in order to be able to enlarge the string if + * needed. + * + * On success, the function returns 1 if the cardinality changed, or 0 + * if the register for this element was not updated. + * On error (if the representation is invalid) -1 is returned. + * + * As a side effect the function may promote the HLL representation from + * sparse to dense: this happens when a register requires to be set to a value + * not representable with the sparse representation, or when the resulting + * size would be greater than server.hll_sparse_max_bytes. */ +int hllSparseAdd(robj *o, unsigned char *ele, size_t elesize) { + struct hllhdr *hdr; + uint8_t oldcount, count, *sparse, *end, *p, *prev, *next; + long index, first, span; + long is_zero = 0, is_xzero = 0, is_val = 0, runlen = 0; + + /* Update the register if this element produced a longer run of zeroes. */ + count = hllPatLen(ele,elesize,&index); + + /* If the count is too big to be representable by the sparse representation + * switch to dense representation. */ + if (count > HLL_SPARSE_VAL_MAX_VALUE) goto promote; + + /* When updating a sparse representation, sometimes we may need to + * enlarge the buffer for up to 3 bytes in the worst case (XZERO split + * into XZERO-VAL-XZERO). Make sure there is enough space right now + * so that the pointers we take during the execution of the function + * will be valid all the time. */ + o->ptr = sdsMakeRoomFor(o->ptr,3); + + /* Step 1: we need to locate the opcode we need to modify to check + * if a value update is actually needed. */ + sparse = p = ((uint8_t*)o->ptr) + HLL_HDR_SIZE; + end = p + sdslen(o->ptr) - HLL_HDR_SIZE; + + first = 0; + prev = NULL; /* Points to previos opcode at the end of the loop. */ + next = NULL; /* Points to the next opcode at the end of the loop. */ + span = 0; + while(p < end) { + long oplen; + + /* Set span to the number of registers covered by this opcode. + * + * This is the most performance critical loop of the sparse + * representation. Sorting the conditionals from the most to the + * least frequent opcode in many-bytes sparse HLLs is faster. */ + oplen = 1; + if (HLL_SPARSE_IS_ZERO(p)) { + span = HLL_SPARSE_ZERO_LEN(p); + } else if (HLL_SPARSE_IS_VAL(p)) { + span = HLL_SPARSE_VAL_LEN(p); + } else { /* XZERO. */ + span = HLL_SPARSE_XZERO_LEN(p); + oplen = 2; + } + /* Break if this opcode covers the register as 'index'. */ + if (index <= first+span-1) break; + prev = p; + p += oplen; + first += span; + } + if (span == 0) return -1; /* Invalid format. */ + + next = HLL_SPARSE_IS_XZERO(p) ? p+2 : p+1; + if (next >= end) next = NULL; + + /* Cache current opcode type to avoid using the macro again and + * again for something that will not change. + * Also cache the run-length of the opcode. */ + if (HLL_SPARSE_IS_ZERO(p)) { + is_zero = 1; + runlen = HLL_SPARSE_ZERO_LEN(p); + } else if (HLL_SPARSE_IS_XZERO(p)) { + is_xzero = 1; + runlen = HLL_SPARSE_XZERO_LEN(p); + } else { + is_val = 1; + runlen = HLL_SPARSE_VAL_LEN(p); + } + + /* Step 2: After the loop: + * + * 'first' stores to the index of the first register covered + * by the current opcode, which is pointed by 'p'. + * + * 'next' ad 'prev' store respectively the next and previous opcode, + * or NULL if the opcode at 'p' is respectively the last or first. + * + * 'span' is set to the number of registers covered by the current + * opcode. + * + * There are different cases in order to update the data structure + * in place without generating it from scratch: + * + * A) If it is a VAL opcode already set to a value >= our 'count' + * no update is needed, regardless of the VAL run-length field. + * In this case PFADD returns 0 since no changes are performed. + * + * B) If it is a VAL opcode with len = 1 (representing only our + * register) and the value is less than 'count', we just update it + * since this is a trivial case. */ + if (is_val) { + oldcount = HLL_SPARSE_VAL_VALUE(p); + /* Case A. */ + if (oldcount >= count) return 0; + + /* Case B. */ + if (runlen == 1) { + HLL_SPARSE_VAL_SET(p,count,1); + goto updated; + } + } + + /* C) Another trivial to handle case is a ZERO opcode with a len of 1. + * We can just replace it with a VAL opcode with our value and len of 1. */ + if (is_zero && runlen == 1) { + HLL_SPARSE_VAL_SET(p,count,1); + goto updated; + } + + /* D) General case. + * + * The other cases are more complex: our register requires to be updated + * and is either currently represented by a VAL opcode with len > 1, + * by a ZERO opcode with len > 1, or by an XZERO opcode. + * + * In those cases the original opcode must be split into muliple + * opcodes. The worst case is an XZERO split in the middle resuling into + * XZERO - VAL - XZERO, so the resulting sequence max length is + * 5 bytes. + * + * We perform the split writing the new sequence into the 'new' buffer + * with 'newlen' as length. Later the new sequence is inserted in place + * of the old one, possibly moving what is on the right a few bytes + * if the new sequence is longer than the older one. */ + uint8_t seq[5], *n = seq; + int last = first+span-1; /* Last register covered by the sequence. */ + int len; + + if (is_zero || is_xzero) { + /* Handle splitting of ZERO / XZERO. */ + if (index != first) { + len = index-first; + if (len > HLL_SPARSE_ZERO_MAX_LEN) { + HLL_SPARSE_XZERO_SET(n,len); + n += 2; + } else { + HLL_SPARSE_ZERO_SET(n,len); + n++; + } + } + HLL_SPARSE_VAL_SET(n,count,1); + n++; + if (index != last) { + len = last-index; + if (len > HLL_SPARSE_ZERO_MAX_LEN) { + HLL_SPARSE_XZERO_SET(n,len); + n += 2; + } else { + HLL_SPARSE_ZERO_SET(n,len); + n++; + } + } + } else { + /* Handle splitting of VAL. */ + int curval = HLL_SPARSE_VAL_VALUE(p); + + if (index != first) { + len = index-first; + HLL_SPARSE_VAL_SET(n,curval,len); + n++; + } + HLL_SPARSE_VAL_SET(n,count,1); + n++; + if (index != last) { + len = last-index; + HLL_SPARSE_VAL_SET(n,curval,len); + n++; + } + } + + /* Step 3: substitute the new sequence with the old one. + * + * Note that we already allocated space on the sds string + * calling sdsMakeRoomFor(). */ + int seqlen = n-seq; + int oldlen = is_xzero ? 2 : 1; + int deltalen = seqlen-oldlen; + + if (deltalen > 0 && + sdslen(o->ptr)+deltalen > server.hll_sparse_max_bytes) goto promote; + if (deltalen && next) memmove(next+deltalen,next,end-next); + sdsIncrLen(o->ptr,deltalen); + memcpy(p,seq,seqlen); + end += deltalen; + +updated: + /* Step 4: Merge adjacent values if possible. + * + * The representation was updated, however the resulting representation + * may not be optimal: adjacent VAL opcodes can sometimes be merged into + * a single one. */ + p = prev ? prev : sparse; + int scanlen = 5; /* Scan up to 5 upcodes starting from prev. */ + while (p < end && scanlen--) { + if (HLL_SPARSE_IS_XZERO(p)) { + p += 2; + continue; + } else if (HLL_SPARSE_IS_ZERO(p)) { + p++; + continue; + } + /* We need two adjacent VAL opcodes to try a merge, having + * the same value, and a len that fits the VAL opcode max len. */ + if (p+1 < end && HLL_SPARSE_IS_VAL(p+1)) { + int v1 = HLL_SPARSE_VAL_VALUE(p); + int v2 = HLL_SPARSE_VAL_VALUE(p+1); + if (v1 == v2) { + int len = HLL_SPARSE_VAL_LEN(p)+HLL_SPARSE_VAL_LEN(p+1); + if (len <= HLL_SPARSE_VAL_MAX_LEN) { + HLL_SPARSE_VAL_SET(p+1,v1,len); + memmove(p,p+1,end-p); + sdsIncrLen(o->ptr,-1); + end--; + /* After a merge we reiterate without incrementing 'p' + * in order to try to merge the just merged value with + * a value on its right. */ + continue; + } + } + } + p++; + } + + /* Invalidate the cached cardinality. */ + hdr = o->ptr; + HLL_INVALIDATE_CACHE(hdr); + return 1; + +promote: /* Promote to dense representation. */ + if (hllSparseToDense(o) == REDIS_ERR) return -1; /* Corrupted HLL. */ + hdr = o->ptr; + + /* We need to call hllDenseAdd() to perform the operation after the + * conversion. However the result must be 1, since if we need to + * convert from sparse to dense a register requires to be updated. + * + * Note that this in turn means that PFADD will make sure the command + * is propagated to slaves / AOF, so if there is a sparse -> dense + * convertion, it will be performed in all the slaves as well. */ + int dense_retval = hllDenseAdd(hdr->registers, ele, elesize); + redisAssert(dense_retval == 1); + return dense_retval; +} + +/* Compute SUM(2^-reg) in the sparse representation. + * PE is an array with a pre-computer table of values 2^-reg indexed by reg. + * As a side effect the integer pointed by 'ezp' is set to the number + * of zero registers. */ +double hllSparseSum(uint8_t *sparse, int sparselen, double *PE, int *ezp, int *invalid) { + double E = 0; + int ez = 0, idx = 0, runlen, regval; + uint8_t *end = sparse+sparselen, *p = sparse; + + while(p < end) { + if (HLL_SPARSE_IS_ZERO(p)) { + runlen = HLL_SPARSE_ZERO_LEN(p); + idx += runlen; + ez += runlen; + /* Increment E at the end of the loop. */ + p++; + } else if (HLL_SPARSE_IS_XZERO(p)) { + runlen = HLL_SPARSE_XZERO_LEN(p); + idx += runlen; + ez += runlen; + /* Increment E at the end of the loop. */ + p += 2; + } else { + runlen = HLL_SPARSE_VAL_LEN(p); + regval = HLL_SPARSE_VAL_VALUE(p); + idx += runlen; + E += PE[regval]*runlen; + p++; + } + } + if (idx != HLL_REGISTERS && invalid) *invalid = 1; + E += ez; /* Add 2^0 'ez' times. */ + *ezp = ez; + return E; +} + +/* ========================= HyperLogLog Count ============================== + * This is the core of the algorithm where the approximated count is computed. + * The function uses the lower level hllDenseSum() and hllSparseSum() functions + * as helpers to compute the SUM(2^-reg) part of the computation, which is + * representation-specific, while all the rest is common. */ + +/* Implements the SUM operation for uint8_t data type which is only used + * internally as speedup for PFCOUNT with multiple keys. */ +double hllRawSum(uint8_t *registers, double *PE, int *ezp) { + double E = 0; + int j, ez = 0; + uint64_t *word = (uint64_t*) registers; + uint8_t *bytes; + + for (j = 0; j < HLL_REGISTERS/8; j++) { + if (*word == 0) { + ez += 8; + } else { + bytes = (uint8_t*) word; + if (bytes[0]) E += PE[bytes[0]]; else ez++; + if (bytes[1]) E += PE[bytes[1]]; else ez++; + if (bytes[2]) E += PE[bytes[2]]; else ez++; + if (bytes[3]) E += PE[bytes[3]]; else ez++; + if (bytes[4]) E += PE[bytes[4]]; else ez++; + if (bytes[5]) E += PE[bytes[5]]; else ez++; + if (bytes[6]) E += PE[bytes[6]]; else ez++; + if (bytes[7]) E += PE[bytes[7]]; else ez++; + } + word++; + } + E += ez; /* 2^(-reg[j]) is 1 when m is 0, add it 'ez' times for every + zero register in the HLL. */ + *ezp = ez; + return E; +} + +/* Return the approximated cardinality of the set based on the armonic + * mean of the registers values. 'hdr' points to the start of the SDS + * representing the String object holding the HLL representation. + * + * If the sparse representation of the HLL object is not valid, the integer + * pointed by 'invalid' is set to non-zero, otherwise it is left untouched. + * + * hllCount() supports a special internal-only encoding of HLL_RAW, that + * is, hdr->registers will point to an uint8_t array of HLL_REGISTERS element. + * This is useful in order to speedup PFCOUNT when called against multiple + * keys (no need to work with 6-bit integers encoding). */ +uint64_t hllCount(struct hllhdr *hdr, int *invalid) { + double m = HLL_REGISTERS; + double E, alpha = 0.7213/(1+1.079/m); + int j, ez; /* Number of registers equal to 0. */ + + /* We precompute 2^(-reg[j]) in a small table in order to + * speedup the computation of SUM(2^-register[0..i]). */ + static int initialized = 0; + static double PE[64]; + if (!initialized) { + PE[0] = 1; /* 2^(-reg[j]) is 1 when m is 0. */ + for (j = 1; j < 64; j++) { + /* 2^(-reg[j]) is the same as 1/2^reg[j]. */ + PE[j] = 1.0/(1ULL << j); + } + initialized = 1; + } + + /* Compute SUM(2^-register[0..i]). */ + if (hdr->encoding == HLL_DENSE) { + E = hllDenseSum(hdr->registers,PE,&ez); + } else if (hdr->encoding == HLL_SPARSE) { + E = hllSparseSum(hdr->registers, + (int)sdslen((sds)hdr)-HLL_HDR_SIZE,PE,&ez,invalid); + } else if (hdr->encoding == HLL_RAW) { + E = hllRawSum(hdr->registers,PE,&ez); + } else { + redisPanic("Unknown HyperLogLog encoding in hllCount()"); + } + + /* Muliply the inverse of E for alpha_m * m^2 to have the raw estimate. */ + E = (1/E)*alpha*m*m; + + /* Use the LINEARCOUNTING algorithm for small cardinalities. + * For larger values but up to 72000 HyperLogLog raw approximation is + * used since linear counting error starts to increase. However HyperLogLog + * shows a strong bias in the range 2.5*16384 - 72000, so we try to + * compensate for it. */ + if (E < m*2.5 && ez != 0) { + E = m*log(m/ez); /* LINEARCOUNTING() */ + } else if (m == 16384 && E < 72000) { + /* We did polynomial regression of the bias for this range, this + * way we can compute the bias for a given cardinality and correct + * according to it. Only apply the correction for P=14 that's what + * we use and the value the correction was verified with. */ + double bias = 5.9119*1.0e-18*(E*E*E*E) + -1.4253*1.0e-12*(E*E*E)+ + 1.2940*1.0e-7*(E*E) + -5.2921*1.0e-3*E+ + 83.3216; + E -= E*(bias/100); + } + /* We don't apply the correction for E > 1/30 of 2^32 since we use + * a 64 bit function and 6 bit counters. To apply the correction for + * 1/30 of 2^64 is not needed since it would require a huge set + * to approach such a value. */ + return (uint64_t) E; +} + +/* Call hllDenseAdd() or hllSparseAdd() according to the HLL encoding. */ +int hllAdd(robj *o, unsigned char *ele, size_t elesize) { + struct hllhdr *hdr = o->ptr; + switch(hdr->encoding) { + case HLL_DENSE: return hllDenseAdd(hdr->registers,ele,elesize); + case HLL_SPARSE: return hllSparseAdd(o,ele,elesize); + default: return -1; /* Invalid representation. */ + } +} + +/* Merge by computing MAX(registers[i],hll[i]) the HyperLogLog 'hll' + * with an array of uint8_t HLL_REGISTERS registers pointed by 'max'. + * + * The hll object must be already validated via isHLLObjectOrReply() + * or in some other way. + * + * If the HyperLogLog is sparse and is found to be invalid, REDIS_ERR + * is returned, otherwise the function always succeeds. */ +int hllMerge(uint8_t *max, robj *hll) { + struct hllhdr *hdr = hll->ptr; + int i; + + if (hdr->encoding == HLL_DENSE) { + uint8_t val; + + for (i = 0; i < HLL_REGISTERS; i++) { + HLL_DENSE_GET_REGISTER(val,hdr->registers,i); + if (val > max[i]) max[i] = val; + } + } else { + uint8_t *p = hll->ptr, *end = p + sdslen(hll->ptr); + long runlen, regval; + + p += HLL_HDR_SIZE; + i = 0; + while(p < end) { + if (HLL_SPARSE_IS_ZERO(p)) { + runlen = HLL_SPARSE_ZERO_LEN(p); + i += runlen; + p++; + } else if (HLL_SPARSE_IS_XZERO(p)) { + runlen = HLL_SPARSE_XZERO_LEN(p); + i += runlen; + p += 2; + } else { + runlen = HLL_SPARSE_VAL_LEN(p); + regval = HLL_SPARSE_VAL_VALUE(p); + while(runlen--) { + if (regval > max[i]) max[i] = (uint8_t)regval; + i++; + } + p++; + } + } + if (i != HLL_REGISTERS) return REDIS_ERR; + } + return REDIS_OK; +} + +/* ========================== HyperLogLog commands ========================== */ + +/* Create an HLL object. We always create the HLL using sparse encoding. + * This will be upgraded to the dense representation as needed. */ +robj *createHLLObject(void) { + robj *o; + struct hllhdr *hdr; + sds s; + uint8_t *p; + int sparselen = HLL_HDR_SIZE + + (((HLL_REGISTERS+(HLL_SPARSE_XZERO_MAX_LEN-1)) / + HLL_SPARSE_XZERO_MAX_LEN)*2); + int aux; + + /* Populate the sparse representation with as many XZERO opcodes as + * needed to represent all the registers. */ + aux = HLL_REGISTERS; + s = sdsnewlen(NULL,sparselen); + p = (uint8_t*)s + HLL_HDR_SIZE; + while(aux) { + int xzero = HLL_SPARSE_XZERO_MAX_LEN; + if (xzero > aux) xzero = aux; + HLL_SPARSE_XZERO_SET(p,xzero); + p += 2; + aux -= xzero; + } + redisAssert((p-(uint8_t*)s) == sparselen); + + /* Create the actual object. */ + o = createObject(REDIS_STRING,s); + hdr = o->ptr; + memcpy(hdr->magic,"HYLL",4); + hdr->encoding = HLL_SPARSE; + return o; +} + +/* Check if the object is a String with a valid HLL representation. + * Return REDIS_OK if this is true, otherwise reply to the client + * with an error and return REDIS_ERR. */ +int isHLLObjectOrReply(redisClient *c, robj *o) { + struct hllhdr *hdr; + + /* Key exists, check type */ + if (checkType(c,o,REDIS_STRING)) + return REDIS_ERR; /* Error already sent. */ + + if (stringObjectLen(o) < sizeof(*hdr)) goto invalid; + hdr = o->ptr; + + /* Magic should be "HYLL". */ + if (hdr->magic[0] != 'H' || hdr->magic[1] != 'Y' || + hdr->magic[2] != 'L' || hdr->magic[3] != 'L') goto invalid; + + if (hdr->encoding > HLL_MAX_ENCODING) goto invalid; + + /* Dense representation string length should match exactly. */ + if (hdr->encoding == HLL_DENSE && + stringObjectLen(o) != HLL_DENSE_SIZE) goto invalid; + + /* All tests passed. */ + return REDIS_OK; + +invalid: + addReplySds(c, + sdsnew("-WRONGTYPE Key is not a valid " + "HyperLogLog string value.\r\n")); + return REDIS_ERR; +} + +/* PFADD var ele ele ele ... ele => :0 or :1 */ +void pfaddCommand(redisClient *c) { + robj *o = lookupKeyWrite(c->db,c->argv[1]); + struct hllhdr *hdr; + int updated = 0, j; + + if (o == NULL) { + /* Create the key with a string value of the exact length to + * hold our HLL data structure. sdsnewlen() when NULL is passed + * is guaranteed to return bytes initialized to zero. */ + o = createHLLObject(); + dbAdd(c->db,c->argv[1],o); + updated++; + } else { + if (isHLLObjectOrReply(c,o) != REDIS_OK) return; + o = dbUnshareStringValue(c->db,c->argv[1],o); + } + /* Perform the low level ADD operation for every element. */ + for (j = 2; j < c->argc; j++) { + int retval = hllAdd(o, (unsigned char*)c->argv[j]->ptr, + sdslen(c->argv[j]->ptr)); + switch(retval) { + case 1: + updated++; + break; + case -1: + addReplySds(c,sdsnew(invalid_hll_err)); + return; + } + } + hdr = o->ptr; + if (updated) { + signalModifiedKey(c->db,c->argv[1]); + notifyKeyspaceEvent(REDIS_NOTIFY_STRING,"pfadd",c->argv[1],c->db->id); + server.dirty++; + HLL_INVALIDATE_CACHE(hdr); + } + addReply(c, updated ? shared.cone : shared.czero); +} + +/* PFCOUNT var -> approximated cardinality of set. */ +void pfcountCommand(redisClient *c) { + robj *o; + struct hllhdr *hdr; + uint64_t card; + + /* Case 1: multi-key keys, cardinality of the union. + * + * When multiple keys are specified, PFCOUNT actually computes + * the cardinality of the merge of the N HLLs specified. */ + if (c->argc > 2) { + uint8_t max[HLL_HDR_SIZE+HLL_REGISTERS], *registers; + int j; + + /* Compute an HLL with M[i] = MAX(M[i]_j). */ + memset(max,0,sizeof(max)); + hdr = (struct hllhdr*) max; + hdr->encoding = HLL_RAW; /* Special internal-only encoding. */ + registers = max + HLL_HDR_SIZE; + for (j = 1; j < c->argc; j++) { + /* Check type and size. */ + robj *o = lookupKeyRead(c->db,c->argv[j]); + if (o == NULL) continue; /* Assume empty HLL for non existing var. */ + if (isHLLObjectOrReply(c,o) != REDIS_OK) return; + + /* Merge with this HLL with our 'max' HHL by setting max[i] + * to MAX(max[i],hll[i]). */ + if (hllMerge(registers,o) == REDIS_ERR) { + addReplySds(c,sdsnew(invalid_hll_err)); + return; + } + } + + /* Compute cardinality of the resulting set. */ + addReplyLongLong(c,hllCount(hdr,NULL)); + return; + } + + /* Case 2: cardinality of the single HLL. + * + * The user specified a single key. Either return the cached value + * or compute one and update the cache. */ + o = lookupKeyRead(c->db,c->argv[1]); + if (o == NULL) { + /* No key? Cardinality is zero since no element was added, otherwise + * we would have a key as HLLADD creates it as a side effect. */ + addReply(c,shared.czero); + } else { + if (isHLLObjectOrReply(c,o) != REDIS_OK) return; + o = dbUnshareStringValue(c->db,c->argv[1],o); + + /* Check if the cached cardinality is valid. */ + hdr = o->ptr; + if (HLL_VALID_CACHE(hdr)) { + /* Just return the cached value. */ + card = (uint64_t)hdr->card[0]; + card |= (uint64_t)hdr->card[1] << 8; + card |= (uint64_t)hdr->card[2] << 16; + card |= (uint64_t)hdr->card[3] << 24; + card |= (uint64_t)hdr->card[4] << 32; + card |= (uint64_t)hdr->card[5] << 40; + card |= (uint64_t)hdr->card[6] << 48; + card |= (uint64_t)hdr->card[7] << 56; + } else { + int invalid = 0; + /* Recompute it and update the cached value. */ + card = hllCount(hdr,&invalid); + if (invalid) { + addReplySds(c,sdsnew(invalid_hll_err)); + return; + } + hdr->card[0] = card & 0xff; + hdr->card[1] = (card >> 8) & 0xff; + hdr->card[2] = (card >> 16) & 0xff; + hdr->card[3] = (card >> 24) & 0xff; + hdr->card[4] = (card >> 32) & 0xff; + hdr->card[5] = (card >> 40) & 0xff; + hdr->card[6] = (card >> 48) & 0xff; + hdr->card[7] = (card >> 56) & 0xff; + /* This is not considered a read-only command even if the + * data structure is not modified, since the cached value + * may be modified and given that the HLL is a Redis string + * we need to propagate the change. */ + signalModifiedKey(c->db,c->argv[1]); + server.dirty++; + } + addReplyLongLong(c,card); + } +} + +/* PFMERGE dest src1 src2 src3 ... srcN => OK */ +void pfmergeCommand(redisClient *c) { + uint8_t max[HLL_REGISTERS]; + struct hllhdr *hdr; + int j; + + /* Compute an HLL with M[i] = MAX(M[i]_j). + * We we the maximum into the max array of registers. We'll write + * it to the target variable later. */ + memset(max,0,sizeof(max)); + for (j = 1; j < c->argc; j++) { + /* Check type and size. */ + robj *o = lookupKeyRead(c->db,c->argv[j]); + if (o == NULL) continue; /* Assume empty HLL for non existing var. */ + if (isHLLObjectOrReply(c,o) != REDIS_OK) return; + + /* Merge with this HLL with our 'max' HHL by setting max[i] + * to MAX(max[i],hll[i]). */ + if (hllMerge(max,o) == REDIS_ERR) { + addReplySds(c,sdsnew(invalid_hll_err)); + return; + } + } + + /* Create / unshare the destination key's value if needed. */ + robj *o = lookupKeyWrite(c->db,c->argv[1]); + if (o == NULL) { + /* Create the key with a string value of the exact length to + * hold our HLL data structure. sdsnewlen() when NULL is passed + * is guaranteed to return bytes initialized to zero. */ + o = createHLLObject(); + dbAdd(c->db,c->argv[1],o); + } else { + /* If key exists we are sure it's of the right type/size + * since we checked when merging the different HLLs, so we + * don't check again. */ + o = dbUnshareStringValue(c->db,c->argv[1],o); + } + + /* Only support dense objects as destination. */ + if (hllSparseToDense(o) == REDIS_ERR) { + addReplySds(c,sdsnew(invalid_hll_err)); + return; + } + + /* Write the resulting HLL to the destination HLL registers and + * invalidate the cached value. */ + hdr = o->ptr; + for (j = 0; j < HLL_REGISTERS; j++) { + HLL_DENSE_SET_REGISTER(hdr->registers,j,max[j]); + } + HLL_INVALIDATE_CACHE(hdr); + + signalModifiedKey(c->db,c->argv[1]); + /* We generate an PFADD event for PFMERGE for semantical simplicity + * since in theory this is a mass-add of elements. */ + notifyKeyspaceEvent(REDIS_NOTIFY_STRING,"pfadd",c->argv[1],c->db->id); + server.dirty++; + addReply(c,shared.ok); +} + +/* ========================== Testing / Debugging ========================== */ + +/* PFSELFTEST + * This command performs a self-test of the HLL registers implementation. + * Something that is not easy to test from within the outside. */ +#define HLL_TEST_CYCLES 1000 +void pfselftestCommand(redisClient *c) { + int j, i; + sds bitcounters = sdsnewlen(NULL,HLL_DENSE_SIZE); + struct hllhdr *hdr = (struct hllhdr*) bitcounters, *hdr2; + robj *o = NULL; + uint8_t bytecounters[HLL_REGISTERS]; + + /* Test 1: access registers. + * The test is conceived to test that the different counters of our data + * structure are accessible and that setting their values both result in + * the correct value to be retained and not affect adjacent values. */ + for (j = 0; j < HLL_TEST_CYCLES; j++) { + /* Set the HLL counters and an array of unsigned byes of the + * same size to the same set of random values. */ + for (i = 0; i < HLL_REGISTERS; i++) { + unsigned int r = rand() & HLL_REGISTER_MAX; + + bytecounters[i] = r; + HLL_DENSE_SET_REGISTER(hdr->registers,i,r); + } + /* Check that we are able to retrieve the same values. */ + for (i = 0; i < HLL_REGISTERS; i++) { + unsigned int val; + + HLL_DENSE_GET_REGISTER(val,hdr->registers,i); + if (val != bytecounters[i]) { + addReplyErrorFormat(c, + "TESTFAILED Register %d should be %d but is %d", + i, (int) bytecounters[i], (int) val); + goto cleanup; + } + } + } + + /* Test 2: approximation error. + * The test adds unique elements and check that the estimated value + * is always reasonable bounds. + * + * We check that the error is smaller than 4 times than the expected + * standard error, to make it very unlikely for the test to fail because + * of a "bad" run. + * + * The test is performed with both dense and sparse HLLs at the same + * time also verifying that the computed cardinality is the same. */ + memset(hdr->registers,0,HLL_DENSE_SIZE-HLL_HDR_SIZE); + o = createHLLObject(); + double relerr = 1.04/sqrt(HLL_REGISTERS); + int64_t checkpoint = 1; + uint64_t seed = (uint64_t)rand() | (uint64_t)rand() << 32; + uint64_t ele; + for (j = 1; j <= 10000000; j++) { + ele = j ^ seed; + hllDenseAdd(hdr->registers,(unsigned char*)&ele,sizeof(ele)); + hllAdd(o,(unsigned char*)&ele,sizeof(ele)); + + /* Make sure that for small cardinalities we use sparse + * encoding. */ + if (j == checkpoint && j < server.hll_sparse_max_bytes/2) { + hdr2 = o->ptr; + if (hdr2->encoding != HLL_SPARSE) { + addReplyError(c, "TESTFAILED sparse encoding not used"); + goto cleanup; + } + } + + /* Check that dense and sparse representations agree. */ + if (j == checkpoint && hllCount(hdr,NULL) != hllCount(o->ptr,NULL)) { + addReplyError(c, "TESTFAILED dense/sparse disagree"); + goto cleanup; + } + + /* Check error. */ + if (j == checkpoint) { + int64_t abserr = checkpoint - (int64_t)hllCount(hdr,NULL); + if (abserr < 0) abserr = -abserr; + if ((uint64_t)abserr >(uint64_t)(relerr * 4 * checkpoint)) { + addReplyErrorFormat(c, + "TESTFAILED Too big error. card:%llu abserr:%llu", + (unsigned long long) checkpoint, + (unsigned long long) abserr); + goto cleanup; + } + checkpoint *= 10; + } + } + + /* Success! */ + addReply(c,shared.ok); + +cleanup: + sdsfree(bitcounters); + if (o) decrRefCount(o); +} + +/* PFDEBUG ... args ... + * Different debugging related operations about the HLL implementation. */ +void pfdebugCommand(redisClient *c) { + char *cmd = c->argv[1]->ptr; + struct hllhdr *hdr; + robj *o; + int j; + + o = lookupKeyRead(c->db,c->argv[2]); + if (o == NULL) { + addReplyError(c,"The specified key does not exist"); + return; + } + if (isHLLObjectOrReply(c,o) != REDIS_OK) return; + o = dbUnshareStringValue(c->db,c->argv[2],o); + hdr = o->ptr; + + /* PFDEBUG GETREG */ + if (!strcasecmp(cmd,"getreg")) { + if (c->argc != 3) goto arityerr; + + if (hdr->encoding == HLL_SPARSE) { + if (hllSparseToDense(o) == REDIS_ERR) { + addReplySds(c,sdsnew(invalid_hll_err)); + return; + } + server.dirty++; /* Force propagation on encoding change. */ + } + + hdr = o->ptr; + addReplyMultiBulkLen(c,HLL_REGISTERS); + for (j = 0; j < HLL_REGISTERS; j++) { + uint8_t val; + + HLL_DENSE_GET_REGISTER(val,hdr->registers,j); + addReplyLongLong(c,val); + } + } + /* PFDEBUG DECODE */ + else if (!strcasecmp(cmd,"decode")) { + uint8_t *p, *end; + if(c->argc != 3) goto arityerr; + + p = o->ptr; + end = p + sdslen(o->ptr); + sds decoded = sdsempty(); + + if (hdr->encoding != HLL_SPARSE) { + addReplyError(c,"HLL encoding is not sparse"); + return; + } + + p += HLL_HDR_SIZE; + while(p < end) { + int runlen, regval; + + if (HLL_SPARSE_IS_ZERO(p)) { + runlen = HLL_SPARSE_ZERO_LEN(p); + p++; + decoded = sdscatprintf(decoded,"z:%d ",runlen); + } else if (HLL_SPARSE_IS_XZERO(p)) { + runlen = HLL_SPARSE_XZERO_LEN(p); + p += 2; + decoded = sdscatprintf(decoded,"Z:%d ",runlen); + } else { + runlen = HLL_SPARSE_VAL_LEN(p); + regval = HLL_SPARSE_VAL_VALUE(p); + p++; + decoded = sdscatprintf(decoded,"v:%d,%d ",regval,runlen); + } + } + decoded = sdstrim(decoded," "); + addReplyBulkCBuffer(c,decoded,sdslen(decoded)); + sdsfree(decoded); + } + /* PFDEBUG ENCODING */ + else if (!strcasecmp(cmd,"encoding")) { + char *encodingstr[2] = {"dense","sparse"}; + if (c->argc != 3) goto arityerr; + + addReplyStatus(c,encodingstr[hdr->encoding]); + } + /* PFDEBUG TODENSE */ + else if (!strcasecmp(cmd,"todense")) { + int conv = 0; + if (c->argc != 3) goto arityerr; + + if (hdr->encoding == HLL_SPARSE) { + if (hllSparseToDense(o) == REDIS_ERR) { + addReplySds(c,sdsnew(invalid_hll_err)); + return; + } + conv = 1; + server.dirty++; /* Force propagation on encoding change. */ + } + addReply(c,conv ? shared.cone : shared.czero); + } else { + addReplyErrorFormat(c,"Unknown PFDEBUG subcommand '%s'", cmd); + } + return; + +arityerr: + addReplyErrorFormat(c, + "Wrong number of arguments for the '%s' subcommand",cmd); +} + diff --git a/src/networking.c b/src/networking.c index bda3f9ee..52eeffe8 100644 --- a/src/networking.c +++ b/src/networking.c @@ -582,7 +582,7 @@ void acceptTcpHandler(aeEventLoop *el, int fd, void *privdata, int mask) { REDIS_NOTUSED(privdata); cfd = anetTcpAccept(server.neterr, fd, cip, sizeof(cip), &cport); - if (cfd == AE_ERR) { + if (cfd == ANET_ERR) { redisLog(REDIS_WARNING,"Accepting client connection: %s", server.neterr); return; } @@ -597,7 +597,7 @@ void acceptUnixHandler(aeEventLoop *el, int fd, void *privdata, int mask) { REDIS_NOTUSED(privdata); cfd = anetUnixAccept(server.neterr, fd); - if (cfd == AE_ERR) { + if (cfd == ANET_ERR) { redisLog(REDIS_WARNING,"Accepting client connection: %s", server.neterr); return; } @@ -662,7 +662,7 @@ void freeClient(redisClient *c) { } /* Log link disconnection with slave */ - if (c->flags & REDIS_SLAVE) { + if ((c->flags & REDIS_SLAVE) && !(c->flags & REDIS_MONITOR)) { char ip[REDIS_IP_STR_LEN]; if (anetPeerToString(c->fd,ip,sizeof(ip),NULL) != -1) { diff --git a/src/object.c b/src/object.c index e4bcaf39..d9f7e815 100644 --- a/src/object.c +++ b/src/object.c @@ -601,7 +601,7 @@ unsigned long estimateObjectIdleTime(robj *o) { } } -/* This is a helper function for the DEBUG command. We need to lookup keys +/* This is a helper function for the OBJECT command. We need to lookup keys * without any modification of LRU or other parameters. */ robj *objectCommandLookup(redisClient *c, robj *key) { dictEntry *de; diff --git a/src/rdb.c b/src/rdb.c index 7e85a5be..590db067 100644 --- a/src/rdb.c +++ b/src/rdb.c @@ -729,9 +729,9 @@ int rdbSave(char *filename) { rioWrite(&rdb,&cksum,8); /* Make sure data will not remain on the OS's output buffers */ - fflush(fp); - fsync(fileno(fp)); - fclose(fp); + if (fflush(fp) == EOF) goto werr; + if (fsync(fileno(fp)) == -1) goto werr; + if (fclose(fp) == EOF) goto werr; /* Use RENAME to make sure the DB file is changed atomically only * if the generate DB file is ok. */ @@ -1127,6 +1127,10 @@ void rdbLoadProgressCallback(rio *r, const void *buf, size_t len) { if (server.loading_process_events_interval_bytes && (r->processed_bytes + len)/server.loading_process_events_interval_bytes > r->processed_bytes/server.loading_process_events_interval_bytes) { + /* The DB can take some non trivial amount of time to load. Update + * our cached time since it is used to create and update the last + * interaction time with clients and for other important things. */ + updateCachedTime(); if (server.masterhost && server.repl_state == REDIS_REPL_TRANSFER) replicationSendNewlineToMaster(); loadingProgress(r->processed_bytes); diff --git a/src/redis-cli.c b/src/redis-cli.c index 59e8d412..87a2e69d 100644 --- a/src/redis-cli.c +++ b/src/redis-cli.c @@ -85,7 +85,7 @@ #define OUTPUT_RAW 1 #define OUTPUT_CSV 2 #define REDIS_CLI_KEEPALIVE_INTERVAL 15 /* seconds */ -#define REDIS_DEFAULT_PIPE_TIMEOUT 30 /* seconds */ +#define REDIS_CLI_DEFAULT_PIPE_TIMEOUT 30 /* seconds */ static redisContext *context; static struct config { @@ -108,6 +108,10 @@ static struct config { int pipe_timeout; int getrdb_mode; int stat_mode; + int scan_mode; + int intrinsic_latency_mode; + int intrinsic_latency_duration; + char *pattern; char *rdb_filename; int bigkeys; int stdinarg; /* get last arg from stdin. (-x option) */ @@ -119,6 +123,7 @@ static struct config { } config; static void usage(); +static void slaveMode(void); char *redisGitSHA1(void); char *redisGitDirty(void); @@ -126,14 +131,18 @@ char *redisGitDirty(void); * Utility functions *--------------------------------------------------------------------------- */ -static long long mstime(void) { +static long long ustime(void) { struct timeval tv; - long long mst; + long long ust; gettimeofday(&tv, NULL); - mst = ((long long)tv.tv_sec)*1000; - mst += (long long)(tv.tv_usec/1000); - return mst; + ust = ((long long)tv.tv_sec)*1000000; + ust += tv.tv_usec; + return ust; +} + +static long long mstime(void) { + return ustime()/1000; } static void cliRefreshPrompt(void) { @@ -629,6 +638,8 @@ static int cliSendCommand(int argc, char **argv, int repeat) { if (!strcasecmp(command,"monitor")) config.monitor_mode = 1; if (!strcasecmp(command,"subscribe") || !strcasecmp(command,"psubscribe")) config.pubsub_mode = 1; + if (!strcasecmp(command,"sync") || + !strcasecmp(command,"psync")) config.slave_mode = 1; /* Setup argument length */ argvlen = malloc(argc*sizeof(size_t)); @@ -650,6 +661,13 @@ static int cliSendCommand(int argc, char **argv, int repeat) { } } + if (config.slave_mode) { + printf("Entering slave output mode... (press Ctrl-C to quit)\n"); + slaveMode(); + config.slave_mode = 0; + return REDIS_ERR; /* Error = slaveMode lost connection to master */ + } + if (cliReadReply(output_raw) != REDIS_OK) { free(argvlen); return REDIS_ERR; @@ -743,6 +761,13 @@ static int parseOptions(int argc, char **argv) { config.slave_mode = 1; } else if (!strcmp(argv[i],"--stat")) { config.stat_mode = 1; + } else if (!strcmp(argv[i],"--scan")) { + config.scan_mode = 1; + } else if (!strcmp(argv[i],"--pattern") && !lastarg) { + config.pattern = argv[++i]; + } else if (!strcmp(argv[i],"--intrinsic-latency") && !lastarg) { + config.intrinsic_latency_mode = 1; + config.intrinsic_latency_duration = atoi(argv[++i]); } else if (!strcmp(argv[i],"--rdb") && !lastarg) { config.getrdb_mode = 1; config.rdb_filename = argv[++i]; @@ -802,33 +827,37 @@ static void usage() { "redis-cli %s\n" "\n" "Usage: redis-cli [OPTIONS] [cmd [arg [arg ...]]]\n" -" -h Server hostname (default: 127.0.0.1)\n" -" -p Server port (default: 6379)\n" -" -s Server socket (overrides hostname and port)\n" -" -a Password to use when connecting to the server\n" -" -r Execute specified command N times\n" +" -h Server hostname (default: 127.0.0.1).\n" +" -p Server port (default: 6379).\n" +" -s Server socket (overrides hostname and port).\n" +" -a Password to use when connecting to the server.\n" +" -r Execute specified command N times.\n" " -i When -r is used, waits seconds per command.\n" -" It is possible to specify sub-second times like -i 0.1\n" -" -n Database number\n" -" -x Read last argument from STDIN\n" -" -d Multi-bulk delimiter in for raw formatting (default: \\n)\n" -" -c Enable cluster mode (follow -ASK and -MOVED redirections)\n" +" It is possible to specify sub-second times like -i 0.1.\n" +" -n Database number.\n" +" -x Read last argument from STDIN.\n" +" -d Multi-bulk delimiter in for raw formatting (default: \\n).\n" +" -c Enable cluster mode (follow -ASK and -MOVED redirections).\n" " --raw Use raw formatting for replies (default when STDOUT is\n" -" not a tty)\n" -" --csv Output in CSV format\n" -" --latency Enter a special mode continuously sampling latency\n" +" not a tty).\n" +" --csv Output in CSV format.\n" +" --latency Enter a special mode continuously sampling latency.\n" " --latency-history Like --latency but tracking latency changes over time.\n" " Default time interval is 15 sec. Change it using -i.\n" -" --slave Simulate a slave showing commands received from the master\n" +" --slave Simulate a slave showing commands received from the master.\n" " --rdb Transfer an RDB dump from remote server to local file.\n" -" --pipe Transfer raw Redis protocol from stdin to server\n" -" --pipe-timeout In --pipe mode, abort with error if after sending all data\n" +" --pipe Transfer raw Redis protocol from stdin to server.\n" +" --pipe-timeout In --pipe mode, abort with error if after sending all data.\n" " no reply is received within seconds.\n" " Default timeout: %d. Use 0 to wait forever.\n" -" --bigkeys Sample Redis keys looking for big keys\n" -" --eval Send an EVAL command using the Lua script at \n" -" --help Output this help and exit\n" -" --version Output version and exit\n" +" --bigkeys Sample Redis keys looking for big keys.\n" +" --scan List all keys using the SCAN command.\n" +" --pattern Useful with --scan to specify a SCAN pattern.\n" +" --intrinsic-latency Run a test to measure intrinsic system latency.\n" +" The test will run for the specified amount of seconds.\n" +" --eval Send an EVAL command using the Lua script at .\n" +" --help Output this help and exit.\n" +" --version Output version and exit.\n" "\n" "Examples:\n" " cat /etc/passwd | redis-cli -x set mypasswd\n" @@ -836,12 +865,14 @@ static void usage() { " redis-cli -r 100 lpush mylist x\n" " redis-cli -r 100 -i 1 info | grep used_memory_human:\n" " redis-cli --eval myscript.lua key1 key2 , arg1 arg2 arg3\n" +" redis-cli --scan --pattern '*:12345*'\n" +"\n" " (Note: when using --eval the comma separates KEYS[] from ARGV[] items)\n" "\n" "When no command is given, redis-cli starts in interactive mode.\n" "Type \"help\" in interactive mode for information on available commands.\n" "\n", - version, REDIS_DEFAULT_PIPE_TIMEOUT); + version, REDIS_CLI_DEFAULT_PIPE_TIMEOUT); sdsfree(version); exit(1); } @@ -866,6 +897,7 @@ static void repl() { sds *argv; config.interactive = 1; + linenoiseSetMultiLine(1); linenoiseSetCompletionCallback(completionCallback); /* Only use history when stdin is a tty. */ @@ -968,6 +1000,10 @@ static int noninteractive(int argc, char **argv) { return retval; } +/*------------------------------------------------------------------------------ + * Eval mode + *--------------------------------------------------------------------------- */ + static int evalMode(int argc, char **argv) { sds script = sdsempty(); FILE *fp; @@ -1006,6 +1042,10 @@ static int evalMode(int argc, char **argv) { return cliSendCommand(argc+3-got_comma, argv2, config.repeat); } +/*------------------------------------------------------------------------------ + * Latency and latency history modes + *--------------------------------------------------------------------------- */ + #define LATENCY_SAMPLE_RATE 10 /* milliseconds. */ #define LATENCY_HISTORY_DEFAULT_INTERVAL 15000 /* milliseconds. */ static void latencyMode(void) { @@ -1050,6 +1090,10 @@ static void latencyMode(void) { } } +/*------------------------------------------------------------------------------ + * Slave mode + *--------------------------------------------------------------------------- */ + /* Sends SYNC and reads the number of bytes in the payload. Used both by * slaveMode() and getRDB(). */ unsigned long long sendSync(int fd) { @@ -1089,6 +1133,7 @@ static void slaveMode(void) { int fd = context->fd; unsigned long long payload = sendSync(fd); char buf[1024]; + int original_output = config.output; fprintf(stderr,"SYNC with master, discarding %llu " "bytes of bulk transfer...\n", payload); @@ -1112,8 +1157,13 @@ static void slaveMode(void) { /* Now we can use hiredis to read the incoming protocol. */ config.output = OUTPUT_CSV; while (cliReadReply(0) == REDIS_OK); + config.output = original_output; } +/*------------------------------------------------------------------------------ + * RDB transfer mode + *--------------------------------------------------------------------------- */ + /* This function implements --rdb, so it uses the replication protocol in order * to fetch the RDB file from a remote server. */ static void getRDB(void) { @@ -1159,6 +1209,10 @@ static void getRDB(void) { exit(0); } +/*------------------------------------------------------------------------------ + * Bulk import (pipe) mode + *--------------------------------------------------------------------------- */ + static void pipeMode(void) { int fd = (int)context->fd; long long errors = 0, replies = 0, obuf_len = 0, obuf_pos = 0; @@ -1310,92 +1364,287 @@ static void pipeMode(void) { exit(0); } +/*------------------------------------------------------------------------------ + * Find big keys + *--------------------------------------------------------------------------- */ + #define TYPE_STRING 0 #define TYPE_LIST 1 #define TYPE_SET 2 #define TYPE_HASH 3 #define TYPE_ZSET 4 +#define TYPE_NONE 5 + +static redisReply *sendScan(unsigned long long *it) { + redisReply *reply = redisCommand(context, "SCAN %llu", *it); + + /* Handle any error conditions */ + if(reply == NULL) { + fprintf(stderr, "\nI/O error\n"); + exit(1); + } else if(reply->type == REDIS_REPLY_ERROR) { + fprintf(stderr, "SCAN error: %s\n", reply->str); + exit(1); + } else if(reply->type != REDIS_REPLY_ARRAY) { + fprintf(stderr, "Non ARRAY response from SCAN!\n"); + exit(1); + } else if(reply->elements != 2) { + fprintf(stderr, "Invalid element count from SCAN!\n"); + exit(1); + } + + /* Validate our types are correct */ + assert(reply->element[0]->type == REDIS_REPLY_STRING); + assert(reply->element[1]->type == REDIS_REPLY_ARRAY); + + /* Update iterator */ + *it = atoi(reply->element[0]->str); + + return reply; +} + +static int getDbSize(void) { + redisReply *reply; + int size; + + reply = redisCommand(context, "DBSIZE"); + + if(reply == NULL || reply->type != REDIS_REPLY_INTEGER) { + fprintf(stderr, "Couldn't determine DBSIZE!\n"); + exit(1); + } + + /* Grab the number of keys and free our reply */ + size = reply->integer; + freeReplyObject(reply); + + return size; +} + +static int toIntType(char *key, char *type) { + if(!strcmp(type, "string")) { + return TYPE_STRING; + } else if(!strcmp(type, "list")) { + return TYPE_LIST; + } else if(!strcmp(type, "set")) { + return TYPE_SET; + } else if(!strcmp(type, "hash")) { + return TYPE_HASH; + } else if(!strcmp(type, "zset")) { + return TYPE_ZSET; + } else if(!strcmp(type, "none")) { + return TYPE_NONE; + } else { + fprintf(stderr, "Unknown type '%s' for key '%s'\n", type, key); + exit(1); + } +} + +static void getKeyTypes(redisReply *keys, int *types) { + redisReply *reply; + int i; + + /* Pipeline TYPE commands */ + for(i=0;ielements;i++) { + redisAppendCommand(context, "TYPE %s", keys->element[i]->str); + } + + /* Retrieve types */ + for(i=0;ielements;i++) { + if(redisGetReply(context, (void**)&reply)!=REDIS_OK) { + fprintf(stderr, "Error getting type for key '%s' (%d: %s)\n", + keys->element[i]->str, context->err, context->errstr); + exit(1); + } else if(reply->type != REDIS_REPLY_STATUS) { + fprintf(stderr, "Invalid reply type (%d) for TYPE on key '%s'!\n", + reply->type, keys->element[i]->str); + exit(1); + } + + types[i] = toIntType(keys->element[i]->str, reply->str); + freeReplyObject(reply); + } +} + +static void getKeySizes(redisReply *keys, int *types, + unsigned long long *sizes) +{ + redisReply *reply; + char *sizecmds[] = {"STRLEN","LLEN","SCARD","HLEN","ZCARD"}; + int i; + + /* Pipeline size commands */ + for(i=0;ielements;i++) { + /* Skip keys that were deleted */ + if(types[i]==TYPE_NONE) + continue; + + redisAppendCommand(context, "%s %s", sizecmds[types[i]], + keys->element[i]->str); + } + + /* Retreive sizes */ + for(i=0;ielements;i++) { + /* Skip keys that dissapeared between SCAN and TYPE */ + if(types[i] == TYPE_NONE) { + sizes[i] = 0; + continue; + } + + /* Retreive size */ + if(redisGetReply(context, (void**)&reply)!=REDIS_OK) { + fprintf(stderr, "Error getting size for key '%s' (%d: %s)\n", + keys->element[i]->str, context->err, context->errstr); + exit(1); + } else if(reply->type != REDIS_REPLY_INTEGER) { + /* Theoretically the key could have been removed and + * added as a different type between TYPE and SIZE */ + fprintf(stderr, + "Warning: %s on '%s' failed (may have changed type)\n", + sizecmds[types[i]], keys->element[i]->str); + sizes[i] = 0; + } else { + sizes[i] = reply->integer; + } + + freeReplyObject(reply); + } +} static void findBigKeys(void) { - unsigned long long biggest[5] = {0,0,0,0,0}; - unsigned long long samples = 0; - redisReply *reply1, *reply2, *reply3 = NULL; - char *sizecmd, *typename[] = {"string","list","set","hash","zset"}; + unsigned long long biggest[5] = {0}, counts[5] = {0}, totalsize[5] = {0}; + unsigned long long sampled = 0, total_keys, totlen=0, *sizes=NULL, it=0; + sds maxkeys[5] = {0}; + char *typename[] = {"string","list","set","hash","zset"}; char *typeunit[] = {"bytes","items","members","fields","members"}; - int type; + redisReply *reply, *keys; + int type, *types=NULL, arrsize=0, i; + double pct; - printf("\n# Press ctrl+c when you have had enough of it... :)\n"); - printf("# You can use -i 0.1 to sleep 0.1 sec every 100 sampled keys\n"); - printf("# in order to reduce server load (usually not needed).\n\n"); - while(1) { - /* Sample with RANDOMKEY */ - reply1 = redisCommand(context,"RANDOMKEY"); - if (reply1 == NULL) { - fprintf(stderr,"\nI/O error\n"); - exit(1); - } else if (reply1->type == REDIS_REPLY_ERROR) { - fprintf(stderr, "RANDOMKEY error: %s\n", - reply1->str); - exit(1); - } else if (reply1->type == REDIS_REPLY_NIL) { - fprintf(stderr, "It looks like the database is empty!\n"); + /* Total keys pre scanning */ + total_keys = getDbSize(); + + /* Status message */ + printf("\n# Scanning the entire keyspace to find biggest keys as well as\n"); + printf("# average sizes per key type. You can use -i 0.1 to sleep 0.1 sec\n"); + printf("# per 100 SCAN commands (not usually needed).\n\n"); + + /* New up sds strings to keep track of overall biggest per type */ + for(i=0;istr); - assert(reply2 && reply2->type == REDIS_REPLY_STATUS); - samples++; + /* SCAN loop */ + do { + /* Calculate approximate percentage completion */ + pct = 100 * (double)sampled/total_keys; - /* Get the key "size" */ - if (!strcmp(reply2->str,"string")) { - sizecmd = "STRLEN"; - type = TYPE_STRING; - } else if (!strcmp(reply2->str,"list")) { - sizecmd = "LLEN"; - type = TYPE_LIST; - } else if (!strcmp(reply2->str,"set")) { - sizecmd = "SCARD"; - type = TYPE_SET; - } else if (!strcmp(reply2->str,"hash")) { - sizecmd = "HLEN"; - type = TYPE_HASH; - } else if (!strcmp(reply2->str,"zset")) { - sizecmd = "ZCARD"; - type = TYPE_ZSET; - } else if (!strcmp(reply2->str,"none")) { - freeReplyObject(reply1); - freeReplyObject(reply2); - continue; - } else { - fprintf(stderr, "Unknown key type '%s' for key '%s'\n", - reply2->str, reply1->str); - exit(1); + /* Grab some keys and point to the keys array */ + reply = sendScan(&it); + keys = reply->element[1]; + + /* Reallocate our type and size array if we need to */ + if(keys->elements > arrsize) { + types = zrealloc(types, sizeof(int)*keys->elements); + sizes = zrealloc(sizes, sizeof(unsigned long long)*keys->elements); + + if(!types || !sizes) { + fprintf(stderr, "Failed to allocate storage for keys!\n"); + exit(1); + } + + arrsize = keys->elements; } - reply3 = redisCommand(context,"%s %s", sizecmd, reply1->str); - if (reply3 && reply3->type == REDIS_REPLY_INTEGER) { - if (biggest[type] < (unsigned)reply3->integer) { - printf("Biggest %-6s found so far '%s' with %llu %s.\n", - typename[type], reply1->str, - (unsigned long long) reply3->integer, - typeunit[type]); - biggest[type] = reply3->integer; + /* Retreive types and then sizes */ + getKeyTypes(keys, types); + getKeySizes(keys, types, sizes); + + /* Now update our stats */ + for(i=0;ielements;i++) { + if((type = types[i]) == TYPE_NONE) + continue; + + totalsize[type] += sizes[i]; + counts[type]++; + totlen += keys->element[i]->len; + sampled++; + + if(biggest[type]element[i]->str, sizes[i], + typeunit[type]); + + /* Keep track of biggest key name for this type */ + maxkeys[type] = sdscpy(maxkeys[type], keys->element[i]->str); + if(!maxkeys[type]) { + fprintf(stderr, "Failed to allocate memory for key!\n"); + exit(1); + } + + /* Keep track of the biggest size for this type */ + biggest[type] = sizes[i]; + } + + /* Update overall progress */ + if(sampled % 1000000 == 0) { + printf("[%05.2f%%] Sampled %llu keys so far\n", pct, sampled); } } - if ((samples % 1000000) == 0) - printf("(%llu keys sampled)\n", samples); - - if ((samples % 100) == 0 && config.interval) + /* Sleep if we've been directed to do so */ + if(sampled && (sampled %100) == 0 && config.interval) { usleep(config.interval); + } + + freeReplyObject(reply); + } while(it != 0); - freeReplyObject(reply1); - freeReplyObject(reply2); - if (reply3) freeReplyObject(reply3); + if(types) zfree(types); + if(sizes) zfree(sizes); + + /* We're done */ + printf("\n-------- summary -------\n\n"); + + printf("Sampled %llu keys in the keyspace!\n", sampled); + printf("Total key length in bytes is %llu (avg len %.2f)\n\n", + totlen, totlen ? (double)totlen/sampled : 0); + + /* Output the biggest keys we found, for types we did find */ + for(i=0;i0) { + printf("Biggest %6s found '%s' has %llu %s\n", typename[i], maxkeys[i], + biggest[i], typeunit[i]); + } } + + printf("\n"); + + for(i=0;itype == REDIS_REPLY_ERROR) { + printf("ERROR: %s\n", reply->str); + exit(1); + } else { + int j; + + cur = strtoull(reply->element[0]->str,NULL,10); + for (j = 0; j < reply->element[1]->elements; j++) + printf("%s\n", reply->element[1]->element[j]->str); + } + freeReplyObject(reply); + } while(cur != 0); + + exit(0); +} + +/*------------------------------------------------------------------------------ + * Intrisic latency mode. + * + * Measure max latency of a running process that does not result from + * syscalls. Basically this software should provide an hint about how much + * time the kernel leaves the process without a chance to run. + *--------------------------------------------------------------------------- */ + +/* This is just some computation the compiler can't optimize out. + * Should run in less than 100-200 microseconds even using very + * slow hardware. Runs in less than 10 microseconds in modern HW. */ +unsigned long compute_something_fast(void) { + unsigned char s[256], i, j, t; + int count = 1000, k; + unsigned long output = 0; + + for (k = 0; k < 256; k++) s[k] = k; + + i = 0; + j = 0; + while(count--) { + i++; + j = j + s[i]; + t = s[i]; + s[i] = s[j]; + s[j] = t; + output += s[(s[i]+s[j])&255]; + } + return output; +} + +static void intrinsicLatencyMode(void) { + long long test_end, run_time, max_latency = 0, runs = 0; + + run_time = config.intrinsic_latency_duration*1000000; + test_end = ustime() + run_time; + + while(1) { + long long start, end, latency; + + start = ustime(); + compute_something_fast(); + end = ustime(); + latency = end-start; + runs++; + if (latency <= 0) continue; + + /* Reporting */ + if (latency > max_latency) { + max_latency = latency; + printf("Max latency so far: %lld microseconds.\n", max_latency); + } + + if (end > test_end) { + printf("\n%lld total runs (avg %lld microseconds per run).\n", + runs, run_time/runs); + printf("Worst run took %.02fx times the avarege.\n", + (double) max_latency / (run_time/runs)); + exit(0); + } + } +} + +/*------------------------------------------------------------------------------ + * Program main() + *--------------------------------------------------------------------------- */ + int main(int argc, char **argv) { int firstarg; @@ -1553,9 +1902,13 @@ int main(int argc, char **argv) { config.cluster_mode = 0; config.slave_mode = 0; config.getrdb_mode = 0; + config.stat_mode = 0; + config.scan_mode = 0; + config.intrinsic_latency_mode = 0; + config.pattern = NULL; config.rdb_filename = NULL; config.pipe_mode = 0; - config.pipe_timeout = REDIS_DEFAULT_PIPE_TIMEOUT; + config.pipe_timeout = REDIS_CLI_DEFAULT_PIPE_TIMEOUT; config.bigkeys = 0; config.stdinarg = 0; config.auth = NULL; @@ -1614,6 +1967,15 @@ int main(int argc, char **argv) { statMode(); } + /* Scan mode */ + if (config.scan_mode) { + if (cliConnect(0) == REDIS_ERR) exit(1); + scanMode(); + } + + /* Intrinsic latency mode */ + if (config.intrinsic_latency_mode) intrinsicLatencyMode(); + /* Start interactive mode when no command is provided */ if (argc == 0 && !config.eval) { /* Note that in repl mode we don't abort on connection error. diff --git a/src/redis.c b/src/redis.c index ddde129a..8a6d85d1 100644 --- a/src/redis.c +++ b/src/redis.c @@ -56,6 +56,10 @@ #include #include +#ifdef _WIN32 +#include "Win32_Interop\Win32_FDAPI.h" +#endif + /* Our shared "common" objects */ struct sharedObjectsStruct shared; @@ -173,12 +177,16 @@ struct redisCommand redisCommandTable[] = { {"zrem",zremCommand,-3,"w",0,NULL,1,1,1,0,0}, {"zremrangebyscore",zremrangebyscoreCommand,4,"w",0,NULL,1,1,1,0,0}, {"zremrangebyrank",zremrangebyrankCommand,4,"w",0,NULL,1,1,1,0,0}, + {"zremrangebylex",zremrangebylexCommand,4,"w",0,NULL,1,1,1,0,0}, {"zunionstore",zunionstoreCommand,-4,"wm",0,zunionInterGetKeys,0,0,0,0,0}, {"zinterstore",zinterstoreCommand,-4,"wm",0,zunionInterGetKeys,0,0,0,0,0}, {"zrange",zrangeCommand,-4,"r",0,NULL,1,1,1,0,0}, {"zrangebyscore",zrangebyscoreCommand,-4,"r",0,NULL,1,1,1,0,0}, {"zrevrangebyscore",zrevrangebyscoreCommand,-4,"r",0,NULL,1,1,1,0,0}, + {"zrangebylex",zrangebylexCommand,-4,"r",0,NULL,1,1,1,0,0}, + {"zrevrangebylex",zrevrangebylexCommand,-4,"r",0,NULL,1,1,1,0,0}, {"zcount",zcountCommand,4,"r",0,NULL,1,1,1,0,0}, + {"zlexcount",zlexcountCommand,4,"r",0,NULL,1,1,1,0,0}, {"zrevrange",zrevrangeCommand,-4,"r",0,NULL,1,1,1,0,0}, {"zcard",zcardCommand,2,"r",0,NULL,1,1,1,0,0}, {"zscore",zscoreCommand,3,"r",0,NULL,1,1,1,0,0}, @@ -223,7 +231,7 @@ struct redisCommand redisCommandTable[] = { {"save",saveCommand,1,"ars",0,NULL,0,0,0,0,0}, {"bgsave",bgsaveCommand,1,"ar",0,NULL,0,0,0,0,0}, {"bgrewriteaof",bgrewriteaofCommand,1,"ar",0,NULL,0,0,0,0,0}, - {"shutdown",shutdownCommand,-1,"arl",0,NULL,0,0,0,0,0}, + {"shutdown",shutdownCommand,-1,"arlt",0,NULL,0,0,0,0,0}, {"lastsave",lastsaveCommand,1,"rR",0,NULL,0,0,0,0,0}, {"type",typeCommand,2,"r",0,NULL,1,1,1,0,0}, {"multi",multiCommand,1,"rs",0,NULL,0,0,0,0,0}, @@ -242,7 +250,7 @@ struct redisCommand redisCommandTable[] = { {"persist",persistCommand,2,"w",0,NULL,1,1,1,0,0}, {"slaveof",slaveofCommand,3,"ast",0,NULL,0,0,0,0,0}, {"debug",debugCommand,-2,"as",0,NULL,0,0,0,0,0}, - {"config",configCommand,-2,"ar",0,NULL,0,0,0,0,0}, + {"config",configCommand,-2,"art",0,NULL,0,0,0,0,0}, {"subscribe",subscribeCommand,-2,"rpslt",0,NULL,0,0,0,0,0}, {"unsubscribe",unsubscribeCommand,-1,"rpslt",0,NULL,0,0,0,0,0}, {"psubscribe",psubscribeCommand,-2,"rpslt",0,NULL,0,0,0,0,0}, @@ -262,7 +270,13 @@ struct redisCommand redisCommandTable[] = { {"script",scriptCommand,-2,"ras",0,NULL,0,0,0,0,0}, {"time",timeCommand,1,"rR",0,NULL,0,0,0,0,0}, {"bitop",bitopCommand,-4,"wm",0,NULL,2,-1,1,0,0}, - {"bitcount",bitcountCommand,-2,"r",0,NULL,1,1,1,0,0} + {"bitcount",bitcountCommand,-2,"r",0,NULL,1,1,1,0,0}, + {"bitpos",bitposCommand,-3,"r",0,NULL,1,1,1,0,0}, + {"pfselftest",pfselftestCommand,1,"r",0,NULL,0,0,0,0,0}, + {"pfadd",pfaddCommand,-2,"wm",0,NULL,1,1,1,0,0}, + {"pfcount",pfcountCommand,-2,"w",0,NULL,1,1,1,0,0}, + {"pfmerge",pfmergeCommand,-2,"wm",0,NULL,1,-1,1,0,0}, + {"pfdebug",pfdebugCommand,-3,"w",0,NULL,0,0,0,0,0} }; /*============================ Utility functions ============================ */ @@ -336,12 +350,12 @@ void redisLogFromHandler(int level, const char *msg) { if (fd == -1) return; ll2string(buf,sizeof(buf),getpid()); if (write(fd,"[",1) == -1) goto err; - if (write(fd,buf,(unsigned int)strlen(buf)) == -1) goto err; + if (write(fd,buf,strlen(buf)) == -1) goto err; if (write(fd," | signal handler] (",20) == -1) goto err; ll2string(buf,sizeof(buf),time(NULL)); - if (write(fd,buf,(unsigned int)strlen(buf)) == -1) goto err; + if (write(fd,buf,strlen(buf)) == -1) goto err; if (write(fd,") ",2) == -1) goto err; - if (write(fd,msg,(unsigned int)strlen(msg)) == -1) goto err; + if (write(fd,msg,strlen(msg)) == -1) goto err; if (write(fd,"\n",1) == -1) goto err; err: if (!log_to_stdout) close(fd); @@ -718,10 +732,10 @@ void activeExpireCycle(int type) { * 2) If last time we hit the time limit, we want to scan all DBs * in this iteration, as there is work to do in some DB and we don't want * expired keys to use memory for too much time. */ - if (dbs_per_call > server.dbnum || timelimit_exit) + if ((dbs_per_call > (unsigned)server.dbnum) || timelimit_exit) dbs_per_call = server.dbnum; - /* We can use at max REDIS_EXPIRELOOKUPS_TIME_PERC percentage of CPU time + /* We can use at max ACTIVE_EXPIRE_CYCLE_SLOW_TIME_PERC percentage of CPU time * per iteration. Since this function gets called with a frequency of * server.hz times per second, the following is the max amount of * microseconds we can spend in this function. */ @@ -749,11 +763,11 @@ void activeExpireCycle(int type) { int ttl_samples; /* If there is nothing to expire try next DB ASAP. */ - if ((num = dictSize(db->expires)) == 0) { + if ((num = (unsigned long) dictSize(db->expires)) == 0) { db->avg_ttl = 0; break; } - slots = dictSlots(db->expires); + slots = (unsigned long)dictSlots(db->expires); now = mstime(); /* When there are less than 1% filled slots getting random @@ -939,7 +953,7 @@ void databasesCron(void) { unsigned int j; /* Don't test more DBs than we have. */ - if (dbs_per_call > server.dbnum) dbs_per_call = server.dbnum; + if (dbs_per_call > (unsigned)server.dbnum) dbs_per_call = server.dbnum; /* Resize */ for (j = 0; j < dbs_per_call; j++) { @@ -962,6 +976,15 @@ void databasesCron(void) { } } +/* We take a cached value of the unix time in the global state because with + * virtual memory and aging there is to store the current time in objects at + * every object access, and accuracy is not needed. To access a global var is + * a lot faster than calling time(NULL) */ +void updateCachedTime(void) { + server.unixtime = time(NULL); + server.mstime = mstime(); +} + /* This is our timer interrupt, called server.hz times per second. * Here is where we do a number of things that need to be done asynchronously. * For instance: @@ -991,12 +1014,8 @@ int serverCron(struct aeEventLoop *eventLoop, long long id, void *clientData) { * handler if we don't return here fast enough. */ if (server.watchdog_period) watchdogScheduleSignal(server.watchdog_period); - /* We take a cached value of the unix time in the global state because - * with virtual memory and aging there is to store the current time - * in objects at every object access, and accuracy is not needed. - * To access a global var is faster than calling time(NULL) */ - server.unixtime = time(NULL); - server.mstime = mstime(); + /* Update the time cache. */ + updateCachedTime(); run_with_period(100) trackOperationsPerSecond(); @@ -1018,6 +1037,9 @@ int serverCron(struct aeEventLoop *eventLoop, long long id, void *clientData) { if (zmalloc_used_memory() > server.stat_peak_memory) server.stat_peak_memory = zmalloc_used_memory(); + /* Sample the RSS here since this is a relatively slow call. */ + server.resident_set_size = zmalloc_get_rss(); + /* We received a SIGTERM, shutting down here in a safe way, as it is * not ok doing so inside the signal handler. */ if (server.shutdown_asap) { @@ -1153,10 +1175,19 @@ int serverCron(struct aeEventLoop *eventLoop, long long id, void *clientData) { } - /* If we postponed an AOF buffer flush, let's try to do it every time the - * cron function is called. */ + /* AOF postponed flush: Try at every cron cycle if the slow fsync + * completed. */ if (server.aof_flush_postponed_start) flushAppendOnlyFile(0); + /* AOF write errors: in this case we have a buffer to flush as well and + * clear the AOF error in case of success to make the DB writable again, + * however to try every second is enough in case of 'hz' is set to + * an higher frequency. */ + run_with_period(1000) { + if (server.aof_last_write_status == REDIS_ERR) + flushAppendOnlyFile(0); + } + /* Close clients that need to be closed asynchronous */ freeClientsInAsyncFreeQueue(); @@ -1288,6 +1319,12 @@ void createSharedObjects(void) { shared.bulkhdr[j] = createObject(REDIS_STRING, sdscatprintf(sdsempty(),"$%d\r\n",j)); } + /* The following two shared objects, minstring and maxstrings, are not + * actually used for their value but as a special object meaning + * respectively the minimum possible string and the maximum possible + * string in string comparisons for the ZRANGEBYLEX command. */ + shared.minstring = createStringObject("minstring",9); + shared.maxstring = createStringObject("maxstring",9); } void initServerConfig() { @@ -1303,6 +1340,7 @@ void initServerConfig() { server.arch_bits = (sizeof(long) == 8) ? 64 : 32; #endif server.port = REDIS_SERVERPORT; + server.tcp_backlog = REDIS_TCP_BACKLOG; server.bindaddr_count = 0; server.unixsocket = NULL; server.unixsocketperm = REDIS_DEFAULT_UNIX_SOCKET_PERM; @@ -1329,7 +1367,7 @@ void initServerConfig() { server.syslog_enabled = REDIS_DEFAULT_SYSLOG_ENABLED; } #else - server.syslog_enabled = REDIS_DEFAULT_SYSLOG_ENABLED; + server.syslog_enabled = REDIS_DEFAULT_SYSLOG_ENABLED; server.syslog_ident = zstrdup(REDIS_DEFAULT_SYSLOG_IDENT); #endif server.syslog_facility = LOG_LOCAL0; @@ -1371,6 +1409,7 @@ void initServerConfig() { server.set_max_intset_entries = REDIS_SET_MAX_INTSET_ENTRIES; server.zset_max_ziplist_entries = REDIS_ZSET_MAX_ZIPLIST_ENTRIES; server.zset_max_ziplist_value = REDIS_ZSET_MAX_ZIPLIST_VALUE; + server.hll_sparse_max_bytes = REDIS_DEFAULT_HLL_SPARSE_MAX_BYTES; server.shutdown_asap = 0; server.repl_ping_slave_period = REDIS_REPL_PING_SLAVE_PERIOD; server.repl_timeout = REDIS_REPL_TIMEOUT; @@ -1448,22 +1487,22 @@ void initServerConfig() { } /* This function will try to raise the max number of open files accordingly to - * the configured max number of clients. It will also account for 32 additional - * file descriptors as we need a few more for persistence, listening - * sockets, log files and so forth. + * the configured max number of clients. It also reserves a number of file + * descriptors (REDIS_MIN_RESERVED_FDS) for extra operations of + * persistence, listening sockets, log files and so forth. * * If it will not be possible to set the limit accordingly to the configured * max number of clients, the function will do the reverse setting * server.maxclients to the value that we can actually handle. */ void adjustOpenFilesLimit(void) { #ifndef _WIN32 - rlim_t maxfiles = server.maxclients+32; + rlim_t maxfiles = server.maxclients+REDIS_MIN_RESERVED_FDS; struct rlimit limit; if (getrlimit(RLIMIT_NOFILE,&limit) == -1) { redisLog(REDIS_WARNING,"Unable to obtain the current NOFILE limit (%s), assuming 1024 and setting the max clients configuration accordingly.", strerror(errno)); - server.maxclients = 1024-32; + server.maxclients = 1024-REDIS_MIN_RESERVED_FDS; } else { rlim_t oldlimit = limit.rlim_cur; @@ -1471,22 +1510,58 @@ void adjustOpenFilesLimit(void) { * for our needs. */ if (oldlimit < maxfiles) { rlim_t f; - + int setrlimit_error = 0; + + /* Try to set the file limit to match 'maxfiles' or at least + * to the higher value supported less than maxfiles. */ f = maxfiles; while(f > oldlimit) { + int decr_step = 16; + limit.rlim_cur = f; limit.rlim_max = f; if (setrlimit(RLIMIT_NOFILE,&limit) != -1) break; - f -= 128; + setrlimit_error = errno; + + /* We failed to set file limit to 'f'. Try with a + * smaller limit decrementing by a few FDs per iteration. */ + if (f < decr_step) break; + f -= decr_step; } + + /* Assume that the limit we get initially is still valid if + * our last try was even lower. */ if (f < oldlimit) f = oldlimit; + if (f != maxfiles) { - server.maxclients = f-32; - redisLog(REDIS_WARNING,"Unable to set the max number of files limit to %d (%s), setting the max clients configuration to %d.", - (int) maxfiles, strerror(errno), (int) server.maxclients); + int old_maxclients = server.maxclients; + server.maxclients = f-REDIS_MIN_RESERVED_FDS; + if (server.maxclients < 1) { + redisLog(REDIS_WARNING,"Your current 'ulimit -n' " + "of %llu is not enough for Redis to start. " + "Please increase your open file limit to at least " + "%llu. Exiting.", + (unsigned long long) oldlimit, + (unsigned long long) maxfiles); + exit(1); + } + redisLog(REDIS_WARNING,"You requested maxclients of %d " + "requiring at least %llu max file descriptors.", + old_maxclients, + (unsigned long long) maxfiles); + redisLog(REDIS_WARNING,"Redis can't set maximum open files " + "to %llu because of OS error: %s.", + (unsigned long long) maxfiles, strerror(setrlimit_error)); + redisLog(REDIS_WARNING,"Current maximum open files is %llu. " + "maxclients has been reduced to %d to compensate for " + "low ulimit. " + "If you need higher maxclients increase 'ulimit -n'.", + (unsigned long long) oldlimit, server.maxclients); } else { - redisLog(REDIS_NOTICE,"Max number of open files set to %d", - (int) maxfiles); + redisLog(REDIS_NOTICE,"Increased maximum number of open files " + "to %llu (it was originally set to %llu).", + (unsigned long long) maxfiles, + (unsigned long long) oldlimit); } } } @@ -1521,9 +1596,11 @@ int listenToPort(int port, int *fds, int *count) { if (server.bindaddr[j] == NULL) { /* Bind * for both IPv6 and IPv4, we enter here only if * server.bindaddr_count == 0. */ - fds[*count] = anetTcp6Server(server.neterr,port,NULL); + fds[*count] = anetTcp6Server(server.neterr,port,NULL, + server.tcp_backlog); if (fds[*count] != ANET_ERR) (*count)++; - fds[*count] = anetTcpServer(server.neterr,port,NULL); + fds[*count] = anetTcpServer(server.neterr,port,NULL, + server.tcp_backlog); if (fds[*count] != ANET_ERR) (*count)++; /* Exit the loop if we were able to bind * on IPv4 or IPv6, * otherwise fds[*count] will be ANET_ERR and we'll print an @@ -1531,10 +1608,12 @@ int listenToPort(int port, int *fds, int *count) { if (*count) break; } else if (strchr(server.bindaddr[j],':')) { /* Bind IPv6 address. */ - fds[*count] = anetTcp6Server(server.neterr,port,server.bindaddr[j]); + fds[*count] = anetTcp6Server(server.neterr,port,server.bindaddr[j], + server.tcp_backlog); } else { /* Bind IPv4 address. */ - fds[*count] = anetTcpServer(server.neterr,port,server.bindaddr[j]); + fds[*count] = anetTcpServer(server.neterr,port,server.bindaddr[j], + server.tcp_backlog); } if (fds[*count] == ANET_ERR) { redisLog(REDIS_WARNING, @@ -1548,6 +1627,27 @@ int listenToPort(int port, int *fds, int *count) { return REDIS_OK; } +/* Resets the stats that we expose via INFO or other means that we want + * to reset via CONFIG RESETSTAT. The function is also used in order to + * initialize these fields in initServer() at server startup. */ +void resetServerStats(void) { + server.stat_numcommands = 0; + server.stat_numconnections = 0; + server.stat_expiredkeys = 0; + server.stat_evictedkeys = 0; + server.stat_keyspace_misses = 0; + server.stat_keyspace_hits = 0; + server.stat_fork_time = 0; + server.stat_rejected_conn = 0; + server.stat_sync_full = 0; + server.stat_sync_partial_ok = 0; + server.stat_sync_partial_err = 0; + memset(server.ops_sec_samples,0,sizeof(server.ops_sec_samples)); + server.ops_sec_idx = 0; + server.ops_sec_last_sample_time = mstime(); + server.ops_sec_last_sample_ops = 0; +} + void initServer() { int j; #ifdef _WIN32 @@ -1568,9 +1668,9 @@ void initServer() { #ifdef _WIN32 /* Force binary mode on all files */ _fmode = _O_BINARY; - _setmode(_fileno(stdin), _O_BINARY); - _setmode(_fileno(stdout), _O_BINARY); - _setmode(_fileno(stderr), _O_BINARY); + setmode(_fileno(stdin), _O_BINARY); + setmode(_fileno(stdout), _O_BINARY); + setmode(_fileno(stderr), _O_BINARY); /* Set C locale, forcing strtod() to work with dots */ setlocale(LC_ALL, "C"); @@ -1602,7 +1702,8 @@ void initServer() { /* Open the listening Unix domain socket. */ if (server.unixsocket != NULL) { unlink(server.unixsocket); /* don't care if this fails */ - server.sofd = anetUnixServer(server.neterr,server.unixsocket,server.unixsocketperm); + server.sofd = anetUnixServer(server.neterr,server.unixsocket, + server.unixsocketperm, server.tcp_backlog); if (server.sofd == ANET_ERR) { redisLog(REDIS_WARNING, "Opening socket: %s", server.neterr); exit(1); @@ -1639,27 +1740,16 @@ void initServer() { server.rdb_save_time_last = -1; server.rdb_save_time_start = -1; server.dirty = 0; - server.stat_numcommands = 0; - server.stat_numconnections = 0; - server.stat_expiredkeys = 0; - server.stat_evictedkeys = 0; + resetServerStats(); + /* A few stats we don't want to reset: server startup time, and peak mem. */ server.stat_starttime = time(NULL); - server.stat_keyspace_misses = 0; - server.stat_keyspace_hits = 0; server.stat_peak_memory = 0; - server.stat_fork_time = 0; - server.stat_rejected_conn = 0; - server.stat_sync_full = 0; - server.stat_sync_partial_ok = 0; - server.stat_sync_partial_err = 0; - memset(server.ops_sec_samples,0,sizeof(server.ops_sec_samples)); - server.ops_sec_idx = 0; - server.ops_sec_last_sample_time = mstime(); - server.ops_sec_last_sample_ops = 0; - server.unixtime = time(NULL); - server.mstime = mstime(); + server.resident_set_size = 0; server.lastbgsave_status = REDIS_OK; + server.aof_last_write_status = REDIS_OK; + server.aof_last_write_errno = 0; server.repl_good_slaves_count = 0; + updateCachedTime(); /* Create the serverCron() time event, that's our main way to process * background operations. */ @@ -1863,7 +1953,7 @@ void forceCommandPropagation(redisClient *c, int flags) { /* Call() is the core of Redis execution of a command */ void call(redisClient *c, int flags) { - long long dirty, start = ustime(), duration; + long long dirty, start, duration; int client_old_flags = c->flags; /* Sent the command to clients in MONITOR mode, only if the commands are @@ -1879,9 +1969,10 @@ void call(redisClient *c, int flags) { c->flags &= ~(REDIS_FORCE_AOF|REDIS_FORCE_REPL); redisOpArrayInit(&server.also_propagate); dirty = server.dirty; + start = ustime(); c->cmd->proc(c); - dirty = server.dirty-dirty; duration = ustime()-start; + dirty = server.dirty-dirty; /* When EVAL is called loading the AOF we don't want commands called * from Lua to go into the slowlog or to populate statistics. */ @@ -1998,15 +2089,22 @@ int processCommand(redisClient *c) { /* Don't accept write commands if there are problems persisting on disk * and if this is a master instance. */ - if (server.stop_writes_on_bgsave_err && - server.saveparamslen > 0 - && server.lastbgsave_status == REDIS_ERR && - server.masterhost != NULL && + if (((server.stop_writes_on_bgsave_err && + server.saveparamslen > 0 && + server.lastbgsave_status == REDIS_ERR) || + server.aof_last_write_status == REDIS_ERR) && + server.masterhost == NULL && (c->cmd->flags & REDIS_CMD_WRITE || c->cmd->proc == pingCommand)) { flagTransaction(c); - addReply(c, shared.bgsaveerr); + if (server.aof_last_write_status == REDIS_OK) + addReply(c, shared.bgsaveerr); + else + addReplySds(c, + sdscatprintf(sdsempty(), + "-MISCONF Errors writing to the AOF file: %s\r\n", + strerror(server.aof_last_write_errno))); return REDIS_OK; } @@ -2160,7 +2258,8 @@ int prepareForShutdown(int flags) { /* Close the listening sockets. Apparently this allows faster restarts. */ closeListeningSockets(1); - redisLog(REDIS_WARNING,"Redis is now ready to exit, bye bye..."); + redisLog(REDIS_WARNING,"%s is now ready to exit, bye bye...", + server.sentinel_mode ? "Sentinel" : "Redis"); return REDIS_OK; } @@ -2284,7 +2383,8 @@ sds genRedisInfoString(char *section) { /* Server */ if (allsections || defsections || !strcasecmp(section,"server")) { #ifndef _WIN32 - struct utsname name; + static int call_uname = 1; + static struct utsname name; #endif char *mode; @@ -2292,8 +2392,13 @@ sds genRedisInfoString(char *section) { else mode = "standalone"; if (sections++) info = sdscat(info,"\r\n"); + #ifndef _WIN32 - uname(&name); + if (call_uname) { + /* Uname can be slow and is always the same output. Cache it. */ + uname(&name); + call_uname = 0; + } #endif info = sdscatprintf(info, @@ -2365,8 +2470,16 @@ sds genRedisInfoString(char *section) { if (allsections || defsections || !strcasecmp(section,"memory")) { char hmem[64]; char peak_hmem[64]; + size_t zmalloc_used = zmalloc_used_memory(); - bytesToHuman(hmem,zmalloc_used_memory()); + /* Peak memory is updated from time to time by serverCron() so it + * may happen that the instantaneous value is slightly bigger than + * the peak value. This may confuse users, so we update the peak + * if found smaller than the current memory usage. */ + if (zmalloc_used > server.stat_peak_memory) + server.stat_peak_memory = zmalloc_used; + + bytesToHuman(hmem,zmalloc_used); bytesToHuman(peak_hmem,server.stat_peak_memory); if (sections++) info = sdscat(info,"\r\n"); #ifdef _WIN32 @@ -2386,7 +2499,7 @@ sds genRedisInfoString(char *section) { (long long)server.stat_peak_memory, peak_hmem, ((long long)lua_gc(server.lua,LUA_GCCOUNT,0))*1024LL, - zmalloc_get_fragmentation_ratio(), + zmalloc_get_fragmentation_ratio(server.resident_set_size), ZMALLOC_LIB ); #else @@ -2400,13 +2513,13 @@ sds genRedisInfoString(char *section) { "used_memory_lua:%lld\r\n" "mem_fragmentation_ratio:%.2f\r\n" "mem_allocator:%s\r\n", - zmalloc_used_memory(), + zmalloc_used, hmem, - zmalloc_get_rss(), + server.resident_set_size, server.stat_peak_memory, peak_hmem, ((long long)lua_gc(server.lua,LUA_GCCOUNT,0))*1024LL, - zmalloc_get_fragmentation_ratio(), + zmalloc_get_fragmentation_ratio(server.resident_set_size), ZMALLOC_LIB ); #endif @@ -2464,7 +2577,8 @@ sds genRedisInfoString(char *section) { "aof_rewrite_scheduled:%d\r\n" "aof_last_rewrite_time_sec:%jd\r\n" "aof_current_rewrite_time_sec:%jd\r\n" - "aof_last_bgrewrite_status:%s\r\n", + "aof_last_bgrewrite_status:%s\r\n" + "aof_last_write_status:%s\r\n", server.loading, server.dirty, server.rdb_child_pid != -1, @@ -2479,7 +2593,8 @@ sds genRedisInfoString(char *section) { (intmax_t)server.aof_rewrite_time_last, (intmax_t)((server.aof_child_pid == -1) ? -1 : time(NULL)-server.aof_rewrite_time_start), - (server.aof_lastbgrewrite_status == REDIS_OK) ? "ok" : "err"); + (server.aof_lastbgrewrite_status == REDIS_OK) ? "ok" : "err", + (server.aof_last_write_status == REDIS_OK) ? "ok" : "err"); #endif #ifdef _WIN32 @@ -2687,7 +2802,7 @@ sds genRedisInfoString(char *section) { } if (state == NULL) continue; if (slave->replstate == REDIS_REPL_ONLINE) - lag = time(NULL) - slave->repl_ack_time; + lag = (long)(time(NULL) - slave->repl_ack_time); info = sdscatprintf(info, "slave%d:ip=%s,port=%d,state=%s," @@ -3191,10 +3306,10 @@ int main(int argc, char **argv) { } j++; } + if (configfile) server.configfile = getAbsolutePath(configfile); resetServerSaveParams(); loadServerConfig(configfile,options); sdsfree(options); - if (configfile) server.configfile = getAbsolutePath(configfile); } else { redisLog(REDIS_WARNING, "Warning: no config file specified, using the default config. In order to specify a config file use %s /path/to/%s.conf", argv[0], server.sentinel_mode ? "sentinel" : "redis"); } diff --git a/src/redis.h b/src/redis.h index bbf81302..f861fd5b 100644 --- a/src/redis.h +++ b/src/redis.h @@ -79,6 +79,7 @@ #define REDIS_MIN_HZ 1 #define REDIS_MAX_HZ 500 #define REDIS_SERVERPORT 6379 /* TCP port */ +#define REDIS_TCP_BACKLOG 511 /* TCP listen backlog */ #define REDIS_MAXIDLETIME 0 /* default client timeout: infinite */ #define REDIS_DEFAULT_DBNUM 16 #define REDIS_CONFIGLINE_MAX 1024 @@ -93,7 +94,7 @@ #define REDIS_MAX_LOGMSG_LEN 1024 /* Default maximum length of syslog messages */ #endif #define REDIS_AOF_REWRITE_PERC 100 -#define REDIS_AOF_REWRITE_MIN_SIZE (1024*1024) +#define REDIS_AOF_REWRITE_MIN_SIZE (64*1024*1024) #define REDIS_AOF_REWRITE_ITEMS_PER_CMD 64 #define REDIS_SLOWLOG_LOG_SLOWER_THAN 10000 #define REDIS_SLOWLOG_MAX_LEN 128 @@ -134,6 +135,7 @@ #define REDIS_IP_STR_LEN INET6_ADDRSTRLEN #define REDIS_PEER_ID_LEN (REDIS_IP_STR_LEN+32) /* Must be enough for ip:port */ #define REDIS_BINDADDR_MAX 16 +#define REDIS_MIN_RESERVED_FDS 32 #define ACTIVE_EXPIRE_CYCLE_LOOKUPS_PER_LOOP 20 /* Loopkups per loop. */ #define ACTIVE_EXPIRE_CYCLE_FAST_DURATION 1000 /* Microseconds */ @@ -150,9 +152,9 @@ #define REDIS_LONGSTR_SIZE 21 /* Bytes needed for long -> str */ #define REDIS_AOF_AUTOSYNC_BYTES (1024*1024*32) /* fdatasync every 32MB */ /* When configuring the Redis eventloop, we setup it so that the total number - * of file descriptors we can handle are server.maxclients + FDSET_INCR + * of file descriptors we can handle are server.maxclients + RESERVED_FDS + FDSET_INCR * that is our safety margin. */ -#define REDIS_EVENTLOOP_FDSET_INCR 128 +#define REDIS_EVENTLOOP_FDSET_INCR (REDIS_MIN_RESERVED_FDS+96) /* Hash table parameters */ #define REDIS_HT_MINFILL 10 /* Minimal hash table fill 10% */ @@ -322,6 +324,9 @@ #define REDIS_ZSET_MAX_ZIPLIST_ENTRIES 128 #define REDIS_ZSET_MAX_ZIPLIST_VALUE 64 +/* HyperLogLog defines */ +#define REDIS_DEFAULT_HLL_SPARSE_MAX_BYTES 3000 + /* Sets operations codes */ #define REDIS_OP_UNION 0 #define REDIS_OP_DIFF 1 @@ -388,16 +393,18 @@ * Data types *----------------------------------------------------------------------------*/ +typedef long long mstime_t; /* millisecond time type. */ + /* A redis object, that is a type able to hold a string / list / set */ /* The actual Redis Object */ -#define REDIS_LRU_CLOCK_MAX ((1<<21)-1) /* Max value of obj->lru */ -#define REDIS_LRU_CLOCK_RESOLUTION 10 /* LRU clock resolution in seconds */ +#define REDIS_LRU_BITS 24 +#define REDIS_LRU_CLOCK_MAX ((1<lru */ +#define REDIS_LRU_CLOCK_RESOLUTION 1 /* LRU clock resolution in seconds */ typedef struct redisObject { unsigned type:4; - unsigned notused:2; /* Not used */ unsigned encoding:4; - unsigned lru:22; /* lru time (relative to server.lruclock) */ + unsigned lru:REDIS_LRU_BITS; /* lru time (relative to server.lruclock) */ int refcount; void *ptr; } robj; @@ -525,7 +532,7 @@ struct sharedObjectsStruct { *masterdownerr, *roslaveerr, *execaborterr, *noautherr, *noreplicaserr, *oomerr, *plus, *messagebulk, *pmessagebulk, *subscribebulk, *unsubscribebulk, *psubscribebulk, *punsubscribebulk, *del, *rpop, *lpop, - *lpush, *emptyscan, + *lpush, *emptyscan, *minstring, *maxstring, *select[REDIS_SHARED_SELECT_CMDS], *integers[REDIS_SHARED_INTEGERS], *mbulkhdr[REDIS_SHARED_BULKHDR_LEN], /* "*\r\n" */ @@ -598,8 +605,7 @@ struct redisServer { dict *commands; /* Command table */ dict *orig_commands; /* Command table before command renaming. */ aeEventLoop *el; - unsigned lruclock:22; /* Clock incrementing every minute, for LRU */ - unsigned lruclock_padding:10; + unsigned lruclock:REDIS_LRU_BITS; /* Clock for LRU eviction */ int shutdown_asap; /* SHUTDOWN needed ASAP */ int activerehashing; /* Incremental rehash in serverCron() */ char *requirepass; /* Pass for AUTH command, or NULL */ @@ -610,6 +616,7 @@ struct redisServer { int sentinel_mode; /* True if this instance is a Sentinel. */ /* Networking */ int port; /* TCP listening port */ + int tcp_backlog; /* TCP listen() backlog */ char *bindaddr[REDIS_BINDADDR_MAX]; /* Addresses we should bind to */ int bindaddr_count; /* Number of addresses in server.bindaddr[] */ char *unixsocket; /* UNIX socket path */ @@ -654,6 +661,7 @@ struct redisServer { long long slowlog_entry_id; /* SLOWLOG current entry ID */ long long slowlog_log_slower_than; /* SLOWLOG time limit (to get logged) */ unsigned long slowlog_max_len; /* SLOWLOG max number of items logged */ + size_t resident_set_size; /* RSS sampled in serverCron(). */ /* The following two are used to track instantaneous "load" in terms * of operations per second. */ long long ops_sec_last_sample_time; /* Timestamp of last sample (in ms) */ @@ -697,6 +705,8 @@ struct redisServer { int aof_lastbgrewrite_status; /* REDIS_OK or REDIS_ERR */ unsigned long aof_delayed_fsync; /* delayed AOF fsync() counter */ int aof_rewrite_incremental_fsync;/* fsync incrementally while rewriting? */ + int aof_last_write_status; /* REDIS_OK or REDIS_ERR */ + int aof_last_write_errno; /* Valid if aof_last_write_status is ERR */ /* RDB persistence */ long long dirty; /* Changes to DB from the last save */ long long dirty_before_bgsave; /* Used to restore dirty on failed BGSAVE */ @@ -764,7 +774,7 @@ struct redisServer { list *repl_scriptcache_fifo; /* First in, first out LRU eviction. */ int repl_scriptcache_size; /* Max number of elements. */ /* Limits */ - unsigned int maxclients; /* Max number of simultaneous clients */ + int maxclients; /* Max number of simultaneous clients */ unsigned long long maxmemory; /* Max number of memory bytes to use */ int maxmemory_policy; /* Policy for key eviction */ int maxmemory_samples; /* Pricision of random sampling */ @@ -786,6 +796,7 @@ struct redisServer { size_t set_max_intset_entries; size_t zset_max_ziplist_entries; size_t zset_max_ziplist_value; + size_t hll_sparse_max_bytes; time_t unixtime; /* Unix time sampled every cron cycle. */ long long mstime; /* Like 'unixtime' but with milliseconds resolution. */ /* Pubsub */ @@ -798,8 +809,8 @@ struct redisServer { redisClient *lua_client; /* The "fake client" to query Redis from Lua */ redisClient *lua_caller; /* The client running EVAL right now, or NULL */ dict *lua_scripts; /* A dictionary of SHA1 -> Lua scripts */ - long long lua_time_limit; /* Script timeout in seconds */ - long long lua_time_start; /* Start time of script */ + mstime_t lua_time_limit; /* Script timeout in milliseconds */ + mstime_t lua_time_start; /* Start time of script, milliseconds time */ int lua_write_dirty; /* True if a write command was called during the execution of the current script. */ int lua_random_dirty; /* True if a random command was called during the @@ -1096,18 +1107,24 @@ unsigned long aofRewriteBufferSize(void); /* Sorted sets data type */ -/* Struct to hold a inclusive/exclusive range spec. */ +/* Struct to hold a inclusive/exclusive range spec by score comparison. */ typedef struct { double min, max; int minex, maxex; /* are min or max exclusive? */ } zrangespec; +/* Struct to hold an inclusive/exclusive range spec by lexicographic comparison. */ +typedef struct { + robj *min, *max; /* May be set to shared.(minstring|maxstring) */ + int minex, maxex; /* are min or max exclusive? */ +} zlexrangespec; + zskiplist *zslCreate(void); void zslFree(zskiplist *zsl); zskiplistNode *zslInsert(zskiplist *zsl, double score, robj *obj); unsigned char *zzlInsert(unsigned char *zl, robj *ele, double score); int zslDelete(zskiplist *zsl, double score, robj *obj); -zskiplistNode *zslFirstInRange(zskiplist *zsl, zrangespec range); +zskiplistNode *zslFirstInRange(zskiplist *zsl, zrangespec *range); double zzlGetScore(unsigned char *sptr); void zzlNext(unsigned char *zl, unsigned char **eptr, unsigned char **sptr); void zzlPrev(unsigned char *zl, unsigned char **eptr, unsigned char **sptr); @@ -1142,6 +1159,8 @@ void populateCommandTable(void); void resetCommandTableStats(void); void adjustOpenFilesLimit(void); void closeListeningSockets(int unlink_unix_socket); +void updateCachedTime(void); +void resetServerStats(void); /* Set data type */ robj *setTypeCreate(robj *value); @@ -1216,6 +1235,7 @@ void setKey(redisDb *db, robj *key, robj *val); int dbExists(redisDb *db, robj *key); robj *dbRandomKey(redisDb *db); int dbDelete(redisDb *db, robj *key); +robj *dbUnshareStringValue(redisDb *db, robj *key, robj *o); long long emptyDb(void(callback)(void*)); int selectDb(redisClient *c, int id); void signalModifiedKey(redisDb *db, robj *key); @@ -1334,12 +1354,16 @@ void zincrbyCommand(redisClient *c); void zrangeCommand(redisClient *c); void zrangebyscoreCommand(redisClient *c); void zrevrangebyscoreCommand(redisClient *c); +void zrangebylexCommand(redisClient *c); +void zrevrangebylexCommand(redisClient *c); void zcountCommand(redisClient *c); +void zlexcountCommand(redisClient *c); void zrevrangeCommand(redisClient *c); void zcardCommand(redisClient *c); void zremCommand(redisClient *c); void zscoreCommand(redisClient *c); void zremrangebyscoreCommand(redisClient *c); +void zremrangebylexCommand(redisClient *c); void multiCommand(redisClient *c); void execCommand(redisClient *c); void discardCommand(redisClient *c); @@ -1388,7 +1412,13 @@ void scriptCommand(redisClient *c); void timeCommand(redisClient *c); void bitopCommand(redisClient *c); void bitcountCommand(redisClient *c); +void bitposCommand(redisClient *c); void replconfCommand(redisClient *c); +void pfselftestCommand(redisClient *c); +void pfaddCommand(redisClient *c); +void pfcountCommand(redisClient *c); +void pfmergeCommand(redisClient *c); +void pfdebugCommand(redisClient *c); #if defined(__GNUC__) void *calloc(size_t count, size_t size) __attribute__ ((deprecated)); diff --git a/src/replication.c b/src/replication.c index efb61961..b6879e7f 100644 --- a/src/replication.c +++ b/src/replication.c @@ -687,9 +687,11 @@ void sendBulkToSlave(aeEventLoop *el, int fd, void *privdata, int mask) { return; } if ((nwritten = write(fd,buf,buflen)) == -1) { - redisLog(REDIS_VERBOSE,"Write error sending DB to slave: %s", - strerror(errno)); - freeClient(slave); + if (errno != EAGAIN) { + redisLog(REDIS_WARNING,"Write error sending DB to slave: %s", + strerror(errno)); + freeClient(slave); + } return; } slave->repldboff += nwritten; @@ -701,6 +703,7 @@ void sendBulkToSlave(aeEventLoop *el, int fd, void *privdata, int mask) { slave->repl_ack_time = server.unixtime; if (aeCreateFileEvent(server.el, slave->fd, AE_WRITABLE, sendReplyToClient, slave) == AE_ERR) { + redisLog(REDIS_WARNING,"Unable to register writable event for slave bulk transfer: %s", strerror(errno)); freeClient(slave); return; } diff --git a/src/rio.c b/src/rio.c index c6ec17e6..6e2becca 100644 --- a/src/rio.c +++ b/src/rio.c @@ -93,6 +93,7 @@ static size_t rioFileWrite(rio *r, const void *buf, size_t len) { if (r->io.file.autosync && r->io.file.buffered >= r->io.file.autosync) { + fflush(r->io.file.fp); aof_fsync(fileno(r->io.file.fp)); r->io.file.buffered = 0; } diff --git a/src/scripting.c b/src/scripting.c index 981f9185..2bd1cb3b 100644 --- a/src/scripting.c +++ b/src/scripting.c @@ -441,7 +441,7 @@ void luaMaskCountHook(lua_State *lua, lua_Debug *ar) { REDIS_NOTUSED(ar); REDIS_NOTUSED(lua); - elapsed = (ustime()/1000) - server.lua_time_start; + elapsed = mstime() - server.lua_time_start; if (elapsed >= server.lua_time_limit && server.lua_timedout == 0) { redisLog(REDIS_WARNING,"Lua slow script detected: still in execution after %lld milliseconds. You can try killing the script using the SCRIPT KILL command.",elapsed); server.lua_timedout = 1; @@ -900,7 +900,7 @@ void evalGenericCommand(redisClient *c, int evalsha) { * We set the hook only if the time limit is enabled as the hook will * make the Lua script execution slower. */ server.lua_caller = c; - server.lua_time_start = ustime()/1000; + server.lua_time_start = mstime(); server.lua_kill = 0; if (server.lua_time_limit > 0 && server.masterhost == NULL) { lua_sethook(lua,luaMaskCountHook,LUA_MASKCOUNT,100000); @@ -958,6 +958,7 @@ void evalGenericCommand(redisClient *c, int evalsha) { rewriteClientCommandArgument(c,0, resetRefCount(createStringObject("EVAL",4))); rewriteClientCommandArgument(c,1,script); + forceCommandPropagation(c,REDIS_PROPAGATE_REPL|REDIS_PROPAGATE_AOF); } } } diff --git a/src/sds.c b/src/sds.c index 56fe870d..7c357701 100644 --- a/src/sds.c +++ b/src/sds.c @@ -295,24 +295,37 @@ sds sdscpy(sds s, const char *t) { /* Like sdscatpritf() but gets va_list instead of being variadic. */ sds sdscatvprintf(sds s, const char *fmt, va_list ap) { va_list cpy; - char *buf, *t; - size_t buflen = 16; + char staticbuf[1024], *buf = staticbuf, *t; + size_t buflen = strlen(fmt)*2; - while(1) { + /* We try to start using a static buffer for speed. + * If not possible we revert to heap allocation. */ + if (buflen > sizeof(staticbuf)) { buf = zmalloc(buflen); if (buf == NULL) return NULL; + } else { + buflen = sizeof(staticbuf); + } + + /* Try with buffers two times bigger every time we fail to + * fit the string in the current buffer size. */ + while(1) { buf[buflen-2] = '\0'; va_copy(cpy,ap); vsnprintf(buf, buflen, fmt, cpy); if (buf[buflen-2] != '\0') { - zfree(buf); + if (buf != staticbuf) zfree(buf); buflen *= 2; + buf = zmalloc(buflen); + if (buf == NULL) return NULL; continue; } break; } + + /* Finally concat the obtained string to the SDS string and return it. */ t = sdscat(s, buf); - zfree(buf); + if (buf != staticbuf) zfree(buf); return t; } diff --git a/src/sentinel.c b/src/sentinel.c index 1576c0f7..4bf466f1 100644 --- a/src/sentinel.c +++ b/src/sentinel.c @@ -54,8 +54,6 @@ extern char **environ; /* ======================== Sentinel global state =========================== */ -typedef long long mstime_t; /* millisecond time type. */ - /* Address object, used to describe an ip:port pair. */ typedef struct sentinelAddr { char *ip; @@ -90,12 +88,13 @@ typedef struct sentinelAddr { #define SENTINEL_TILT_TRIGGER 2000 #define SENTINEL_TILT_PERIOD (SENTINEL_PING_PERIOD*30) #define SENTINEL_DEFAULT_SLAVE_PRIORITY 100 -#define SENTINEL_SLAVE_RECONF_RETRY_PERIOD 10000 +#define SENTINEL_SLAVE_RECONF_TIMEOUT 10000 #define SENTINEL_DEFAULT_PARALLEL_SYNCS 1 #define SENTINEL_MIN_LINK_RECONNECT_PERIOD 15000 #define SENTINEL_DEFAULT_FAILOVER_TIMEOUT (60*3*1000) #define SENTINEL_MAX_PENDING_COMMANDS 100 #define SENTINEL_ELECTION_TIMEOUT 10000 +#define SENTINEL_MAX_DESYNC 1000 /* Failover machine different states. */ #define SENTINEL_FAILOVER_STATE_NONE 0 /* No failover in progress. */ @@ -140,6 +139,10 @@ typedef struct sentinelRedisInstance { mstime_t pc_last_activity; /* Last time we received any message. */ mstime_t last_avail_time; /* Last time the instance replied to ping with a reply we consider valid. */ + mstime_t last_ping_time; /* Last time a pending ping was sent in the + context of the current command connection + with the instance. 0 if still not sent or + if pong already received. */ mstime_t last_pong_time; /* Last time the instance replied to ping, whatever the reply was. That's used to check if the link is idle and must be reconnected. */ @@ -377,6 +380,8 @@ void sentinelDiscardReplyCallback(redisAsyncContext *c, void *reply, void *privd int sentinelSendSlaveOf(sentinelRedisInstance *ri, char *host, int port); char *sentinelVoteLeader(sentinelRedisInstance *master, uint64_t req_epoch, char *req_runid, uint64_t *leader_epoch); void sentinelFlushConfig(void); +void sentinelGenerateInitialMonitorEvents(void); +int sentinelSendPing(sentinelRedisInstance *ri); /* ========================= Dictionary types =============================== */ @@ -419,6 +424,7 @@ dictType leaderVotesDictType = { void sentinelCommand(redisClient *c); void sentinelInfoCommand(redisClient *c); void sentinelSetCommand(redisClient *c); +void sentinelPublishCommand(redisClient *c); struct redisCommand sentinelcmds[] = { {"ping",pingCommand,1,"",0,NULL,0,0,0,0,0}, @@ -427,7 +433,9 @@ struct redisCommand sentinelcmds[] = { {"unsubscribe",unsubscribeCommand,-1,"",0,NULL,0,0,0,0,0}, {"psubscribe",psubscribeCommand,-2,"",0,NULL,0,0,0,0,0}, {"punsubscribe",punsubscribeCommand,-1,"",0,NULL,0,0,0,0,0}, - {"info",sentinelInfoCommand,-1,"",0,NULL,0,0,0,0,0} + {"publish",sentinelPublishCommand,3,"",0,NULL,0,0,0,0,0}, + {"info",sentinelInfoCommand,-1,"",0,NULL,0,0,0,0,0}, + {"shutdown",shutdownCommand,-1,"",0,NULL,0,0,0,0,0} }; /* This function overwrites a few normal Redis config default with Sentinel @@ -466,10 +474,20 @@ void initSentinel(void) { void sentinelIsRunning(void) { redisLog(REDIS_WARNING,"Sentinel runid is %s", server.runid); - if (server.configfile == NULL || access(server.configfile,W_OK) == -1) { - redisLog(REDIS_WARNING,"Sentinel started without a config file, or config file not writable. Exiting..."); + if (server.configfile == NULL) { + redisLog(REDIS_WARNING, + "Sentinel started without a config file. Exiting..."); + exit(1); + } else if (access(server.configfile,W_OK) == -1) { + redisLog(REDIS_WARNING, + "Sentinel config file %s is not writable: %s. Exiting...", + server.configfile,strerror(errno)); exit(1); } + + /* We want to generate a +monitor event for every configured master + * at startup. */ + sentinelGenerateInitialMonitorEvents(); } /* ============================== sentinelAddr ============================== */ @@ -601,6 +619,22 @@ void sentinelEvent(int level, char *type, sentinelRedisInstance *ri, } } +/* This function is called only at startup and is used to generate a + * +monitor event for every configured master. The same events are also + * generated when a master to monitor is added at runtime via the + * SENTINEL MONITOR command. */ +void sentinelGenerateInitialMonitorEvents(void) { + dictIterator *di; + dictEntry *de; + + di = dictGetIterator(sentinel.masters); + while((de = dictNext(di)) != NULL) { + sentinelRedisInstance *ri = dictGetVal(de); + sentinelEvent(REDIS_WARNING,"+monitor",ri,"%@ quorum %d",ri->quorum); + } + dictReleaseIterator(di); +} + /* ============================ script execution ============================ */ /* Release a script job structure and all the associated data. */ @@ -696,7 +730,9 @@ void sentinelRunPendingScripts(void) { (ln = listNext(&li)) != NULL) { sentinelScriptJob *sj = ln->value; +#ifndef _WIN32 pid_t pid; +#endif /* Skip if already running. */ if (sj->flags & SENTINEL_SCRIPT_RUNNING) continue; @@ -715,7 +751,7 @@ void sentinelRunPendingScripts(void) { int j = 1; int pos = 0; while(sj->argv[j]) { - int arglen = strlen(sj->argv[j]); + int arglen = (int)strlen(sj->argv[j]); memcpy(args+pos, sj->argv[j], arglen); pos += arglen; memcpy(args+pos, " ", 1); @@ -1038,6 +1074,11 @@ sentinelRedisInstance *createSentinelRedisInstance(char *name, int flags, char * ri->cc_conn_time = 0; ri->pc_conn_time = 0; ri->pc_last_activity = 0; + /* We set the last_ping_time to "now" even if we actually don't have yet + * a connection with the node, nor we sent a ping. + * This is useful to detect a timeout in case we'll not be able to connect + * with the node at all. */ + ri->last_ping_time = mstime(); ri->last_avail_time = mstime(); ri->last_pong_time = mstime(); ri->last_pub_time = mstime(); @@ -1274,6 +1315,7 @@ void sentinelResetMaster(sentinelRedisInstance *ri, int flags) { sdsfree(ri->slave_master_host); ri->runid = NULL; ri->slave_master_host = NULL; + ri->last_ping_time = mstime(); ri->last_avail_time = mstime(); ri->last_pong_time = mstime(); ri->role_reported_time = mstime(); @@ -1400,6 +1442,24 @@ sentinelAddr *sentinelGetCurrentMasterAddress(sentinelRedisInstance *master) { } } +/* This function sets the down_after_period field value in 'master' to all + * the slaves and sentinel instances connected to this master. */ +void sentinelPropagateDownAfterPeriod(sentinelRedisInstance *master) { + dictIterator *di; + dictEntry *de; + int j; + dict *d[] = {master->slaves, master->sentinels, NULL}; + + for (j = 0; d[j]; j++) { + di = dictGetIterator(d[j]); + while((de = dictNext(di)) != NULL) { + sentinelRedisInstance *ri = dictGetVal(de); + ri->down_after_period = master->down_after_period; + } + dictReleaseIterator(di); + } +} + /* ============================ Config handling ============================= */ char *sentinelHandleConfiguration(char **argv, int argc) { sentinelRedisInstance *ri; @@ -1425,6 +1485,7 @@ char *sentinelHandleConfiguration(char **argv, int argc) { ri->down_after_period = atoi(argv[2]); if (ri->down_after_period <= 0) return "negative or zero time parameter."; + sentinelPropagateDownAfterPeriod(ri); } else if (!strcasecmp(argv[0],"failover-timeout") && argc == 3) { /* failover-timeout */ ri = sentinelGetMasterByName(argv[1]); @@ -1457,13 +1518,26 @@ char *sentinelHandleConfiguration(char **argv, int argc) { ri = sentinelGetMasterByName(argv[1]); if (!ri) return "No such master with specified name."; ri->auth_pass = sdsnew(argv[2]); + } else if (!strcasecmp(argv[0],"current-epoch") && argc == 2) { + /* current-epoch */ + unsigned long long current_epoch = strtoull(argv[1],NULL,10); + if (current_epoch > sentinel.current_epoch) + sentinel.current_epoch = current_epoch; } else if (!strcasecmp(argv[0],"config-epoch") && argc == 3) { /* config-epoch */ ri = sentinelGetMasterByName(argv[1]); if (!ri) return "No such master with specified name."; ri->config_epoch = strtoull(argv[2],NULL,10); + /* The following update of current_epoch is not really useful as + * now the current epoch is persisted on the config file, but + * we leave this check here for redundancy. */ if (ri->config_epoch > sentinel.current_epoch) sentinel.current_epoch = ri->config_epoch; + } else if (!strcasecmp(argv[0],"leader-epoch") && argc == 3) { + /* leader-epoch */ + ri = sentinelGetMasterByName(argv[1]); + if (!ri) return "No such master with specified name."; + ri->leader_epoch = strtoull(argv[2],NULL,10); } else if (!strcasecmp(argv[0],"known-slave") && argc == 4) { sentinelRedisInstance *slave; @@ -1502,13 +1576,13 @@ char *sentinelHandleConfiguration(char **argv, int argc) { void rewriteConfigSentinelOption(struct rewriteConfigState *state) { dictIterator *di, *di2; dictEntry *de; + sds line; /* For every master emit a "sentinel monitor" config entry. */ di = dictGetIterator(sentinel.masters); while((de = dictNext(di)) != NULL) { sentinelRedisInstance *master, *ri; sentinelAddr *master_addr; - sds line; /* sentinel monitor */ master = dictGetVal(de); @@ -1572,6 +1646,12 @@ void rewriteConfigSentinelOption(struct rewriteConfigState *state) { master->name, (unsigned long long) master->config_epoch); rewriteConfigRewriteLine(state,"sentinel",line,1); + /* sentinel leader-epoch */ + line = sdscatprintf(sdsempty(), + "sentinel leader-epoch %s %llu", + master->name, (unsigned long long) master->leader_epoch); + rewriteConfigRewriteLine(state,"sentinel",line,1); + /* sentinel known-slave */ di2 = dictGetIterator(master->slaves); while((de = dictNext(di2)) != NULL) { @@ -1607,6 +1687,12 @@ void rewriteConfigSentinelOption(struct rewriteConfigState *state) { } dictReleaseIterator(di2); } + + /* sentinel current-epoch is a global state valid for all the masters. */ + line = sdscatprintf(sdsempty(), + "sentinel current-epoch %llu", (unsigned long long) sentinel.current_epoch); + rewriteConfigRewriteLine(state,"sentinel",line,1); + dictReleaseIterator(di); } @@ -1618,21 +1704,23 @@ void rewriteConfigSentinelOption(struct rewriteConfigState *state) { * * On failure the function logs a warning on the Redis log. */ void sentinelFlushConfig(void) { - int fd; + int fd = -1; int saved_hz = server.hz; + int rewrite_status; server.hz = REDIS_DEFAULT_HZ; - if (rewriteConfig(server.configfile) != -1) { - /* Rewrite succeded, fsync it. */ - if ((fd = open(server.configfile,O_RDONLY, 0)) != -1) { - fsync(fd); - close(fd); - } - } else { - redisLog(REDIS_WARNING,"WARNING: Senitnel was not able to save the new configuration on disk!!!: %s", strerror(errno)); - } + rewrite_status = rewriteConfig(server.configfile); server.hz = saved_hz; + + if (rewrite_status == -1) goto werr; + if ((fd = open(server.configfile,O_RDONLY,0)) == -1) goto werr; + if (fsync(fd) == -1) goto werr; + if (close(fd) == EOF) goto werr; return; + +werr: + if (fd != -1) close(fd); + redisLog(REDIS_WARNING,"WARNING: Sentinel was not able to save the new configuration on disk!!!: %s", strerror(errno)); } /* ====================== hiredis connection handling ======================= */ @@ -1703,6 +1791,23 @@ void sentinelSendAuthIfNeeded(sentinelRedisInstance *ri, redisAsyncContext *c) { } } +/* Use CLIENT SETNAME to name the connection in the Redis instance as + * sentinel-- + * The connection type is "cmd" or "pubsub" as specified by 'type'. + * + * This makes it possible to list all the sentinel instances connected + * to a Redis servewr with CLIENT LIST, grepping for a specific name format. */ +void sentinelSetClientName(sentinelRedisInstance *ri, redisAsyncContext *c, char *type) { + char name[64]; + + snprintf(name,sizeof(name),"sentinel-%.8s-%s",server.runid,type); + if (redisAsyncCommand(c, sentinelDiscardReplyCallback, NULL, + "CLIENT SETNAME %s", name) == REDIS_OK) + { + ri->pending_commands++; + } +} + /* Create the async connections for the specified instance if the instance * is disconnected. Note that the SRI_DISCONNECTED flag is set even if just * one of the two links (commands and pub/sub) is missing. */ @@ -1725,6 +1830,10 @@ void sentinelReconnectInstance(sentinelRedisInstance *ri) { redisAsyncSetDisconnectCallback(ri->cc, sentinelDisconnectCallback); sentinelSendAuthIfNeeded(ri,ri->cc); + sentinelSetClientName(ri,ri->cc,"cmd"); + + /* Send a PING ASAP when reconnecting. */ + sentinelSendPing(ri); } } /* Pub / Sub */ @@ -1745,6 +1854,7 @@ void sentinelReconnectInstance(sentinelRedisInstance *ri) { redisAsyncSetDisconnectCallback(ri->pc, sentinelDisconnectCallback); sentinelSendAuthIfNeeded(ri,ri->pc); + sentinelSetClientName(ri,ri->pc,"pubsub"); /* Now we subscribe to the Sentinels "Hello" channel. */ retval = redisAsyncCommand(ri->pc, sentinelReceiveHelloMessages, NULL, "SUBSCRIBE %s", @@ -1789,7 +1899,7 @@ void sentinelRefreshInstanceInfo(sentinelRedisInstance *ri, const char *info) { ri->master_link_down_time = 0; /* Process line by line. */ - lines = sdssplitlen(info,strlen(info),"\r\n",2,&numlines); + lines = sdssplitlen(info,(int)strlen(info),"\r\n",2,&numlines); for (j = 0; j < numlines; j++) { sentinelRedisInstance *slave; sds l = lines[j]; @@ -1908,8 +2018,20 @@ void sentinelRefreshInstanceInfo(sentinelRedisInstance *ri, const char *info) { ri->role_reported_time = mstime(); ri->role_reported = role; if (role == SRI_SLAVE) ri->slave_conf_change_time = mstime(); + /* Log the event with +role-change if the new role is coherent or + * with -role-change if there is a mismatch with the current config. */ + sentinelEvent(REDIS_VERBOSE, + ((ri->flags & (SRI_MASTER|SRI_SLAVE)) == role) ? + "+role-change" : "-role-change", + ri, "%@ new reported role is %s", + role == SRI_MASTER ? "master" : "slave", + ri->flags & SRI_MASTER ? "master" : "slave"); } + /* None of the following conditions are processed when in tilt mode, so + * return asap. */ + if (sentinel.tilt) return; + /* Handle master -> slave role switch. */ if ((ri->flags & SRI_MASTER) && role == SRI_SLAVE) { /* Nothing to do, but masters claiming to be slaves are @@ -1921,8 +2043,7 @@ void sentinelRefreshInstanceInfo(sentinelRedisInstance *ri, const char *info) { if ((ri->flags & SRI_SLAVE) && role == SRI_MASTER) { /* If this is a promoted slave we can change state to the * failover state machine. */ - if (!sentinel.tilt && - (ri->master->flags & SRI_FAILOVER_IN_PROGRESS) && + if ((ri->master->flags & SRI_FAILOVER_IN_PROGRESS) && (ri->master->failover_state == SENTINEL_FAILOVER_STATE_WAIT_PROMOTION)) { @@ -1940,7 +2061,7 @@ void sentinelRefreshInstanceInfo(sentinelRedisInstance *ri, const char *info) { ri->master,"%@"); sentinelCallClientReconfScript(ri->master,SENTINEL_LEADER, "start",ri->master->addr,ri->addr); - } else if (!sentinel.tilt) { + } else { /* A slave turned into a master. We want to force our view and * reconfigure as slave. Wait some time after the change before * going forward, to receive new configs if any. */ @@ -1960,7 +2081,7 @@ void sentinelRefreshInstanceInfo(sentinelRedisInstance *ri, const char *info) { } /* Handle slaves replicating to a different master address. */ - if ((ri->flags & SRI_SLAVE) && !sentinel.tilt && + if ((ri->flags & SRI_SLAVE) && role == SRI_SLAVE && (ri->slave_master_port != ri->master->addr->port || strcasecmp(ri->slave_master_host,ri->master->addr->ip))) @@ -1981,10 +2102,6 @@ void sentinelRefreshInstanceInfo(sentinelRedisInstance *ri, const char *info) { } } - /* None of the following conditions are processed when in tilt mode, so - * return asap. */ - if (sentinel.tilt) return; - /* Detect if the slave that is in the process of being reconfigured * changed state. */ if ((ri->flags & SRI_SLAVE) && role == SRI_SLAVE && @@ -2051,6 +2168,7 @@ void sentinelPingReplyCallback(redisAsyncContext *c, void *reply, void *privdata strncmp(r->str,"MASTERDOWN",10) == 0) { ri->last_avail_time = mstime(); + ri->last_ping_time = 0; /* Flag the pong as received. */ } else { /* Send a SCRIPT KILL command if the instance appears to be * down because of a busy script. */ @@ -2085,17 +2203,107 @@ void sentinelPublishReplyCallback(redisAsyncContext *c, void *reply, void *privd ri->last_pub_time = mstime(); } +/* Process an hello message received via Pub/Sub in master or slave instance, + * or sent directly to this sentinel via the (fake) PUBLISH command of Sentinel. + * + * If the master name specified in the message is not known, the message is + * discareded. */ +void sentinelProcessHelloMessage(char *hello, int hello_len) { + /* Format is composed of 8 tokens: + * 0=ip,1=port,2=runid,3=current_epoch,4=master_name, + * 5=master_ip,6=master_port,7=master_config_epoch. */ + int numtokens, port, removed, master_port; + uint64_t current_epoch, master_config_epoch; + char **token = sdssplitlen(hello, hello_len, ",", 1, &numtokens); + sentinelRedisInstance *si, *master; + + if (numtokens == 8) { + /* Obtain a reference to the master this hello message is about */ + master = sentinelGetMasterByName(token[4]); + if (!master) goto cleanup; /* Unknown master, skip the message. */ + + /* First, try to see if we already have this sentinel. */ + port = atoi(token[1]); + master_port = atoi(token[6]); + si = getSentinelRedisInstanceByAddrAndRunID( + master->sentinels,token[0],port,token[2]); + current_epoch = strtoull(token[3],NULL,10); + master_config_epoch = strtoull(token[7],NULL,10); + + if (!si) { + /* If not, remove all the sentinels that have the same runid + * OR the same ip/port, because it's either a restart or a + * network topology change. */ + removed = removeMatchingSentinelsFromMaster(master,token[0],port, + token[2]); + if (removed) { + sentinelEvent(REDIS_NOTICE,"-dup-sentinel",master, + "%@ #duplicate of %s:%d or %s", + token[0],port,token[2]); + } + + /* Add the new sentinel. */ + si = createSentinelRedisInstance(NULL,SRI_SENTINEL, + token[0],port,master->quorum,master); + if (si) { + sentinelEvent(REDIS_NOTICE,"+sentinel",si,"%@"); + /* The runid is NULL after a new instance creation and + * for Sentinels we don't have a later chance to fill it, + * so do it now. */ + si->runid = sdsnew(token[2]); + sentinelFlushConfig(); + } + } + + /* Update local current_epoch if received current_epoch is greater.*/ + if (current_epoch > sentinel.current_epoch) { + sentinel.current_epoch = current_epoch; + sentinelFlushConfig(); + sentinelEvent(REDIS_WARNING,"+new-epoch",master,"%llu", + (unsigned long long) sentinel.current_epoch); + } + + /* Update master info if received configuration is newer. */ + if (master->config_epoch < master_config_epoch) { + master->config_epoch = master_config_epoch; + if (master_port != master->addr->port || + strcmp(master->addr->ip, token[5])) + { + sentinelAddr *old_addr; + + sentinelEvent(REDIS_WARNING,"+switch-master", + master,"%s %s %d %s %d", + master->name, + master->addr->ip, master->addr->port, + token[5], master_port); + + old_addr = dupSentinelAddr(master->addr); + sentinelResetMasterAndChangeAddress(master, token[5], master_port); + sentinelCallClientReconfScript(master, + SENTINEL_OBSERVER,"start", + old_addr,master->addr); + releaseSentinelAddr(old_addr); + } + } + + /* Update the state of the Sentinel. */ + if (si) si->last_hello_time = mstime(); + } + +cleanup: + sdsfreesplitres(token,numtokens); +} + + /* This is our Pub/Sub callback for the Hello channel. It's useful in order * to discover other sentinels attached at the same master. */ void sentinelReceiveHelloMessages(redisAsyncContext *c, void *reply, void *privdata) { - sentinelRedisInstance *ri = c->data, *master; + sentinelRedisInstance *ri = c->data; redisReply *r; if (!reply || !ri) return; r = reply; - master = (ri->flags & SRI_MASTER) ? ri : ri->master; - /* Update the last activity in the pubsub channel. Note that since we * receive our messages as well this timestamp can be used to detect * if the link is probably disconnected even if it seems otherwise. */ @@ -2113,91 +2321,7 @@ void sentinelReceiveHelloMessages(redisAsyncContext *c, void *reply, void *privd /* We are not interested in meeting ourselves */ if (strstr(r->element[2]->str,server.runid) != NULL) return; - { - /* Format is composed of 8 tokens: - * 0=ip,1=port,2=runid,3=current_epoch,4=master_name, - * 5=master_ip,6=master_port,7=master_config_epoch. */ - int numtokens, port, removed, master_port; - uint64_t current_epoch, master_config_epoch; - char **token = sdssplitlen(r->element[2]->str, - r->element[2]->len, - ",",1,&numtokens); - sentinelRedisInstance *si; - - if (numtokens == 8) { - sentinelRedisInstance *msgmaster; - - /* First, try to see if we already have this sentinel. */ - port = atoi(token[1]); - master_port = atoi(token[6]); - si = getSentinelRedisInstanceByAddrAndRunID( - master->sentinels,token[0],port,token[2]); - current_epoch = strtoull(token[3],NULL,10); - master_config_epoch = strtoull(token[7],NULL,10); - - if (!si) { - /* If not, remove all the sentinels that have the same runid - * OR the same ip/port, because it's either a restart or a - * network topology change. */ - removed = removeMatchingSentinelsFromMaster(master,token[0],port, - token[2]); - if (removed) { - sentinelEvent(REDIS_NOTICE,"-dup-sentinel",master, - "%@ #duplicate of %s:%d or %s", - token[0],port,token[2]); - } - - /* Add the new sentinel. */ - si = createSentinelRedisInstance(NULL,SRI_SENTINEL, - token[0],port,master->quorum,master); - if (si) { - sentinelEvent(REDIS_NOTICE,"+sentinel",si,"%@"); - /* The runid is NULL after a new instance creation and - * for Sentinels we don't have a later chance to fill it, - * so do it now. */ - si->runid = sdsnew(token[2]); - sentinelFlushConfig(); - } - } - - /* Update local current_epoch if received current_epoch is greater.*/ - if (current_epoch > sentinel.current_epoch) { - sentinel.current_epoch = current_epoch; - sentinelEvent(REDIS_WARNING,"+new-epoch",ri,"%llu", - (unsigned long long) sentinel.current_epoch); - } - - /* Update master info if received configuration is newer. */ - if ((msgmaster = sentinelGetMasterByName(token[4])) != NULL) { - if (msgmaster->config_epoch < master_config_epoch) { - msgmaster->config_epoch = master_config_epoch; - if (master_port != msgmaster->addr->port || - strcmp(msgmaster->addr->ip, token[5])) - { - sentinelAddr *old_addr; - - sentinelEvent(REDIS_WARNING,"+switch-master", - msgmaster,"%s %s %d %s %d", - msgmaster->name, - msgmaster->addr->ip, msgmaster->addr->port, - token[5], master_port); - - old_addr = dupSentinelAddr(msgmaster->addr); - sentinelResetMasterAndChangeAddress(msgmaster, - token[5], master_port); - sentinelCallClientReconfScript(msgmaster, - SENTINEL_OBSERVER,"start", - old_addr,msgmaster->addr); - releaseSentinelAddr(old_addr); - } - } - } - - /* Update the state of the Sentinel. */ - if (si) si->last_hello_time = mstime(); - } - sdsfreesplitres(token,numtokens); - } + sentinelProcessHelloMessage(r->element[2]->str, r->element[2]->len); } /* Send an "Hello" message via Pub/Sub to the specified 'ri' Redis @@ -2239,11 +2363,31 @@ int sentinelSendHello(sentinelRedisInstance *ri) { return REDIS_OK; } +/* Send a PING to the specified instance and refresh the last_ping_time + * if it is zero (that is, if we received a pong for the previous ping). + * + * On error zero is returned, and we can't consider the PING command + * queued in the connection. */ +int sentinelSendPing(sentinelRedisInstance *ri) { + int retval = redisAsyncCommand(ri->cc, + sentinelPingReplyCallback, NULL, "PING"); + if (retval == REDIS_OK) { + ri->pending_commands++; + /* We update the ping time only if we received the pong for + * the previous ping, otherwise we are technically waiting + * since the first ping that did not received a reply. */ + if (ri->last_ping_time == 0) ri->last_ping_time = mstime(); + return 1; + } else { + return 0; + } +} + /* Send periodic PING, INFO, and PUBLISH to the Hello channel to * the specified master or slave instance. */ -void sentinelPingInstance(sentinelRedisInstance *ri) { +void sentinelSendPeriodicCommands(sentinelRedisInstance *ri) { mstime_t now = mstime(); - mstime_t info_period; + mstime_t info_period, ping_period; int retval; /* Return ASAP if we have already a PING or INFO already pending, or @@ -2269,6 +2413,12 @@ void sentinelPingInstance(sentinelRedisInstance *ri) { info_period = SENTINEL_INFO_PERIOD; } + /* We ping instances every time the last received pong is older than + * the configured 'down-after-milliseconds' time, but every second + * anyway if 'down-after-milliseconds' is greater than 1 second. */ + ping_period = ri->down_after_period; + if (ping_period > SENTINEL_PING_PERIOD) ping_period = SENTINEL_PING_PERIOD; + if ((ri->flags & SRI_SENTINEL) == 0 && (ri->info_refresh == 0 || (now - ri->info_refresh) > info_period)) @@ -2276,18 +2426,12 @@ void sentinelPingInstance(sentinelRedisInstance *ri) { /* Send INFO to masters and slaves, not sentinels. */ retval = redisAsyncCommand(ri->cc, sentinelInfoReplyCallback, NULL, "INFO"); - if (retval != REDIS_OK) return; - ri->pending_commands++; - } else if ((now - ri->last_pong_time) > SENTINEL_PING_PERIOD) { + if (retval == REDIS_OK) ri->pending_commands++; + } else if ((now - ri->last_pong_time) > ping_period) { /* Send PING to all the three kinds of instances. */ - retval = redisAsyncCommand(ri->cc, - sentinelPingReplyCallback, NULL, "PING"); - if (retval != REDIS_OK) return; - ri->pending_commands++; - } else if ((ri->flags & SRI_SENTINEL) == 0 && - (now - ri->last_pub_time) > SENTINEL_PUBLISH_PERIOD) - { - /* PUBLISH hello messages to masters and slaves. */ + sentinelSendPing(ri); + } else if ((now - ri->last_pub_time) > SENTINEL_PUBLISH_PERIOD) { + /* PUBLISH hello messages to all the three kinds of instances. */ sentinelSendHello(ri); } } @@ -2361,6 +2505,11 @@ void addReplySentinelRedisInstance(redisClient *c, sentinelRedisInstance *ri) { fields++; } + addReplyBulkCString(c,"last-ping-sent"); + addReplyBulkLongLong(c, + ri->last_ping_time ? (mstime() - ri->last_ping_time) : 0); + fields++; + addReplyBulkCString(c,"last-ok-ping-reply"); addReplyBulkLongLong(c,mstime() - ri->last_avail_time); fields++; @@ -2381,6 +2530,10 @@ void addReplySentinelRedisInstance(redisClient *c, sentinelRedisInstance *ri) { fields++; } + addReplyBulkCString(c,"down-after-milliseconds"); + addReplyBulkLongLong(c,ri->down_after_period); + fields++; + /* Masters and Slaves */ if (ri->flags & (SRI_MASTER|SRI_SLAVE)) { addReplyBulkCString(c,"info-refresh"); @@ -2415,10 +2568,6 @@ void addReplySentinelRedisInstance(redisClient *c, sentinelRedisInstance *ri) { addReplyBulkLongLong(c,ri->quorum); fields++; - addReplyBulkCString(c,"down-after-milliseconds"); - addReplyBulkLongLong(c,ri->down_after_period); - fields++; - addReplyBulkCString(c,"failover-timeout"); addReplyBulkLongLong(c,ri->failover_timeout); fields++; @@ -2495,7 +2644,7 @@ void addReplyDictOfRedisInstances(redisClient *c, dict *instances) { dictEntry *de; di = dictGetIterator(instances); - addReplyMultiBulkLen(c,dictSize(instances)); + addReplyMultiBulkLen(c,(long)dictSize(instances)); while((de = dictNext(di)) != NULL) { sentinelRedisInstance *ri = dictGetVal(de); @@ -2529,6 +2678,7 @@ void sentinelCommand(redisClient *c) { /* SENTINEL MASTER */ sentinelRedisInstance *ri; + if (c->argc != 3) goto numargserr; if ((ri = sentinelGetMasterByNameOrReplyError(c,c->argv[2])) == NULL) return; addReplySentinelRedisInstance(c,ri); @@ -2598,8 +2748,6 @@ void sentinelCommand(redisClient *c) { ri = sentinelGetMasterByName(c->argv[2]->ptr); if (ri == NULL) { addReply(c,shared.nullmultibulk); - } else if (ri->info_refresh == 0) { - addReplySds(c,sdsnew("-IDONTKNOW I have not enough information to reply. Please ask another Sentinel.\r\n")); } else { sentinelAddr *addr = sentinelGetCurrentMasterAddress(ri); @@ -2634,6 +2782,7 @@ void sentinelCommand(redisClient *c) { sentinelPendingScriptsCommand(c); } else if (!strcasecmp(c->argv[1]->ptr,"monitor")) { /* SENTINEL MONITOR */ + sentinelRedisInstance *ri; long quorum, port; char buf[32]; @@ -2649,9 +2798,11 @@ void sentinelCommand(redisClient *c) { addReplyError(c,"Invalid IP address specified"); return; } - if (createSentinelRedisInstance(c->argv[2]->ptr,SRI_MASTER, - c->argv[3]->ptr,port,quorum,NULL) == NULL) - { + + /* Parameters are valid. Try to create the master instance. */ + ri = createSentinelRedisInstance(c->argv[2]->ptr,SRI_MASTER, + c->argv[3]->ptr,port,quorum,NULL); + if (ri == NULL) { switch(errno) { case EBUSY: addReplyError(c,"Duplicated master name"); @@ -2665,6 +2816,7 @@ void sentinelCommand(redisClient *c) { } } else { sentinelFlushConfig(); + sentinelEvent(REDIS_WARNING,"+monitor",ri,"%@ quorum %d",ri->quorum); addReply(c,shared.ok); } } else if (!strcasecmp(c->argv[1]->ptr,"remove")) { @@ -2673,6 +2825,7 @@ void sentinelCommand(redisClient *c) { if ((ri = sentinelGetMasterByNameOrReplyError(c,c->argv[2])) == NULL) return; + sentinelEvent(REDIS_WARNING,"-monitor",ri,"%@"); dictDelete(sentinel.masters,c->argv[2]->ptr); sentinelFlushConfig(); addReply(c,shared.ok); @@ -2703,8 +2856,9 @@ void sentinelInfoCommand(redisClient *c) { } if (!strcasecmp(section,"server") || defsections) { - sds serversection = genRedisInfoString("server"); + sds serversection; if (sections++) info = sdscat(info,"\r\n"); + serversection = genRedisInfoString("server"); info = sdscatlen(info,serversection,sdslen(serversection)); sdsfree(serversection); } @@ -2771,6 +2925,7 @@ void sentinelSetCommand(redisClient *c) { if (getLongLongFromObject(o,&ll) == REDIS_ERR || ll <= 0) goto badfmt; ri->down_after_period = ll; + sentinelPropagateDownAfterPeriod(ri); changes++; } else if (!strcasecmp(option,"failover-timeout")) { /* failover-timeout */ @@ -2782,7 +2937,7 @@ void sentinelSetCommand(redisClient *c) { /* parallel-syncs */ if (getLongLongFromObject(o,&ll) == REDIS_ERR || ll <= 0) goto badfmt; - ri->parallel_syncs = ll; + ri->parallel_syncs = (int)ll; changes++; } else if (!strcasecmp(option,"notification-script")) { /* notification-script */ @@ -2812,12 +2967,19 @@ void sentinelSetCommand(redisClient *c) { sdsfree(ri->auth_pass); ri->auth_pass = strlen(value) ? sdsnew(value) : NULL; changes++; + } else if (!strcasecmp(option,"quorum")) { + /* quorum */ + if (getLongLongFromObject(o,&ll) == REDIS_ERR || ll <= 0) + goto badfmt; + ri->quorum = (int)ll; + changes++; } else { addReplyErrorFormat(c,"Unknown option '%s' for SENTINEL SET", option); if (changes) sentinelFlushConfig(); return; } + sentinelEvent(REDIS_WARNING,"+set",ri,"%@ %s %s",option,value); } if (changes) sentinelFlushConfig(); @@ -2830,20 +2992,42 @@ badfmt: /* Bad format errors */ value, option); } +/* Our fake PUBLISH command: it is actually useful only to receive hello messages + * from the other sentinel instances, and publishing to a channel other than + * SENTINEL_HELLO_CHANNEL is forbidden. + * + * Because we have a Sentinel PUBLISH, the code to send hello messages is the same + * for all the three kind of instances: masters, slaves, sentinels. */ +void sentinelPublishCommand(redisClient *c) { + if (strcmp(c->argv[1]->ptr,SENTINEL_HELLO_CHANNEL)) { + addReplyError(c, "Only HELLO messages are accepted by Sentinel instances."); + return; + } + sentinelProcessHelloMessage(c->argv[2]->ptr,(int)sdslen(c->argv[2]->ptr)); + addReplyLongLong(c,1); +} + /* ===================== SENTINEL availability checks ======================= */ /* Is this instance down from our point of view? */ void sentinelCheckSubjectivelyDown(sentinelRedisInstance *ri) { - mstime_t elapsed = mstime() - ri->last_avail_time; + mstime_t elapsed = 0; + + if (ri->last_ping_time) + elapsed = mstime() - ri->last_ping_time; /* Check if we are in need for a reconnection of one of the * links, because we are detecting low activity. * * 1) Check if the command link seems connected, was connected not less - * than SENTINEL_MIN_LINK_RECONNECT_PERIOD, but still we have an - * idle time that is greater than down_after_period / 2 seconds. */ + * than SENTINEL_MIN_LINK_RECONNECT_PERIOD, but still we have a + * pending ping for more than half the timeout. */ if (ri->cc && (mstime() - ri->cc_conn_time) > SENTINEL_MIN_LINK_RECONNECT_PERIOD && + ri->last_ping_time != 0 && /* Ther is a pending ping... */ + /* The pending ping is delayed, and we did not received + * error replies as well. */ + (mstime() - ri->last_ping_time) > (ri->down_after_period/2) && (mstime() - ri->last_pong_time) > (ri->down_after_period/2)) { sentinelKillLink(ri,ri->cc); @@ -3026,6 +3210,7 @@ void sentinelAskMasterStateToOtherSentinels(sentinelRedisInstance *master, int f char *sentinelVoteLeader(sentinelRedisInstance *master, uint64_t req_epoch, char *req_runid, uint64_t *leader_epoch) { if (req_epoch > sentinel.current_epoch) { sentinel.current_epoch = req_epoch; + sentinelFlushConfig(); sentinelEvent(REDIS_WARNING,"+new-epoch",master,"%llu", (unsigned long long) sentinel.current_epoch); } @@ -3035,13 +3220,14 @@ char *sentinelVoteLeader(sentinelRedisInstance *master, uint64_t req_epoch, char sdsfree(master->leader); master->leader = sdsnew(req_runid); master->leader_epoch = sentinel.current_epoch; + sentinelFlushConfig(); sentinelEvent(REDIS_WARNING,"+vote-for-leader",master,"%s %llu", master->leader, (unsigned long long) master->leader_epoch); /* If we did not voted for ourselves, set the master failover start * time to now, in order to force a delay before we can start a * failover for the same master. */ if (strcasecmp(master->leader,server.runid)) - master->failover_start_time = mstime(); + master->failover_start_time = mstime()+rand()%SENTINEL_MAX_DESYNC; } *leader_epoch = master->leader_epoch; @@ -3062,7 +3248,7 @@ int sentinelLeaderIncr(dict *counters, char *runid) { if (de) { oldval = dictGetUnsignedIntegerVal(de); dictSetUnsignedIntegerVal(de,oldval+1); - return oldval+1; + return (int)oldval+1; } else { de = dictAddRaw(counters,runid); redisAssert(de != NULL); @@ -3186,7 +3372,7 @@ void sentinelStartFailover(sentinelRedisInstance *master) { sentinelEvent(REDIS_WARNING,"+new-epoch",master,"%llu", (unsigned long long) sentinel.current_epoch); sentinelEvent(REDIS_WARNING,"+try-failover",master,"%@"); - master->failover_start_time = mstime(); + master->failover_start_time = mstime()+rand()%SENTINEL_MAX_DESYNC; master->failover_state_change_time = mstime(); } @@ -3337,7 +3523,7 @@ void sentinelFailoverWaitStart(sentinelRedisInstance *ri) { /* The election timeout is the MIN between SENTINEL_ELECTION_TIMEOUT * and the configured failover timeout. */ if (election_timeout > ri->failover_timeout) - election_timeout = ri->failover_timeout; + election_timeout = (int)ri->failover_timeout; /* Abort the failover if I'm not the leader after some time. */ if (mstime() - ri->failover_start_time > election_timeout) { sentinelEvent(REDIS_WARNING,"-failover-abort-not-elected",ri,"%@"); @@ -3496,14 +3682,17 @@ void sentinelFailoverReconfNextSlave(sentinelRedisInstance *master) { /* Skip the promoted slave, and already configured slaves. */ if (slave->flags & (SRI_PROMOTED|SRI_RECONF_DONE)) continue; - /* Clear the SRI_RECONF_SENT flag if too much time elapsed without - * the slave moving forward to the next state. */ + /* If too much time elapsed without the slave moving forward to + * the next state, consider it reconfigured even if it is not. + * Sentinels will detect the slave as misconfigured and fix its + * configuration later. */ if ((slave->flags & SRI_RECONF_SENT) && (mstime() - slave->slave_reconf_sent_time) > - SENTINEL_SLAVE_RECONF_RETRY_PERIOD) + SENTINEL_SLAVE_RECONF_TIMEOUT) { sentinelEvent(REDIS_NOTICE,"-slave-reconf-sent-timeout",slave,"%@"); slave->flags &= ~SRI_RECONF_SENT; + slave->flags |= SRI_RECONF_DONE; } /* Nothing to do for instances that are disconnected or already @@ -3594,7 +3783,7 @@ void sentinelHandleRedisInstance(sentinelRedisInstance *ri) { /* ========== MONITORING HALF ============ */ /* Every kind of instance */ sentinelReconnectInstance(ri); - sentinelPingInstance(ri); + sentinelSendPeriodicCommands(ri); /* ============== ACTING HALF ============= */ /* We don't proceed with the acting half if we are in TILT mode. diff --git a/src/sort.c b/src/sort.c index c878c35b..500811b7 100644 --- a/src/sort.c +++ b/src/sort.c @@ -231,8 +231,15 @@ void sortCommand(redisClient *c) { } else if (!strcasecmp(c->argv[j]->ptr,"alpha")) { alpha = 1; } else if (!strcasecmp(c->argv[j]->ptr,"limit") && leftargs >= 2) { - if ((getLongFromObjectOrReply(c, c->argv[j+1], &limit_start, NULL) != REDIS_OK) || - (getLongFromObjectOrReply(c, c->argv[j+2], &limit_count, NULL) != REDIS_OK)) return; + if ((getLongFromObjectOrReply(c, c->argv[j+1], &limit_start, NULL) + != REDIS_OK) || + (getLongFromObjectOrReply(c, c->argv[j+2], &limit_count, NULL) + != REDIS_OK)) + { + decrRefCount(sortval); + listRelease(operations); + return; + } j+=2; } else if (!strcasecmp(c->argv[j]->ptr,"store") && leftargs >= 1) { storekey = c->argv[j+1]; diff --git a/src/t_string.c b/src/t_string.c index edf51e54..a77b83f7 100644 --- a/src/t_string.c +++ b/src/t_string.c @@ -215,12 +215,7 @@ void setrangeCommand(redisClient *c) { return; /* Create a copy when the object is shared or encoded. */ - if (o->refcount != 1 || o->encoding != REDIS_ENCODING_RAW) { - robj *decoded = getDecodedObject(o); - o = createStringObject(decoded->ptr, sdslen(decoded->ptr)); - decrRefCount(decoded); - dbOverwrite(c->db,c->argv[1],o); - } + o = dbUnshareStringValue(c->db,c->argv[1],o); } if (sdslen(value) > 0) { @@ -433,15 +428,8 @@ void appendCommand(redisClient *c) { if (checkStringLength(c,totlen) != REDIS_OK) return; - /* If the object is shared or encoded, we have to make a copy */ - if (o->refcount != 1 || o->encoding != REDIS_ENCODING_RAW) { - robj *decoded = getDecodedObject(o); - o = createStringObject(decoded->ptr, sdslen(decoded->ptr)); - decrRefCount(decoded); - dbOverwrite(c->db,c->argv[1],o); - } - /* Append the value */ + o = dbUnshareStringValue(c->db,c->argv[1],o); o->ptr = sdscatlen(o->ptr,append->ptr,sdslen(append->ptr)); totlen = sdslen(o->ptr); } diff --git a/src/t_zset.c b/src/t_zset.c index e5181a21..79e8619e 100644 --- a/src/t_zset.c +++ b/src/t_zset.c @@ -51,9 +51,14 @@ #include "redis.h" #include + #ifdef _WIN32 #define bzero(b,len) (memset((b), '\0', (len)), (void) 0) #endif + +static int zslLexValueGteMin(robj *value, zlexrangespec *spec); +static int zslLexValueLteMax(robj *value, zlexrangespec *spec); + zskiplistNode *zslCreateNode(int level, double score, robj *obj) { zskiplistNode *zn = zmalloc(sizeof(*zn)+level*sizeof(struct zskiplistLevel)); zn->score = score; @@ -237,18 +242,18 @@ int zslIsInRange(zskiplist *zsl, zrangespec *range) { /* Find the first node that is contained in the specified range. * Returns NULL when no element is contained in the range. */ -zskiplistNode *zslFirstInRange(zskiplist *zsl, zrangespec range) { +zskiplistNode *zslFirstInRange(zskiplist *zsl, zrangespec *range) { zskiplistNode *x; int i; /* If everything is out of range, return early. */ - if (!zslIsInRange(zsl,&range)) return NULL; + if (!zslIsInRange(zsl,range)) return NULL; x = zsl->header; for (i = zsl->level-1; i >= 0; i--) { /* Go forward while *OUT* of range. */ while (x->level[i].forward && - !zslValueGteMin(x->level[i].forward->score,&range)) + !zslValueGteMin(x->level[i].forward->score,range)) x = x->level[i].forward; } @@ -257,24 +262,24 @@ zskiplistNode *zslFirstInRange(zskiplist *zsl, zrangespec range) { redisAssert(x != NULL); /* Check if score <= max. */ - if (!zslValueLteMax(x->score,&range)) return NULL; + if (!zslValueLteMax(x->score,range)) return NULL; return x; } /* Find the last node that is contained in the specified range. * Returns NULL when no element is contained in the range. */ -zskiplistNode *zslLastInRange(zskiplist *zsl, zrangespec range) { +zskiplistNode *zslLastInRange(zskiplist *zsl, zrangespec *range) { zskiplistNode *x; int i; /* If everything is out of range, return early. */ - if (!zslIsInRange(zsl,&range)) return NULL; + if (!zslIsInRange(zsl,range)) return NULL; x = zsl->header; for (i = zsl->level-1; i >= 0; i--) { /* Go forward while *IN* range. */ while (x->level[i].forward && - zslValueLteMax(x->level[i].forward->score,&range)) + zslValueLteMax(x->level[i].forward->score,range)) x = x->level[i].forward; } @@ -282,7 +287,7 @@ zskiplistNode *zslLastInRange(zskiplist *zsl, zrangespec range) { redisAssert(x != NULL); /* Check if score >= min. */ - if (!zslValueGteMin(x->score,&range)) return NULL; + if (!zslValueGteMin(x->score,range)) return NULL; return x; } @@ -290,16 +295,16 @@ zskiplistNode *zslLastInRange(zskiplist *zsl, zrangespec range) { * Min and max are inclusive, so a score >= min || score <= max is deleted. * Note that this function takes the reference to the hash table view of the * sorted set, in order to remove the elements from the hash table too. */ -unsigned long zslDeleteRangeByScore(zskiplist *zsl, zrangespec range, dict *dict) { +unsigned long zslDeleteRangeByScore(zskiplist *zsl, zrangespec *range, dict *dict) { zskiplistNode *update[ZSKIPLIST_MAXLEVEL], *x; unsigned long removed = 0; int i; x = zsl->header; for (i = zsl->level-1; i >= 0; i--) { - while (x->level[i].forward && (range.minex ? - x->level[i].forward->score <= range.min : - x->level[i].forward->score < range.min)) + while (x->level[i].forward && (range->minex ? + x->level[i].forward->score <= range->min : + x->level[i].forward->score < range->min)) x = x->level[i].forward; update[i] = x; } @@ -308,7 +313,38 @@ unsigned long zslDeleteRangeByScore(zskiplist *zsl, zrangespec range, dict *dict x = x->level[0].forward; /* Delete nodes while in range. */ - while (x && (range.maxex ? x->score < range.max : x->score <= range.max)) { + while (x && + (range->maxex ? x->score < range->max : x->score <= range->max)) + { + zskiplistNode *next = x->level[0].forward; + zslDeleteNode(zsl,x,update); + dictDelete(dict,x->obj); + zslFreeNode(x); + removed++; + x = next; + } + return removed; +} + +unsigned long zslDeleteRangeByLex(zskiplist *zsl, zlexrangespec *range, dict *dict) { + zskiplistNode *update[ZSKIPLIST_MAXLEVEL], *x; + unsigned long removed = 0; + int i; + + + x = zsl->header; + for (i = zsl->level-1; i >= 0; i--) { + while (x->level[i].forward && + !zslLexValueGteMin(x->level[i].forward->obj,range)) + x = x->level[i].forward; + update[i] = x; + } + + /* Current node is the last with score < or <= min. */ + x = x->level[0].forward; + + /* Delete nodes while in range. */ + while (x && zslLexValueLteMax(x->obj,range)) { zskiplistNode *next = x->level[0].forward; zslDeleteNode(zsl,x,update); dictDelete(dict,x->obj); @@ -433,6 +469,171 @@ static int zslParseRange(robj *min, robj *max, zrangespec *spec) { return REDIS_OK; } +/* ------------------------ Lexicographic ranges ---------------------------- */ + +/* Parse max or min argument of ZRANGEBYLEX. + * (foo means foo (open interval) + * [foo means foo (closed interval) + * - means the min string possible + * + means the max string possible + * + * If the string is valid the *dest pointer is set to the redis object + * that will be used for the comparision, and ex will be set to 0 or 1 + * respectively if the item is exclusive or inclusive. REDIS_OK will be + * returned. + * + * If the string is not a valid range REDIS_ERR is returned, and the value + * of *dest and *ex is undefined. */ +int zslParseLexRangeItem(robj *item, robj **dest, int *ex) { + char *c = item->ptr; + + switch(c[0]) { + case '+': + if (c[1] != '\0') return REDIS_ERR; + *ex = 0; + *dest = shared.maxstring; + incrRefCount(shared.maxstring); + return REDIS_OK; + case '-': + if (c[1] != '\0') return REDIS_ERR; + *ex = 0; + *dest = shared.minstring; + incrRefCount(shared.minstring); + return REDIS_OK; + case '(': + *ex = 1; + *dest = createStringObject(c+1,sdslen(c)-1); + return REDIS_OK; + case '[': + *ex = 0; + *dest = createStringObject(c+1,sdslen(c)-1); + return REDIS_OK; + default: + return REDIS_ERR; + } +} + +/* Populate the rangespec according to the objects min and max. + * + * Return REDIS_OK on success. On error REDIS_ERR is returned. + * When OK is returned the structure must be freed with zslFreeLexRange(), + * otherwise no release is needed. */ +static int zslParseLexRange(robj *min, robj *max, zlexrangespec *spec) { + /* The range can't be valid if objects are integer encoded. + * Every item must start with ( or [. */ + if (min->encoding == REDIS_ENCODING_INT || + max->encoding == REDIS_ENCODING_INT) return REDIS_ERR; + + spec->min = spec->max = NULL; + if (zslParseLexRangeItem(min, &spec->min, &spec->minex) == REDIS_ERR || + zslParseLexRangeItem(max, &spec->max, &spec->maxex) == REDIS_ERR) { + if (spec->min) decrRefCount(spec->min); + if (spec->max) decrRefCount(spec->max); + return REDIS_ERR; + } else { + return REDIS_OK; + } +} + +/* Free a lex range structure, must be called only after zelParseLexRange() + * populated the structure with success (REDIS_OK returned). */ +void zslFreeLexRange(zlexrangespec *spec) { + decrRefCount(spec->min); + decrRefCount(spec->max); +} + +/* This is just a wrapper to compareStringObjects() that is able to + * handle shared.minstring and shared.maxstring as the equivalent of + * -inf and +inf for strings */ +int compareStringObjectsForLexRange(robj *a, robj *b) { + if (a == b) return 0; /* This makes sure that we handle inf,inf and + -inf,-inf ASAP. One special case less. */ + if (a == shared.minstring || b == shared.maxstring) return -1; + if (a == shared.maxstring || b == shared.minstring) return 1; + return compareStringObjects(a,b); +} + +static int zslLexValueGteMin(robj *value, zlexrangespec *spec) { + return spec->minex ? + (compareStringObjectsForLexRange(value,spec->min) > 0) : + (compareStringObjectsForLexRange(value,spec->min) >= 0); +} + +static int zslLexValueLteMax(robj *value, zlexrangespec *spec) { + return spec->maxex ? + (compareStringObjectsForLexRange(value,spec->max) < 0) : + (compareStringObjectsForLexRange(value,spec->max) <= 0); +} + +/* Returns if there is a part of the zset is in the lex range. */ +int zslIsInLexRange(zskiplist *zsl, zlexrangespec *range) { + zskiplistNode *x; + + /* Test for ranges that will always be empty. */ + if (compareStringObjectsForLexRange(range->min,range->max) > 1 || + (compareStringObjects(range->min,range->max) == 0 && + (range->minex || range->maxex))) + return 0; + x = zsl->tail; + if (x == NULL || !zslLexValueGteMin(x->obj,range)) + return 0; + x = zsl->header->level[0].forward; + if (x == NULL || !zslLexValueLteMax(x->obj,range)) + return 0; + return 1; +} + +/* Find the first node that is contained in the specified lex range. + * Returns NULL when no element is contained in the range. */ +zskiplistNode *zslFirstInLexRange(zskiplist *zsl, zlexrangespec *range) { + zskiplistNode *x; + int i; + + /* If everything is out of range, return early. */ + if (!zslIsInLexRange(zsl,range)) return NULL; + + x = zsl->header; + for (i = zsl->level-1; i >= 0; i--) { + /* Go forward while *OUT* of range. */ + while (x->level[i].forward && + !zslLexValueGteMin(x->level[i].forward->obj,range)) + x = x->level[i].forward; + } + + /* This is an inner range, so the next node cannot be NULL. */ + x = x->level[0].forward; + redisAssert(x != NULL); + + /* Check if score <= max. */ + if (!zslLexValueLteMax(x->obj,range)) return NULL; + return x; +} + +/* Find the last node that is contained in the specified range. + * Returns NULL when no element is contained in the range. */ +zskiplistNode *zslLastInLexRange(zskiplist *zsl, zlexrangespec *range) { + zskiplistNode *x; + int i; + + /* If everything is out of range, return early. */ + if (!zslIsInLexRange(zsl,range)) return NULL; + + x = zsl->header; + for (i = zsl->level-1; i >= 0; i--) { + /* Go forward while *IN* range. */ + while (x->level[i].forward && + zslLexValueLteMax(x->level[i].forward->obj,range)) + x = x->level[i].forward; + } + + /* This is an inner range, so this node cannot be NULL. */ + redisAssert(x != NULL); + + /* Check if score >= min. */ + if (!zslLexValueGteMin(x->obj,range)) return NULL; + return x; +} + /*----------------------------------------------------------------------------- * Ziplist-backed sorted set API *----------------------------------------------------------------------------*/ @@ -458,6 +659,24 @@ double zzlGetScore(unsigned char *sptr) { return score; } +/* Return a ziplist element as a Redis string object. + * This simple abstraction can be used to simplifies some code at the + * cost of some performance. */ +robj *ziplistGetObject(unsigned char *sptr) { + unsigned char *vstr; + unsigned int vlen; + long long vlong; + + redisAssert(sptr != NULL); + redisAssert(ziplistGet(sptr,&vstr,&vlen,&vlong)); + + if (vstr) { + return createStringObject((char*)vstr,vlen); + } else { + return createStringObjectFromLongLong(vlong); + } +} + /* Compare element in sorted set with given element. */ int zzlCompareElements(unsigned char *eptr, unsigned char *cstr, unsigned int clen) { unsigned char *vstr; @@ -549,21 +768,21 @@ int zzlIsInRange(unsigned char *zl, zrangespec *range) { /* Find pointer to the first element contained in the specified range. * Returns NULL when no element is contained in the range. */ -unsigned char *zzlFirstInRange(unsigned char *zl, zrangespec range) { +unsigned char *zzlFirstInRange(unsigned char *zl, zrangespec *range) { unsigned char *eptr = ziplistIndex(zl,0), *sptr; double score; /* If everything is out of range, return early. */ - if (!zzlIsInRange(zl,&range)) return NULL; + if (!zzlIsInRange(zl,range)) return NULL; while (eptr != NULL) { sptr = ziplistNext(zl,eptr); redisAssert(sptr != NULL); score = zzlGetScore(sptr); - if (zslValueGteMin(score,&range)) { + if (zslValueGteMin(score,range)) { /* Check if score <= max. */ - if (zslValueLteMax(score,&range)) + if (zslValueLteMax(score,range)) return eptr; return NULL; } @@ -577,21 +796,112 @@ unsigned char *zzlFirstInRange(unsigned char *zl, zrangespec range) { /* Find pointer to the last element contained in the specified range. * Returns NULL when no element is contained in the range. */ -unsigned char *zzlLastInRange(unsigned char *zl, zrangespec range) { +unsigned char *zzlLastInRange(unsigned char *zl, zrangespec *range) { unsigned char *eptr = ziplistIndex(zl,-2), *sptr; double score; /* If everything is out of range, return early. */ - if (!zzlIsInRange(zl,&range)) return NULL; + if (!zzlIsInRange(zl,range)) return NULL; while (eptr != NULL) { sptr = ziplistNext(zl,eptr); redisAssert(sptr != NULL); score = zzlGetScore(sptr); - if (zslValueLteMax(score,&range)) { + if (zslValueLteMax(score,range)) { /* Check if score >= min. */ - if (zslValueGteMin(score,&range)) + if (zslValueGteMin(score,range)) + return eptr; + return NULL; + } + + /* Move to previous element by moving to the score of previous element. + * When this returns NULL, we know there also is no element. */ + sptr = ziplistPrev(zl,eptr); + if (sptr != NULL) + redisAssert((eptr = ziplistPrev(zl,sptr)) != NULL); + else + eptr = NULL; + } + + return NULL; +} + +static int zzlLexValueGteMin(unsigned char *p, zlexrangespec *spec) { + robj *value = ziplistGetObject(p); + int res = zslLexValueGteMin(value,spec); + decrRefCount(value); + return res; +} + +static int zzlLexValueLteMax(unsigned char *p, zlexrangespec *spec) { + robj *value = ziplistGetObject(p); + int res = zslLexValueLteMax(value,spec); + decrRefCount(value); + return res; +} + +/* Returns if there is a part of the zset is in range. Should only be used + * internally by zzlFirstInRange and zzlLastInRange. */ +int zzlIsInLexRange(unsigned char *zl, zlexrangespec *range) { + unsigned char *p; + + /* Test for ranges that will always be empty. */ + if (compareStringObjectsForLexRange(range->min,range->max) > 1 || + (compareStringObjects(range->min,range->max) == 0 && + (range->minex || range->maxex))) + return 0; + + p = ziplistIndex(zl,-2); /* Last element. */ + if (p == NULL) return 0; + if (!zzlLexValueGteMin(p,range)) + return 0; + + p = ziplistIndex(zl,0); /* First element. */ + redisAssert(p != NULL); + if (!zzlLexValueLteMax(p,range)) + return 0; + + return 1; +} + +/* Find pointer to the first element contained in the specified lex range. + * Returns NULL when no element is contained in the range. */ +unsigned char *zzlFirstInLexRange(unsigned char *zl, zlexrangespec *range) { + unsigned char *eptr = ziplistIndex(zl,0), *sptr; + + /* If everything is out of range, return early. */ + if (!zzlIsInLexRange(zl,range)) return NULL; + + while (eptr != NULL) { + if (zzlLexValueGteMin(eptr,range)) { + /* Check if score <= max. */ + if (zzlLexValueLteMax(eptr,range)) + return eptr; + return NULL; + } + + /* Move to next element. */ + sptr = ziplistNext(zl,eptr); /* This element score. Skip it. */ + redisAssert(sptr != NULL); + eptr = ziplistNext(zl,sptr); /* Next element. */ + } + + return NULL; +} + +/* Find pointer to the last element contained in the specified lex range. + * Returns NULL when no element is contained in the range. */ +unsigned char *zzlLastInLexRange(unsigned char *zl, zlexrangespec *range) { + unsigned char *eptr = ziplistIndex(zl,-2), *sptr; + + /* If everything is out of range, return early. */ + if (!zzlIsInLexRange(zl,range)) return NULL; + + while (eptr != NULL) { + if (zzlLexValueLteMax(eptr,range)) { + /* Check if score >= min. */ + if (zzlLexValueGteMin(eptr,range)) return eptr; return NULL; } @@ -705,7 +1015,7 @@ unsigned char *zzlInsert(unsigned char *zl, robj *ele, double score) { return zl; } -unsigned char *zzlDeleteRangeByScore(unsigned char *zl, zrangespec range, unsigned long *deleted) { +unsigned char *zzlDeleteRangeByScore(unsigned char *zl, zrangespec *range, unsigned long *deleted) { unsigned char *eptr, *sptr; double score; unsigned long num = 0; @@ -719,7 +1029,34 @@ unsigned char *zzlDeleteRangeByScore(unsigned char *zl, zrangespec range, unsign * byte and ziplistNext will return NULL. */ while ((sptr = ziplistNext(zl,eptr)) != NULL) { score = zzlGetScore(sptr); - if (zslValueLteMax(score,&range)) { + if (zslValueLteMax(score,range)) { + /* Delete both the element and the score. */ + zl = ziplistDelete(zl,&eptr); + zl = ziplistDelete(zl,&eptr); + num++; + } else { + /* No longer in range. */ + break; + } + } + + if (deleted != NULL) *deleted = num; + return zl; +} + +unsigned char *zzlDeleteRangeByLex(unsigned char *zl, zlexrangespec *range, unsigned long *deleted) { + unsigned char *eptr, *sptr; + unsigned long num = 0; + + if (deleted != NULL) *deleted = 0; + + eptr = zzlFirstInLexRange(zl,range); + if (eptr == NULL) return zl; + + /* When the tail of the ziplist is deleted, eptr will point to the sentinel + * byte and ziplistNext will return NULL. */ + while ((sptr = ziplistNext(zl,eptr)) != NULL) { + if (zzlLexValueLteMax(eptr,range)) { /* Delete both the element and the score. */ zl = ziplistDelete(zl,&eptr); zl = ziplistDelete(zl,&eptr); @@ -1041,31 +1378,86 @@ void zremCommand(redisClient *c) { addReplyLongLong(c,deleted); } -void zremrangebyscoreCommand(redisClient *c) { +/* Implements ZREMRANGEBYRANK, ZREMRANGEBYSCORE, ZREMRANGEBYLEX commands. */ +#define ZRANGE_RANK 0 +#define ZRANGE_SCORE 1 +#define ZRANGE_LEX 2 +void zremrangeGenericCommand(redisClient *c, int rangetype) { robj *key = c->argv[1]; robj *zobj; - zrangespec range; int keyremoved = 0; unsigned long deleted; + zrangespec range; + zlexrangespec lexrange; + long start, end, llen; - /* Parse the range arguments. */ - if (zslParseRange(c->argv[2],c->argv[3],&range) != REDIS_OK) { - addReplyError(c,"min or max is not a float"); - return; + /* Step 1: Parse the range. */ + if (rangetype == ZRANGE_RANK) { + if ((getLongFromObjectOrReply(c,c->argv[2],&start,NULL) != REDIS_OK) || + (getLongFromObjectOrReply(c,c->argv[3],&end,NULL) != REDIS_OK)) + return; + } else if (rangetype == ZRANGE_SCORE) { + if (zslParseRange(c->argv[2],c->argv[3],&range) != REDIS_OK) { + addReplyError(c,"min or max is not a float"); + return; + } + } else if (rangetype == ZRANGE_LEX) { + if (zslParseLexRange(c->argv[2],c->argv[3],&lexrange) != REDIS_OK) { + addReplyError(c,"min or max not valid string range item"); + return; + } } + /* Step 2: Lookup & range sanity checks if needed. */ if ((zobj = lookupKeyWriteOrReply(c,key,shared.czero)) == NULL || - checkType(c,zobj,REDIS_ZSET)) return; + checkType(c,zobj,REDIS_ZSET)) goto cleanup; + if (rangetype == ZRANGE_RANK) { + /* Sanitize indexes. */ + llen = zsetLength(zobj); + if (start < 0) start = llen+start; + if (end < 0) end = llen+end; + if (start < 0) start = 0; + + /* Invariant: start >= 0, so this test will be true when end < 0. + * The range is empty when start > end or start >= length. */ + if (start > end || start >= llen) { + addReply(c,shared.czero); + goto cleanup; + } + if (end >= llen) end = llen-1; + } + + /* Step 3: Perform the range deletion operation. */ if (zobj->encoding == REDIS_ENCODING_ZIPLIST) { - zobj->ptr = zzlDeleteRangeByScore(zobj->ptr,range,&deleted); + switch(rangetype) { + case ZRANGE_RANK: + zobj->ptr = zzlDeleteRangeByRank(zobj->ptr,start+1,end+1,&deleted); + break; + case ZRANGE_SCORE: + zobj->ptr = zzlDeleteRangeByScore(zobj->ptr,&range,&deleted); + break; + case ZRANGE_LEX: + zobj->ptr = zzlDeleteRangeByLex(zobj->ptr,&lexrange,&deleted); + break; + } if (zzlLength(zobj->ptr) == 0) { dbDelete(c->db,key); keyremoved = 1; } } else if (zobj->encoding == REDIS_ENCODING_SKIPLIST) { zset *zs = zobj->ptr; - deleted = zslDeleteRangeByScore(zs->zsl,range,zs->dict); + switch(rangetype) { + case ZRANGE_RANK: + deleted = zslDeleteRangeByRank(zs->zsl,start+1,end+1,zs->dict); + break; + case ZRANGE_SCORE: + deleted = zslDeleteRangeByScore(zs->zsl,&range,zs->dict); + break; + case ZRANGE_LEX: + deleted = zslDeleteRangeByLex(zs->zsl,&lexrange,zs->dict); + break; + } if (htNeedsResize(zs->dict)) dictResize(zs->dict); if (dictSize(zs->dict) == 0) { dbDelete(c->db,key); @@ -1075,74 +1467,31 @@ void zremrangebyscoreCommand(redisClient *c) { redisPanic("Unknown sorted set encoding"); } + /* Step 4: Notifications and reply. */ if (deleted) { + char *event[3] = {"zremrangebyrank","zremrangebyscore","zremrangebylex"}; signalModifiedKey(c->db,key); - notifyKeyspaceEvent(REDIS_NOTIFY_ZSET,"zrembyscore",key,c->db->id); + notifyKeyspaceEvent(REDIS_NOTIFY_ZSET,event[rangetype],key,c->db->id); if (keyremoved) notifyKeyspaceEvent(REDIS_NOTIFY_GENERIC,"del",key,c->db->id); } server.dirty += deleted; addReplyLongLong(c,deleted); + +cleanup: + if (rangetype == ZRANGE_LEX) zslFreeLexRange(&lexrange); } void zremrangebyrankCommand(redisClient *c) { - robj *key = c->argv[1]; - robj *zobj; - long start; - long end; - int llen; - unsigned long deleted; - int keyremoved = 0; + zremrangeGenericCommand(c,ZRANGE_RANK); +} - if ((getLongFromObjectOrReply(c, c->argv[2], &start, NULL) != REDIS_OK) || - (getLongFromObjectOrReply(c, c->argv[3], &end, NULL) != REDIS_OK)) return; +void zremrangebyscoreCommand(redisClient *c) { + zremrangeGenericCommand(c,ZRANGE_SCORE); +} - if ((zobj = lookupKeyWriteOrReply(c,key,shared.czero)) == NULL || - checkType(c,zobj,REDIS_ZSET)) return; - - /* Sanitize indexes. */ - llen = zsetLength(zobj); - if (start < 0) start = llen+start; - if (end < 0) end = llen+end; - if (start < 0) start = 0; - - /* Invariant: start >= 0, so this test will be true when end < 0. - * The range is empty when start > end or start >= length. */ - if (start > end || start >= llen) { - addReply(c,shared.czero); - return; - } - if (end >= llen) end = llen-1; - - if (zobj->encoding == REDIS_ENCODING_ZIPLIST) { - /* Correct for 1-based rank. */ - zobj->ptr = zzlDeleteRangeByRank(zobj->ptr,start+1,end+1,&deleted); - if (zzlLength(zobj->ptr) == 0) { - dbDelete(c->db,key); - keyremoved = 1; - } - } else if (zobj->encoding == REDIS_ENCODING_SKIPLIST) { - zset *zs = zobj->ptr; - - /* Correct for 1-based rank. */ - deleted = zslDeleteRangeByRank(zs->zsl,start+1,end+1,zs->dict); - if (htNeedsResize(zs->dict)) dictResize(zs->dict); - if (dictSize(zs->dict) == 0) { - dbDelete(c->db,key); - keyremoved = 1; - } - } else { - redisPanic("Unknown sorted set encoding"); - } - - if (deleted) { - signalModifiedKey(c->db,key); - notifyKeyspaceEvent(REDIS_NOTIFY_ZSET,"zrembyrank",key,c->db->id); - if (keyremoved) - notifyKeyspaceEvent(REDIS_NOTIFY_GENERIC,"del",key,c->db->id); - } - server.dirty += deleted; - addReplyLongLong(c,deleted); +void zremrangebylexCommand(redisClient *c) { + zremrangeGenericCommand(c,ZRANGE_LEX); } typedef struct { @@ -1276,7 +1625,7 @@ int zuiLength(zsetopsrc *op) { return intsetLen(op->subject->ptr); } else if (op->encoding == REDIS_ENCODING_HT) { dict *ht = op->subject->ptr; - return dictSize(ht); + return (int)dictSize(ht); } else { redisPanic("Unknown set encoding"); } @@ -1886,9 +2235,9 @@ void genericZrangebyscoreCommand(redisClient *c, int reverse) { /* If reversed, get the last node in range as starting point. */ if (reverse) { - eptr = zzlLastInRange(zl,range); + eptr = zzlLastInRange(zl,&range); } else { - eptr = zzlFirstInRange(zl,range); + eptr = zzlFirstInRange(zl,&range); } /* No "first" element in the specified interval. */ @@ -1954,9 +2303,9 @@ void genericZrangebyscoreCommand(redisClient *c, int reverse) { /* If reversed, get the last node in range as starting point. */ if (reverse) { - ln = zslLastInRange(zsl,range); + ln = zslLastInRange(zsl,&range); } else { - ln = zslFirstInRange(zsl,range); + ln = zslFirstInRange(zsl,&range); } /* No "first" element in the specified interval. */ @@ -2043,7 +2392,7 @@ void zcountCommand(redisClient *c) { double score; /* Use the first element in range as the starting point */ - eptr = zzlFirstInRange(zl,range); + eptr = zzlFirstInRange(zl,&range); /* No "first" element */ if (eptr == NULL) { @@ -2075,7 +2424,7 @@ void zcountCommand(redisClient *c) { unsigned long rank; /* Find first element in range */ - zn = zslFirstInRange(zsl, range); + zn = zslFirstInRange(zsl, &range); /* Use rank of first element, if any, to determine preliminary count */ if (zn != NULL) { @@ -2083,7 +2432,7 @@ void zcountCommand(redisClient *c) { count = (zsl->length - (rank - 1)); /* Find last element in range */ - zn = zslLastInRange(zsl, range); + zn = zslLastInRange(zsl, &range); /* Use rank of last element, if any, to determine the actual count */ if (zn != NULL) { @@ -2098,6 +2447,270 @@ void zcountCommand(redisClient *c) { addReplyLongLong(c, count); } +void zlexcountCommand(redisClient *c) { + robj *key = c->argv[1]; + robj *zobj; + zlexrangespec range; + int count = 0; + + /* Parse the range arguments */ + if (zslParseLexRange(c->argv[2],c->argv[3],&range) != REDIS_OK) { + addReplyError(c,"min or max not valid string range item"); + return; + } + + /* Lookup the sorted set */ + if ((zobj = lookupKeyReadOrReply(c, key, shared.czero)) == NULL || + checkType(c, zobj, REDIS_ZSET)) + { + zslFreeLexRange(&range); + return; + } + + if (zobj->encoding == REDIS_ENCODING_ZIPLIST) { + unsigned char *zl = zobj->ptr; + unsigned char *eptr, *sptr; + + /* Use the first element in range as the starting point */ + eptr = zzlFirstInLexRange(zl,&range); + + /* No "first" element */ + if (eptr == NULL) { + zslFreeLexRange(&range); + addReply(c, shared.czero); + return; + } + + /* First element is in range */ + sptr = ziplistNext(zl,eptr); + redisAssertWithInfo(c,zobj,zzlLexValueLteMax(eptr,&range)); + + /* Iterate over elements in range */ + while (eptr) { + /* Abort when the node is no longer in range. */ + if (!zzlLexValueLteMax(eptr,&range)) { + break; + } else { + count++; + zzlNext(zl,&eptr,&sptr); + } + } + } else if (zobj->encoding == REDIS_ENCODING_SKIPLIST) { + zset *zs = zobj->ptr; + zskiplist *zsl = zs->zsl; + zskiplistNode *zn; + unsigned long rank; + + /* Find first element in range */ + zn = zslFirstInLexRange(zsl, &range); + + /* Use rank of first element, if any, to determine preliminary count */ + if (zn != NULL) { + rank = zslGetRank(zsl, zn->score, zn->obj); + count = (zsl->length - (rank - 1)); + + /* Find last element in range */ + zn = zslLastInLexRange(zsl, &range); + + /* Use rank of last element, if any, to determine the actual count */ + if (zn != NULL) { + rank = zslGetRank(zsl, zn->score, zn->obj); + count -= (zsl->length - rank); + } + } + } else { + redisPanic("Unknown sorted set encoding"); + } + + zslFreeLexRange(&range); + addReplyLongLong(c, count); +} + +/* This command implements ZRANGEBYLEX, ZREVRANGEBYLEX. */ +void genericZrangebylexCommand(redisClient *c, int reverse) { + zlexrangespec range; + robj *key = c->argv[1]; + robj *zobj; + long offset = 0, limit = -1; + unsigned long rangelen = 0; + void *replylen = NULL; + int minidx, maxidx; + + /* Parse the range arguments. */ + if (reverse) { + /* Range is given as [max,min] */ + maxidx = 2; minidx = 3; + } else { + /* Range is given as [min,max] */ + minidx = 2; maxidx = 3; + } + + if (zslParseLexRange(c->argv[minidx],c->argv[maxidx],&range) != REDIS_OK) { + addReplyError(c,"min or max not valid string range item"); + return; + } + + /* Parse optional extra arguments. Note that ZCOUNT will exactly have + * 4 arguments, so we'll never enter the following code path. */ + if (c->argc > 4) { + int remaining = c->argc - 4; + int pos = 4; + + while (remaining) { + if (remaining >= 3 && !strcasecmp(c->argv[pos]->ptr,"limit")) { + if ((getLongFromObjectOrReply(c, c->argv[pos+1], &offset, NULL) != REDIS_OK) || + (getLongFromObjectOrReply(c, c->argv[pos+2], &limit, NULL) != REDIS_OK)) return; + pos += 3; remaining -= 3; + } else { + zslFreeLexRange(&range); + addReply(c,shared.syntaxerr); + return; + } + } + } + + /* Ok, lookup the key and get the range */ + if ((zobj = lookupKeyReadOrReply(c,key,shared.emptymultibulk)) == NULL || + checkType(c,zobj,REDIS_ZSET)) + { + zslFreeLexRange(&range); + return; + } + + if (zobj->encoding == REDIS_ENCODING_ZIPLIST) { + unsigned char *zl = zobj->ptr; + unsigned char *eptr, *sptr; + unsigned char *vstr; + unsigned int vlen; + long long vlong; + + /* If reversed, get the last node in range as starting point. */ + if (reverse) { + eptr = zzlLastInLexRange(zl,&range); + } else { + eptr = zzlFirstInLexRange(zl,&range); + } + + /* No "first" element in the specified interval. */ + if (eptr == NULL) { + addReply(c, shared.emptymultibulk); + zslFreeLexRange(&range); + return; + } + + /* Get score pointer for the first element. */ + redisAssertWithInfo(c,zobj,eptr != NULL); + sptr = ziplistNext(zl,eptr); + + /* We don't know in advance how many matching elements there are in the + * list, so we push this object that will represent the multi-bulk + * length in the output buffer, and will "fix" it later */ + replylen = addDeferredMultiBulkLength(c); + + /* If there is an offset, just traverse the number of elements without + * checking the score because that is done in the next loop. */ + while (eptr && offset--) { + if (reverse) { + zzlPrev(zl,&eptr,&sptr); + } else { + zzlNext(zl,&eptr,&sptr); + } + } + + while (eptr && limit--) { + /* Abort when the node is no longer in range. */ + if (reverse) { + if (!zzlLexValueGteMin(eptr,&range)) break; + } else { + if (!zzlLexValueLteMax(eptr,&range)) break; + } + + /* We know the element exists, so ziplistGet should always + * succeed. */ + redisAssertWithInfo(c,zobj,ziplistGet(eptr,&vstr,&vlen,&vlong)); + + rangelen++; + if (vstr == NULL) { + addReplyBulkLongLong(c,vlong); + } else { + addReplyBulkCBuffer(c,vstr,vlen); + } + + /* Move to next node */ + if (reverse) { + zzlPrev(zl,&eptr,&sptr); + } else { + zzlNext(zl,&eptr,&sptr); + } + } + } else if (zobj->encoding == REDIS_ENCODING_SKIPLIST) { + zset *zs = zobj->ptr; + zskiplist *zsl = zs->zsl; + zskiplistNode *ln; + + /* If reversed, get the last node in range as starting point. */ + if (reverse) { + ln = zslLastInLexRange(zsl,&range); + } else { + ln = zslFirstInLexRange(zsl,&range); + } + + /* No "first" element in the specified interval. */ + if (ln == NULL) { + addReply(c, shared.emptymultibulk); + zslFreeLexRange(&range); + return; + } + + /* We don't know in advance how many matching elements there are in the + * list, so we push this object that will represent the multi-bulk + * length in the output buffer, and will "fix" it later */ + replylen = addDeferredMultiBulkLength(c); + + /* If there is an offset, just traverse the number of elements without + * checking the score because that is done in the next loop. */ + while (ln && offset--) { + if (reverse) { + ln = ln->backward; + } else { + ln = ln->level[0].forward; + } + } + + while (ln && limit--) { + /* Abort when the node is no longer in range. */ + if (reverse) { + if (!zslLexValueGteMin(ln->obj,&range)) break; + } else { + if (!zslLexValueLteMax(ln->obj,&range)) break; + } + + rangelen++; + addReplyBulk(c,ln->obj); + + /* Move to next node */ + if (reverse) { + ln = ln->backward; + } else { + ln = ln->level[0].forward; + } + } + } else { + redisPanic("Unknown sorted set encoding"); + } + + zslFreeLexRange(&range); + setDeferredMultiBulkLength(c, replylen, rangelen); +} + +void zrangebylexCommand(redisClient *c) { + genericZrangebylexCommand(c,0); +} + +void zrevrangebylexCommand(redisClient *c) { + genericZrangebylexCommand(c,1); +} + void zcardCommand(redisClient *c) { robj *key = c->argv[1]; robj *zobj; diff --git a/src/version.h b/src/version.h index 016fabb9..b8436714 100644 --- a/src/version.h +++ b/src/version.h @@ -1 +1 @@ -#define REDIS_VERSION "2.8.4" +#define REDIS_VERSION "2.8.9" diff --git a/src/zmalloc.c b/src/zmalloc.c index e7ba9fa9..555d40a0 100644 --- a/src/zmalloc.c +++ b/src/zmalloc.c @@ -353,8 +353,8 @@ size_t zmalloc_get_rss(void) { #endif /* Fragmentation = RSS / allocated-bytes */ -float zmalloc_get_fragmentation_ratio(void) { - return (float)zmalloc_get_rss()/zmalloc_used_memory(); +float zmalloc_get_fragmentation_ratio(size_t rss) { + return (float)rss/zmalloc_used_memory(); } #if defined(HAVE_PROC_SMAPS) diff --git a/src/zmalloc.h b/src/zmalloc.h index 8c4d983e..8a8a199f 100644 --- a/src/zmalloc.h +++ b/src/zmalloc.h @@ -79,7 +79,7 @@ char *zstrdup(const char *s); size_t zmalloc_used_memory(void); void zmalloc_enable_thread_safeness(void); void zmalloc_set_oom_handler(void (*oom_handler)(size_t)); -float zmalloc_get_fragmentation_ratio(void); +float zmalloc_get_fragmentation_ratio(size_t rss); size_t zmalloc_get_rss(void); size_t zmalloc_get_private_dirty(void); void zlibc_free(void *ptr); diff --git a/tests/sentinel-tests/00-base.tcl b/tests/sentinel-tests/00-base.tcl new file mode 100644 index 00000000..26758de0 --- /dev/null +++ b/tests/sentinel-tests/00-base.tcl @@ -0,0 +1,126 @@ +# Check the basic monitoring and failover capabilities. + +source "../sentinel-tests/includes/init-tests.tcl" + +if {$::simulate_error} { + test "This test will fail" { + fail "Simulated error" + } +} + +test "Basic failover works if the master is down" { + set old_port [RI $master_id tcp_port] + set addr [S 0 SENTINEL GET-MASTER-ADDR-BY-NAME mymaster] + assert {[lindex $addr 1] == $old_port} + kill_instance redis $master_id + foreach_sentinel_id id { + wait_for_condition 1000 50 { + [lindex [S $id SENTINEL GET-MASTER-ADDR-BY-NAME mymaster] 1] != $old_port + } else { + fail "At least one Sentinel did not received failover info" + } + } + restart_instance redis $master_id + set addr [S 0 SENTINEL GET-MASTER-ADDR-BY-NAME mymaster] + set master_id [get_instance_id_by_port redis [lindex $addr 1]] +} + +test "New master [join $addr {:}] role matches" { + assert {[RI $master_id role] eq {master}} +} + +test "All the other slaves now point to the new master" { + foreach_redis_id id { + if {$id != $master_id && $id != 0} { + wait_for_condition 1000 50 { + [RI $id master_port] == [lindex $addr 1] + } else { + fail "Redis ID $id not configured to replicate with new master" + } + } + } +} + +test "The old master eventually gets reconfigured as a slave" { + wait_for_condition 1000 50 { + [RI 0 master_port] == [lindex $addr 1] + } else { + fail "Old master not reconfigured as slave of new master" + } +} + +test "ODOWN is not possible without N (quorum) Sentinels reports" { + foreach_sentinel_id id { + S $id SENTINEL SET mymaster quorum [expr $sentinels+1] + } + set old_port [RI $master_id tcp_port] + set addr [S 0 SENTINEL GET-MASTER-ADDR-BY-NAME mymaster] + assert {[lindex $addr 1] == $old_port} + kill_instance redis $master_id + + # Make sure failover did not happened. + set addr [S 0 SENTINEL GET-MASTER-ADDR-BY-NAME mymaster] + assert {[lindex $addr 1] == $old_port} + restart_instance redis $master_id +} + +test "Failover is not possible without majority agreement" { + foreach_sentinel_id id { + S $id SENTINEL SET mymaster quorum $quorum + } + + # Crash majority of sentinels + for {set id 0} {$id < $quorum} {incr id} { + kill_instance sentinel $id + } + + # Kill the current master + kill_instance redis $master_id + + # Make sure failover did not happened. + set addr [S $quorum SENTINEL GET-MASTER-ADDR-BY-NAME mymaster] + assert {[lindex $addr 1] == $old_port} + restart_instance redis $master_id + + # Cleanup: restart Sentinels to monitor the master. + for {set id 0} {$id < $quorum} {incr id} { + restart_instance sentinel $id + } +} + +test "Failover works if we configure for absolute agreement" { + foreach_sentinel_id id { + S $id SENTINEL SET mymaster quorum $sentinels + } + + # Wait for Sentinels to monitor the master again + foreach_sentinel_id id { + wait_for_condition 1000 50 { + [dict get [S $id SENTINEL MASTER mymaster] info-refresh] < 100000 + } else { + fail "At least one Sentinel is not monitoring the master" + } + } + + kill_instance redis $master_id + + foreach_sentinel_id id { + wait_for_condition 1000 50 { + [lindex [S $id SENTINEL GET-MASTER-ADDR-BY-NAME mymaster] 1] != $old_port + } else { + fail "At least one Sentinel did not received failover info" + } + } + restart_instance redis $master_id + set addr [S 0 SENTINEL GET-MASTER-ADDR-BY-NAME mymaster] + set master_id [get_instance_id_by_port redis [lindex $addr 1]] + + # Set the min ODOWN agreement back to strict majority. + foreach_sentinel_id id { + S $id SENTINEL SET mymaster quorum $quorum + } +} + +test "New master [join $addr {:}] role matches" { + assert {[RI $master_id role] eq {master}} +} diff --git a/tests/sentinel-tests/01-conf-update.tcl b/tests/sentinel-tests/01-conf-update.tcl new file mode 100644 index 00000000..4625ebd4 --- /dev/null +++ b/tests/sentinel-tests/01-conf-update.tcl @@ -0,0 +1,39 @@ +# Test Sentinel configuration consistency after partitions heal. + +source "../sentinel-tests/includes/init-tests.tcl" + +test "We can failover with Sentinel 1 crashed" { + set old_port [RI $master_id tcp_port] + set addr [S 0 SENTINEL GET-MASTER-ADDR-BY-NAME mymaster] + assert {[lindex $addr 1] == $old_port} + + # Crash Sentinel 1 + kill_instance sentinel 1 + + kill_instance redis $master_id + foreach_sentinel_id id { + if {$id != 1} { + wait_for_condition 1000 50 { + [lindex [S $id SENTINEL GET-MASTER-ADDR-BY-NAME mymaster] 1] != $old_port + } else { + fail "Sentinel $id did not received failover info" + } + } + } + restart_instance redis $master_id + set addr [S 0 SENTINEL GET-MASTER-ADDR-BY-NAME mymaster] + set master_id [get_instance_id_by_port redis [lindex $addr 1]] +} + +test "After Sentinel 1 is restarted, its config gets updated" { + restart_instance sentinel 1 + wait_for_condition 1000 50 { + [lindex [S 1 SENTINEL GET-MASTER-ADDR-BY-NAME mymaster] 1] != $old_port + } else { + fail "Restarted Sentinel did not received failover info" + } +} + +test "New master [join $addr {:}] role matches" { + assert {[RI $master_id role] eq {master}} +} diff --git a/tests/sentinel-tests/02-slaves-reconf.tcl b/tests/sentinel-tests/02-slaves-reconf.tcl new file mode 100644 index 00000000..843c62dc --- /dev/null +++ b/tests/sentinel-tests/02-slaves-reconf.tcl @@ -0,0 +1,83 @@ +# Check that slaves are reconfigured at a latter time if they are partitioned. +# +# Here we should test: +# 1) That slaves point to the new master after failover. +# 2) That partitioned slaves point to new master when they are partitioned +# away during failover and return at a latter time. + +source "../sentinel-tests/includes/init-tests.tcl" + +proc 03_test_slaves_replication {} { + uplevel 1 { + test "Check that slaves replicate from current master" { + set master_port [RI $master_id tcp_port] + foreach_redis_id id { + if {$id == $master_id} continue + if {[instance_is_killed redis $id]} continue + wait_for_condition 1000 50 { + [RI $id master_port] == $master_port + } else { + fail "Redis slave $id is replicating from wrong master" + } + } + } + } +} + +proc 03_crash_and_failover {} { + uplevel 1 { + test "Crash the master and force a failover" { + set old_port [RI $master_id tcp_port] + set addr [S 0 SENTINEL GET-MASTER-ADDR-BY-NAME mymaster] + assert {[lindex $addr 1] == $old_port} + kill_instance redis $master_id + foreach_sentinel_id id { + wait_for_condition 1000 50 { + [lindex [S $id SENTINEL GET-MASTER-ADDR-BY-NAME mymaster] 1] != $old_port + } else { + fail "At least one Sentinel did not received failover info" + } + } + restart_instance redis $master_id + set addr [S 0 SENTINEL GET-MASTER-ADDR-BY-NAME mymaster] + set master_id [get_instance_id_by_port redis [lindex $addr 1]] + } + } +} + +03_test_slaves_replication +03_crash_and_failover +03_test_slaves_replication + +test "Kill a slave instance" { + foreach_redis_id id { + if {$id == $master_id} continue + set killed_slave_id $id + kill_instance redis $id + break + } +} + +03_crash_and_failover +03_test_slaves_replication + +test "Wait for failover to end" { + set inprogress 1 + while {$inprogress} { + set inprogress 0 + foreach_sentinel_id id { + if {[dict exists [S $id SENTINEL MASTER mymaster] failover-state]} { + incr inprogress + } + } + if {$inprogress} {after 100} + } +} + +test "Restart killed slave and test replication of slaves again..." { + restart_instance redis $killed_slave_id +} + +# Now we check if the slave rejoining the partition is reconfigured even +# if the failover finished. +03_test_slaves_replication diff --git a/tests/sentinel-tests/03-runtime-reconf.tcl b/tests/sentinel-tests/03-runtime-reconf.tcl new file mode 100644 index 00000000..426596c3 --- /dev/null +++ b/tests/sentinel-tests/03-runtime-reconf.tcl @@ -0,0 +1 @@ +# Test runtime reconfiguration command SENTINEL SET. diff --git a/tests/sentinel-tests/04-slave-selection.tcl b/tests/sentinel-tests/04-slave-selection.tcl new file mode 100644 index 00000000..3d2ca648 --- /dev/null +++ b/tests/sentinel-tests/04-slave-selection.tcl @@ -0,0 +1,5 @@ +# Test slave selection algorithm. +# +# This unit should test: +# 1) That when there are no suitable slaves no failover is performed. +# 2) That among the available slaves, the one with better offset is picked. diff --git a/tests/sentinel-tests/includes/init-tests.tcl b/tests/sentinel-tests/includes/init-tests.tcl new file mode 100644 index 00000000..cb359ea1 --- /dev/null +++ b/tests/sentinel-tests/includes/init-tests.tcl @@ -0,0 +1,67 @@ +# Initialization tests -- most units will start including this. + +test "(init) Restart killed instances" { + foreach type {redis sentinel} { + foreach_${type}_id id { + if {[get_instance_attrib $type $id pid] == -1} { + puts -nonewline "$type/$id " + flush stdout + restart_instance $type $id + } + } + } +} + +set redis_slaves 4 +test "(init) Create a master-slaves cluster of [expr $redis_slaves+1] instances" { + create_redis_master_slave_cluster [expr {$redis_slaves+1}] +} +set master_id 0 + +test "(init) Sentinels can start monitoring a master" { + set sentinels [llength $::sentinel_instances] + set quorum [expr {$sentinels/2+1}] + foreach_sentinel_id id { + catch {S $id SENTINEL REMOVE mymaster} + S $id SENTINEL MONITOR mymaster \ + [get_instance_attrib redis $master_id host] \ + [get_instance_attrib redis $master_id port] $quorum + } + foreach_sentinel_id id { + assert {[S $id sentinel master mymaster] ne {}} + S $id SENTINEL SET mymaster down-after-milliseconds 2000 + S $id SENTINEL SET mymaster failover-timeout 20000 + S $id SENTINEL SET mymaster parallel-syncs 10 + } +} + +test "(init) Sentinels can talk with the master" { + foreach_sentinel_id id { + wait_for_condition 100 50 { + [catch {S $id SENTINEL GET-MASTER-ADDR-BY-NAME mymaster}] == 0 + } else { + fail "Sentinel $id can't talk with the master." + } + } +} + +test "(init) Sentinels are able to auto-discover other sentinels" { + set sentinels [llength $::sentinel_instances] + foreach_sentinel_id id { + wait_for_condition 100 50 { + [dict get [S $id SENTINEL MASTER mymaster] num-other-sentinels] == ($sentinels-1) + } else { + fail "At least some sentinel can't detect some other sentinel" + } + } +} + +test "(init) Sentinels are able to auto-discover slaves" { + foreach_sentinel_id id { + wait_for_condition 100 50 { + [dict get [S $id SENTINEL MASTER mymaster] num-slaves] == $redis_slaves + } else { + fail "At least some sentinel can't detect some slave" + } + } +} diff --git a/tests/sentinel-tmp/.gitignore b/tests/sentinel-tmp/.gitignore new file mode 100644 index 00000000..f581f73e --- /dev/null +++ b/tests/sentinel-tmp/.gitignore @@ -0,0 +1,2 @@ +redis_* +sentinel_* diff --git a/tests/sentinel.tcl b/tests/sentinel.tcl new file mode 100644 index 00000000..b69d6adb --- /dev/null +++ b/tests/sentinel.tcl @@ -0,0 +1,406 @@ +# Sentinel test suite. Copyright (C) 2014 Salvatore Sanfilippo antirez@gmail.com +# This softare is released under the BSD License. See the COPYING file for +# more information. + +package require Tcl 8.5 + +set tcl_precision 17 +source tests/support/redis.tcl +source tests/support/util.tcl +source tests/support/server.tcl +source tests/support/test.tcl + +set ::verbose 0 +set ::pause_on_error 0 +set ::simulate_error 0 +set ::sentinel_instances {} +set ::redis_instances {} +set ::sentinel_base_port 20000 +set ::redis_base_port 30000 +set ::instances_count 5 ; # How many Sentinels / Instances we use at max +set ::pids {} ; # We kill everything at exit +set ::dirs {} ; # We remove all the temp dirs at exit +set ::run_matching {} ; # If non empty, only tests matching pattern are run. + +if {[catch {cd tests/sentinel-tmp}]} { + puts "tests/sentinel-tmp directory not found." + puts "Please run this test from the Redis source root." + exit 1 +} + +# Spawn a redis or sentinel instance, depending on 'type'. +proc spawn_instance {type base_port count} { + for {set j 0} {$j < $count} {incr j} { + set port [find_available_port $base_port] + incr base_port + puts "Starting $type #$j at port $port" + + # Create a directory for this Sentinel. + set dirname "${type}_${j}" + lappend ::dirs $dirname + catch {exec rm -rf $dirname} + file mkdir $dirname + + # Write the Sentinel config file. + set cfgfile [file join $dirname $type.conf] + set cfg [open $cfgfile w] + puts $cfg "port $port" + puts $cfg "dir ./$dirname" + puts $cfg "logfile log.txt" + close $cfg + + # Finally exec it and remember the pid for later cleanup. + if {$type eq "redis"} { + set prgname redis-server + } else { + set prgname redis-sentinel + } + set pid [exec ../../src/${prgname} $cfgfile &] + lappend ::pids $pid + + # Check availability + if {[server_is_up 127.0.0.1 $port 100] == 0} { + abort_sentinel_test "Problems starting $type #$j: ping timeout" + } + + # Push the instance into the right list + lappend ::${type}_instances [list \ + pid $pid \ + host 127.0.0.1 \ + port $port \ + link [redis 127.0.0.1 $port] \ + ] + } +} + +proc cleanup {} { + puts "Cleaning up..." + foreach pid $::pids { + catch { + if { $::tcl_platform(platform) == "windows" } { + kill_proc2 $pid + } else { + exec kill -9 $pid + } + } + } + foreach dir $::dirs { + catch {exec rm -rf $dir} + } +} + +proc abort_sentinel_test msg { + puts "WARNING: Aborting the test." + puts ">>>>>>>> $msg" + cleanup + exit 1 +} + +proc parse_options {} { + for {set j 0} {$j < [llength $::argv]} {incr j} { + set opt [lindex $::argv $j] + set val [lindex $::argv [expr $j+1]] + if {$opt eq "--single"} { + incr j + set ::run_matching "*${val}*" + } elseif {$opt eq "--pause-on-error"} { + set ::pause_on_error 1 + } elseif {$opt eq "--fail"} { + set ::simulate_error 1 + } elseif {$opt eq "--help"} { + puts "Hello, I'm sentinel.tcl and I run Sentinel unit tests." + puts "\nOptions:" + puts "--single Only runs tests specified by pattern." + puts "--pause-on-error Pause for manual inspection on error." + puts "--fail Simulate a test failure." + puts "--help Shows this help." + exit 0 + } else { + puts "Unknown option $opt" + exit 1 + } + } +} + +proc main {} { + parse_options + spawn_instance sentinel $::sentinel_base_port $::instances_count + spawn_instance redis $::redis_base_port $::instances_count + run_tests + cleanup +} + +# If --pause-on-error option was passed at startup this function is called +# on error in order to give the developer a chance to understand more about +# the error condition while the instances are still running. +proc pause_on_error {} { + puts "" + puts [colorstr yellow "*** Please inspect the error now ***"] + puts "\nType \"continue\" to resume the test, \"help\" for help screen.\n" + while 1 { + puts -nonewline "> " + flush stdout + set line [gets stdin] + set argv [split $line " "] + set cmd [lindex $argv 0] + if {$cmd eq {continue}} { + break + } elseif {$cmd eq {show-sentinel-logs}} { + set count 10 + if {[lindex $argv 1] ne {}} {set count [lindex $argv 1]} + foreach_sentinel_id id { + puts "=== SENTINEL $id ====" + puts [exec tail -$count sentinel_$id/log.txt] + puts "---------------------\n" + } + } elseif {$cmd eq {ls}} { + foreach_redis_id id { + puts -nonewline "Redis $id" + set errcode [catch { + set str {} + append str "@[RI $id tcp_port]: " + append str "[RI $id role] " + if {[RI $id role] eq {slave}} { + append str "[RI $id master_host]:[RI $id master_port]" + } + set str + } retval] + if {$errcode} { + puts " -- $retval" + } else { + puts $retval + } + } + foreach_sentinel_id id { + puts -nonewline "Sentinel $id" + set errcode [catch { + set str {} + append str "@[SI $id tcp_port]: " + append str "[join [S $id sentinel get-master-addr-by-name mymaster]]" + set str + } retval] + if {$errcode} { + puts " -- $retval" + } else { + puts $retval + } + } + } elseif {$cmd eq {help}} { + puts "ls List Sentinel and Redis instances." + puts "show-sentinel-logs \[N\] Show latest N lines of logs." + puts "S cmd ... arg Call command in Sentinel ." + puts "R cmd ... arg Call command in Redis ." + puts "SI Show Sentinel INFO ." + puts "RI Show Sentinel INFO ." + puts "continue Resume test." + } else { + set errcode [catch {eval $line} retval] + if {$retval ne {}} {puts "$retval"} + } + } +} + +# We redefine 'test' as for Sentinel we don't use the server-client +# architecture for the test, everything is sequential. +proc test {descr code} { + set ts [clock format [clock seconds] -format %H:%M:%S] + puts -nonewline "$ts> $descr: " + flush stdout + + if {[catch {set retval [uplevel 1 $code]} error]} { + if {[string match "assertion:*" $error]} { + set msg [string range $error 10 end] + puts [colorstr red $msg] + if {$::pause_on_error} pause_on_error + puts "(Jumping to next unit after error)" + return -code continue + } else { + # Re-raise, let handler up the stack take care of this. + error $error $::errorInfo + } + } else { + puts [colorstr green OK] + } +} + +proc run_tests {} { + set tests [lsort [glob ../sentinel-tests/*]] + foreach test $tests { + if {$::run_matching ne {} && [string match $::run_matching $test] == 0} { + continue + } + if {[file isdirectory $test]} continue + puts [colorstr yellow "Testing unit: [lindex [file split $test] end]"] + source $test + } +} + +# The "S" command is used to interact with the N-th Sentinel. +# The general form is: +# +# S command arg arg arg ... +# +# Example to ping the Sentinel 0 (first instance): S 0 PING +proc S {n args} { + set s [lindex $::sentinel_instances $n] + [dict get $s link] {*}$args +} + +# Like R but to chat with Redis instances. +proc R {n args} { + set r [lindex $::redis_instances $n] + [dict get $r link] {*}$args +} + +proc get_info_field {info field} { + set fl [string length $field] + append field : + foreach line [split $info "\n"] { + set line [string trim $line "\r\n "] + if {[string range $line 0 $fl] eq $field} { + return [string range $line [expr {$fl+1}] end] + } + } + return {} +} + +proc SI {n field} { + get_info_field [S $n info] $field +} + +proc RI {n field} { + get_info_field [R $n info] $field +} + +# Iterate over IDs of sentinel or redis instances. +proc foreach_instance_id {instances idvar code} { + upvar 1 $idvar id + for {set id 0} {$id < [llength $instances]} {incr id} { + set errcode [catch {uplevel 1 $code} result] + if {$errcode == 1} { + error $result $::errorInfo $::errorCode + } elseif {$errcode == 4} { + continue + } elseif {$errcode == 3} { + break + } elseif {$errcode != 0} { + return -code $errcode $result + } + } +} + +proc foreach_sentinel_id {idvar code} { + set errcode [catch {uplevel 1 [list foreach_instance_id $::sentinel_instances $idvar $code]} result] + return -code $errcode $result +} + +proc foreach_redis_id {idvar code} { + set errcode [catch {uplevel 1 [list foreach_instance_id $::redis_instances $idvar $code]} result] + return -code $errcode $result +} + +# Get the specific attribute of the specified instance type, id. +proc get_instance_attrib {type id attrib} { + dict get [lindex [set ::${type}_instances] $id] $attrib +} + +# Set the specific attribute of the specified instance type, id. +proc set_instance_attrib {type id attrib newval} { + set d [lindex [set ::${type}_instances] $id] + dict set d $attrib $newval + lset ::${type}_instances $id $d +} + +# Create a master-slave cluster of the given number of total instances. +# The first instance "0" is the master, all others are configured as +# slaves. +proc create_redis_master_slave_cluster n { + foreach_redis_id id { + if {$id == 0} { + # Our master. + R $id slaveof no one + R $id flushall + } elseif {$id < $n} { + R $id slaveof [get_instance_attrib redis 0 host] \ + [get_instance_attrib redis 0 port] + } else { + # Instances not part of the cluster. + R $id slaveof no one + } + } + # Wait for all the slaves to sync. + wait_for_condition 1000 50 { + [RI 0 connected_slaves] == ($n-1) + } else { + fail "Unable to create a master-slaves cluster." + } +} + +proc get_instance_id_by_port {type port} { + foreach_${type}_id id { + if {[get_instance_attrib $type $id port] == $port} { + return $id + } + } + fail "Instance $type port $port not found." +} + +# Kill an instance of the specified type/id with SIGKILL. +# This function will mark the instance PID as -1 to remember that this instance +# is no longer running and will remove its PID from the list of pids that +# we kill at cleanup. +# +# The instance can be restarted with restart-instance. +proc kill_instance {type id} { + set pid [get_instance_attrib $type $id pid] + if {$pid == -1} { + error "You tried to kill $type $id twice." + } + if { $::tcl_platform(platform) == "windows" } { + kill_proc2 $pid + } else { + exec kill -9 $pid + } + set_instance_attrib $type $id pid -1 + set_instance_attrib $type $id link you_tried_to_talk_with_killed_instance + + # Remove the PID from the list of pids to kill at exit. + set ::pids [lsearch -all -inline -not -exact $::pids $pid] +} + +# Return true of the instance of the specified type/id is killed. +proc instance_is_killed {type id} { + set pid [get_instance_attrib $type $id pid] + return $pid == -1 +} + +# Restart an instance previously killed by kill_instance +proc restart_instance {type id} { + set dirname "${type}_${id}" + set cfgfile [file join $dirname $type.conf] + set port [get_instance_attrib $type $id port] + + # Execute the instance with its old setup and append the new pid + # file for cleanup. + if {$type eq "redis"} { + set prgname redis-server + } else { + set prgname redis-sentinel + } + set pid [exec ../../src/${prgname} $cfgfile &] + set_instance_attrib $type $id pid $pid + lappend ::pids $pid + + # Check that the instance is running + if {[server_is_up 127.0.0.1 $port 100] == 0} { + abort_sentinel_test "Problems starting $type #$j: ping timeout" + } + + # Connect with it with a fresh link + set_instance_attrib $type $id link [redis 127.0.0.1 $port] +} + +if {[catch main e]} { + puts $::errorInfo + cleanup +} diff --git a/tests/support/server.tcl b/tests/support/server.tcl index 8b62ffd9..ffde9fcb 100644 --- a/tests/support/server.tcl +++ b/tests/support/server.tcl @@ -55,6 +55,10 @@ if { $tcl_platform(platform) != "windows" } { test "Check for memory leaks (pid $pid)" { set output {0 leaks} catch {exec leaks $pid} output + if {[string match {*process does not exist*} $output]} { + # In a few tests we kill the server process. + set output "0 leaks" + } set output } {*0 leaks*} } @@ -137,7 +141,7 @@ if { $tcl_platform(platform) != "windows" } { proc ping_server {host port} { set retval 0 if {[catch { - set fd [socket $::host $::port] + set fd [socket $host $port] fconfigure $fd -translation binary puts $fd "PING\r\n" flush $fd @@ -159,6 +163,22 @@ proc ping_server {host port} { return $retval } +# Return 1 if the server at the specified addr is reachable by PING, otherwise +# returns 0. Performs a try every 50 milliseconds for the specified number +# of retries. +proc server_is_up {host port retrynum} { + after 10 ;# Use a small delay to make likely a first-try success. + set retval 0 + while {[incr retrynum -1]} { + if {[catch {ping_server $host $port} ping]} { + set ping 0 + } + if {$ping} {return 1} + after 50 + } + return 0 +} + # doesn't really belong here, but highly coupled to code in start_server proc tags {tags code} { set ::tags [concat $::tags $tags] @@ -249,23 +269,13 @@ proc start_server {options {code undefined}} { # check that the server actually started # ugly but tries to be as fast as possible... if {$::valgrind} {set retrynum 1000} else {set retrynum 100} - set serverisup 0 if {$::verbose} { puts -nonewline "=== ($tags) Starting server ${::host}:${::port} " } - after 10 if {$code ne "undefined"} { - while {[incr retrynum -1]} { - catch { - if {[ping_server $::host $::port]} { - set serverisup 1 - } - } - if {$serverisup} break - after 50 - } + set serverisup [server_is_up $::host $::port $retrynum] } else { set serverisup 1 } diff --git a/tests/support/test.tcl b/tests/support/test.tcl index 480c674e..bf2cb0e2 100644 --- a/tests/support/test.tcl +++ b/tests/support/test.tcl @@ -53,41 +53,17 @@ proc assert_type {type key} { # executed. proc wait_for_condition {maxtries delay e _else_ elsescript} { while {[incr maxtries -1] >= 0} { - if {[uplevel 1 [list expr $e]]} break + set errcode [catch {uplevel 1 [list expr $e]} result] + if {$errcode == 0} { + if {$result} break + } else { + return -code $errcode $result + } after $delay } if {$maxtries == -1} { - uplevel 1 $elsescript - } -} - -# Test if TERM looks like to support colors -proc color_term {} { - expr {[info exists ::env(TERM)] && [string match *xterm* $::env(TERM)]} -} - -proc colorstr {color str} { - if {[color_term]} { - set b 0 - if {[string range $color 0 4] eq {bold-}} { - set b 1 - set color [string range $color 5 end] - } - switch $color { - red {set colorcode {31}} - green {set colorcode {32}} - yellow {set colorcode {33}} - blue {set colorcode {34}} - magenta {set colorcode {35}} - cyan {set colorcode {36}} - white {set colorcode {37}} - default {set colorcode {37}} - } - if {$colorcode ne {}} { - return "\033\[$b;${colorcode};40m$str\033\[0m" - } - } else { - return $str + set errcode [catch [uplevel 1 $elsescript] result] + return -code $errcode $result } } diff --git a/tests/support/util.tcl b/tests/support/util.tcl index c5a6853b..e49ea229 100644 --- a/tests/support/util.tcl +++ b/tests/support/util.tcl @@ -312,3 +312,48 @@ proc csvstring s { proc roundFloat f { format "%.10g" $f } + +proc find_available_port start { + for {set j $start} {$j < $start+1024} {incr j} { + if {[catch { + set fd [socket 127.0.0.1 $j] + }]} { + return $j + } else { + close $fd + } + } + if {$j == $start+1024} { + error "Can't find a non busy port in the $start-[expr {$start+1023}] range." + } +} + +# Test if TERM looks like to support colors +proc color_term {} { + expr {[info exists ::env(TERM)] && [string match *xterm* $::env(TERM)]} +} + +proc colorstr {color str} { + if {[color_term]} { + set b 0 + if {[string range $color 0 4] eq {bold-}} { + set b 1 + set color [string range $color 5 end] + } + switch $color { + red {set colorcode {31}} + green {set colorcode {32}} + yellow {set colorcode {33}} + blue {set colorcode {34}} + magenta {set colorcode {35}} + cyan {set colorcode {36}} + white {set colorcode {37}} + default {set colorcode {37}} + } + if {$colorcode ne {}} { + return "\033\[$b;${colorcode};40m$str\033\[0m" + } + } else { + return $str + } +} diff --git a/tests/test_helper.tcl b/tests/test_helper.tcl index 344fabb8..5f958e50 100644 --- a/tests/test_helper.tcl +++ b/tests/test_helper.tcl @@ -47,6 +47,7 @@ set ::all_tests { unit/dump unit/bitops unit/memefficiency + unit/hyperloglog } # Index to the next test to run in the ::all_tests list. set ::next_test 0 @@ -164,21 +165,6 @@ proc cleanup {} { if {!$::quiet} {puts "OK"} } -proc find_available_port start { - for {set j $start} {$j < $start+1024} {incr j} { - if {[catch { - set fd [socket 127.0.0.1 $j] - }]} { - return $j - } else { - close $fd - } - } - if {$j == $start+1024} { - error "Can't find a non busy port in the $start-[expr {$start+1023}] range." - } -} - proc test_server_main {} { cleanup set tclsh [info nameofexecutable] @@ -204,6 +190,7 @@ proc test_server_main {} { # Setup global state for the test server set ::idle_clients {} set ::active_clients {} + array set ::active_clients_task {} array set ::clients_start_time {} set ::clients_time_history {} set ::failed_tests {} @@ -217,9 +204,12 @@ proc test_server_main {} { # may be used in the future in order to detect test clients taking too much # time to execute the task. proc test_server_cron {} { + # Do some work here. + after 100 test_server_cron } proc accept_test_clients {fd addr port} { + fconfigure $fd -encoding binary fileevent $fd readable [list read_from_test_client $fd] } @@ -253,14 +243,17 @@ proc read_from_test_client fd { puts "\[$completed_tests_count/$all_tests_count [colorstr yellow $status]\]: $data ($elapsed seconds)" lappend ::clients_time_history $elapsed $data signal_idle_client $fd + set ::active_clients_task($fd) DONE } elseif {$status eq {ok}} { if {!$::quiet} { puts "\[[colorstr green $status]\]: $data" } + set ::active_clients_task($fd) "(OK) $data" } elseif {$status eq {err}} { set err "\[[colorstr red $status]\]: $data" puts $err lappend ::failed_tests $err + set ::active_clients_task($fd) "(ERR) $data" } elseif {$status eq {exception}} { puts "\[[colorstr red $status]\]: $data" foreach p $::clients_pids { @@ -268,7 +261,7 @@ proc read_from_test_client fd { } exit 1 } elseif {$status eq {testing}} { - # No op + set ::active_clients_task($fd) "(IN PROGRESS) $data" } else { if {!$::quiet} { puts "\[$status\]: $data" @@ -282,10 +275,24 @@ proc signal_idle_client fd { # Remove this fd from the list of active clients. set ::active_clients \ [lsearch -all -inline -not -exact $::active_clients $fd] + + if 0 { + # The following loop is only useful for debugging tests that may + # enter an infinite loop. Commented out normally. + foreach x $::active_clients { + if {[info exist ::active_clients_task($x)]} { + puts "$x => $::active_clients_task($x)" + } else { + puts "$x => ???" + } + } + } + # New unit to process? if {$::next_test != [llength $::all_tests]} { if {!$::quiet} { puts [colorstr bold-white "Testing [lindex $::all_tests $::next_test]"] + set ::active_clients_task($fd) "ASSIGNED: $fd ([lindex $::all_tests $::next_test])" } set ::clients_start_time($fd) [clock seconds] send_data_packet $fd run [lindex $::all_tests $::next_test] @@ -326,6 +333,7 @@ proc the_end {} { # to read the command, execute, reply... all this in a loop. proc test_client_main server_port { set ::test_server_fd [socket localhost $server_port] + fconfigure $::test_server_fd -encoding binary send_data_packet $::test_server_fd ready [pid] while 1 { set bytes [gets $::test_server_fd] diff --git a/tests/unit/bitops.tcl b/tests/unit/bitops.tcl index dade8923..89631098 100644 --- a/tests/unit/bitops.tcl +++ b/tests/unit/bitops.tcl @@ -52,7 +52,7 @@ start_server {tags {"bitops"}} { } } - test {BITCOUNT fuzzing} { + test {BITCOUNT fuzzing without start/end} { for {set j 0} {$j < 100} {incr j} { set str [randstring 0 3000] r set str $str @@ -60,6 +60,20 @@ start_server {tags {"bitops"}} { } } + test {BITCOUNT fuzzing with start/end} { + for {set j 0} {$j < 100} {incr j} { + set str [randstring 0 3000] + r set str $str + set l [string length $str] + set start [randomInt $l] + set end [randomInt $l] + if {$start > $end} { + lassign [list $end $start] start end + } + assert {[r bitcount str $start $end] == [count_bits [string range $str $start $end]]} + } + } + test {BITCOUNT with start, end} { r set s "foobar" assert_equal [r bitcount s 0 -1] [count_bits "foobar"] @@ -84,6 +98,18 @@ start_server {tags {"bitops"}} { } } {1} + test {BITCOUNT misaligned prefix} { + r del str + r set str ab + r bitcount str 1 -1 + } {3} + + test {BITCOUNT misaligned prefix + full words + remainder} { + r del str + r set str __PPxxxxxxxxxxxxxxxxRR__ + r bitcount str 2 -3 + } {74} + test {BITOP NOT (empty string)} { r set s "" r bitop not dest s @@ -177,4 +203,139 @@ start_server {tags {"bitops"}} { r set a "\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00" r bitop or x a b } {32} + + test {BITPOS bit=0 with empty key returns 0} { + r del str + r bitpos str 0 + } {0} + + test {BITPOS bit=1 with empty key returns -1} { + r del str + r bitpos str 1 + } {-1} + + test {BITPOS bit=0 with string less than 1 word works} { + r set str "\xff\xf0\x00" + r bitpos str 0 + } {12} + + test {BITPOS bit=1 with string less than 1 word works} { + r set str "\x00\x0f\x00" + r bitpos str 1 + } {12} + + test {BITPOS bit=0 starting at unaligned address} { + r set str "\xff\xf0\x00" + r bitpos str 0 1 + } {12} + + test {BITPOS bit=1 starting at unaligned address} { + r set str "\x00\x0f\xff" + r bitpos str 1 1 + } {12} + + test {BITPOS bit=0 unaligned+full word+reminder} { + r del str + r set str "\xff\xff\xff" ; # Prefix + # Followed by two (or four in 32 bit systems) full words + r append str "\xff\xff\xff\xff\xff\xff\xff\xff" + r append str "\xff\xff\xff\xff\xff\xff\xff\xff" + r append str "\xff\xff\xff\xff\xff\xff\xff\xff" + # First zero bit. + r append str "\x0f" + assert {[r bitpos str 0] == 216} + assert {[r bitpos str 0 1] == 216} + assert {[r bitpos str 0 2] == 216} + assert {[r bitpos str 0 3] == 216} + assert {[r bitpos str 0 4] == 216} + assert {[r bitpos str 0 5] == 216} + assert {[r bitpos str 0 6] == 216} + assert {[r bitpos str 0 7] == 216} + assert {[r bitpos str 0 8] == 216} + } + + test {BITPOS bit=1 unaligned+full word+reminder} { + r del str + r set str "\x00\x00\x00" ; # Prefix + # Followed by two (or four in 32 bit systems) full words + r append str "\x00\x00\x00\x00\x00\x00\x00\x00" + r append str "\x00\x00\x00\x00\x00\x00\x00\x00" + r append str "\x00\x00\x00\x00\x00\x00\x00\x00" + # First zero bit. + r append str "\xf0" + assert {[r bitpos str 1] == 216} + assert {[r bitpos str 1 1] == 216} + assert {[r bitpos str 1 2] == 216} + assert {[r bitpos str 1 3] == 216} + assert {[r bitpos str 1 4] == 216} + assert {[r bitpos str 1 5] == 216} + assert {[r bitpos str 1 6] == 216} + assert {[r bitpos str 1 7] == 216} + assert {[r bitpos str 1 8] == 216} + } + + test {BITPOS bit=1 returns -1 if string is all 0 bits} { + r set str "" + for {set j 0} {$j < 20} {incr j} { + assert {[r bitpos str 1] == -1} + r append str "\x00" + } + } + + test {BITPOS bit=0 works with intervals} { + r set str "\x00\xff\x00" + assert {[r bitpos str 0 0 -1] == 0} + assert {[r bitpos str 0 1 -1] == 16} + assert {[r bitpos str 0 2 -1] == 16} + assert {[r bitpos str 0 2 200] == 16} + assert {[r bitpos str 0 1 1] == -1} + } + + test {BITPOS bit=1 works with intervals} { + r set str "\x00\xff\x00" + assert {[r bitpos str 1 0 -1] == 8} + assert {[r bitpos str 1 1 -1] == 8} + assert {[r bitpos str 1 2 -1] == -1} + assert {[r bitpos str 1 2 200] == -1} + assert {[r bitpos str 1 1 1] == 8} + } + + test {BITPOS bit=0 changes behavior if end is given} { + r set str "\xff\xff\xff" + assert {[r bitpos str 0] == 24} + assert {[r bitpos str 0 0] == 24} + assert {[r bitpos str 0 0 -1] == -1} + } + + test {BITPOS bit=1 fuzzy testing using SETBIT} { + r del str + set max 524288; # 64k + set first_one_pos -1 + for {set j 0} {$j < 1000} {incr j} { + assert {[r bitpos str 1] == $first_one_pos} + set pos [randomInt $max] + r setbit str $pos 1 + if {$first_one_pos == -1 || $first_one_pos > $pos} { + # Update the position of the first 1 bit in the array + # if the bit we set is on the left of the previous one. + set first_one_pos $pos + } + } + } + + test {BITPOS bit=0 fuzzy testing using SETBIT} { + set max 524288; # 64k + set first_zero_pos $max + r set str [string repeat "\xff" [expr $max/8]] + for {set j 0} {$j < 1000} {incr j} { + assert {[r bitpos str 0] == $first_zero_pos} + set pos [randomInt $max] + r setbit str $pos 0 + if {$first_zero_pos > $pos} { + # Update the position of the first 0 bit in the array + # if the bit we clear is on the left of the previous one. + set first_zero_pos $pos + } + } + } } diff --git a/tests/unit/hyperloglog.tcl b/tests/unit/hyperloglog.tcl new file mode 100644 index 00000000..af86e68e --- /dev/null +++ b/tests/unit/hyperloglog.tcl @@ -0,0 +1,159 @@ +start_server {tags {"hll"}} { + test {HyperLogLog self test passes} { + catch {r pfselftest} e + set e + } {OK} + + test {PFADD without arguments creates an HLL value} { + r pfadd hll + r exists hll + } {1} + + test {Approximated cardinality after creation is zero} { + r pfcount hll + } {0} + + test {PFADD returns 1 when at least 1 reg was modified} { + r pfadd hll a b c + } {1} + + test {PFADD returns 0 when no reg was modified} { + r pfadd hll a b c + } {0} + + test {PFADD works with empty string (regression)} { + r pfadd hll "" + } + + # Note that the self test stresses much better the + # cardinality estimation error. We are testing just the + # command implementation itself here. + test {PFCOUNT returns approximated cardinality of set} { + r del hll + set res {} + r pfadd hll 1 2 3 4 5 + lappend res [r pfcount hll] + # Call it again to test cached value invalidation. + r pfadd hll 6 7 8 8 9 10 + lappend res [r pfcount hll] + set res + } {5 10} + + test {HyperLogLogs are promote from sparse to dense} { + r del hll + r config set hll-sparse-max-bytes 3000 + set n 0 + while {$n < 100000} { + set elements {} + for {set j 0} {$j < 100} {incr j} {lappend elements [expr rand()]} + incr n 100 + r pfadd hll {*}$elements + set card [r pfcount hll] + set err [expr {abs($card-$n)}] + assert {$err < (double($card)/100)*5} + if {$n < 1000} { + assert {[r pfdebug encoding hll] eq {sparse}} + } elseif {$n > 10000} { + assert {[r pfdebug encoding hll] eq {dense}} + } + } + } + + test {HyperLogLog sparse encoding stress test} { + for {set x 0} {$x < 1000} {incr x} { + r del hll1 hll2 + set numele [randomInt 100] + set elements {} + for {set j 0} {$j < $numele} {incr j} { + lappend elements [expr rand()] + } + # Force dense representation of hll2 + r pfadd hll2 + r pfdebug todense hll2 + r pfadd hll1 {*}$elements + r pfadd hll2 {*}$elements + assert {[r pfdebug encoding hll1] eq {sparse}} + assert {[r pfdebug encoding hll2] eq {dense}} + # Cardinality estimated should match exactly. + assert {[r pfcount hll1] eq [r pfcount hll2]} + } + } + + test {Corrupted sparse HyperLogLogs are detected: Additionl at tail} { + r del hll + r pfadd hll a b c + r append hll "hello" + set e {} + catch {r pfcount hll} e + set e + } {*INVALIDOBJ*} + + test {Corrupted sparse HyperLogLogs are detected: Broken magic} { + r del hll + r pfadd hll a b c + r setrange hll 0 "0123" + set e {} + catch {r pfcount hll} e + set e + } {*WRONGTYPE*} + + test {Corrupted sparse HyperLogLogs are detected: Invalid encoding} { + r del hll + r pfadd hll a b c + r setrange hll 4 "x" + set e {} + catch {r pfcount hll} e + set e + } {*WRONGTYPE*} + + test {Corrupted dense HyperLogLogs are detected: Wrong length} { + r del hll + r pfadd hll a b c + r setrange hll 4 "\x00" + set e {} + catch {r pfcount hll} e + set e + } {*WRONGTYPE*} + + test {PFADD, PFCOUNT, PFMERGE type checking works} { + r set foo bar + catch {r pfadd foo 1} e + assert_match {*WRONGTYPE*} $e + catch {r pfcount foo} e + assert_match {*WRONGTYPE*} $e + catch {r pfmerge bar foo} e + assert_match {*WRONGTYPE*} $e + catch {r pfmerge foo bar} e + assert_match {*WRONGTYPE*} $e + } + + test {PFMERGE results on the cardinality of union of sets} { + r del hll hll1 hll2 hll3 + r pfadd hll1 a b c + r pfadd hll2 b c d + r pfadd hll3 c d e + r pfmerge hll hll1 hll2 hll3 + r pfcount hll + } {5} + + test {PFCOUNT multiple-keys merge returns cardinality of union} { + r del hll1 hll2 hll3 + for {set x 1} {$x < 10000} {incr x} { + # Force dense representation of hll2 + r pfadd hll1 "foo-$x" + r pfadd hll2 "bar-$x" + r pfadd hll3 "zap-$x" + + set card [r pfcount hll1 hll2 hll3] + set realcard [expr {$x*3}] + set err [expr {abs($card-$realcard)}] + assert {$err < (double($card)/100)*5} + } + } + + test {PFDEBUG GETREG returns the HyperLogLog raw registers} { + r del hll + r pfadd hll 1 2 3 + llength [r pfdebug getreg hll] + } {16384} +} diff --git a/tests/unit/scripting.tcl b/tests/unit/scripting.tcl index ec5230bf..a17892d3 100644 --- a/tests/unit/scripting.tcl +++ b/tests/unit/scripting.tcl @@ -417,5 +417,17 @@ start_server {tags {"scripting repl"}} { } set res } {a 1} + + test {EVALSHA replication when first call is readonly} { + r del x + r eval {if tonumber(KEYS[1]) > 0 then redis.call('incr', 'x') end} 1 0 + r evalsha 38fe3ddf5284a1d48f37f824b4c4e826879f3cb9 1 0 + r evalsha 38fe3ddf5284a1d48f37f824b4c4e826879f3cb9 1 1 + wait_for_condition 50 100 { + [r -1 get x] eq {1} + } else { + fail "Expected 1 in x, but value is '[r -1 get x]'" + } + } } } diff --git a/tests/unit/type/zset.tcl b/tests/unit/type/zset.tcl index effb7d60..f07d5518 100644 --- a/tests/unit/type/zset.tcl +++ b/tests/unit/type/zset.tcl @@ -296,6 +296,62 @@ start_server {tags {"zset"}} { assert_error "*not*float*" {r zrangebyscore fooz 1 NaN} } + proc create_default_lex_zset {} { + create_zset zset {0 alpha 0 bar 0 cool 0 down + 0 elephant 0 foo 0 great 0 hill + 0 omega} + } + + test "ZRANGEBYLEX/ZREVRANGEBYLEX/ZCOUNT basics" { + create_default_lex_zset + + # inclusive range + assert_equal {alpha bar cool} [r zrangebylex zset - \[cool] + assert_equal {bar cool down} [r zrangebylex zset \[bar \[down] + assert_equal {great hill omega} [r zrangebylex zset \[g +] + assert_equal {cool bar alpha} [r zrevrangebylex zset \[cool -] + assert_equal {down cool bar} [r zrevrangebylex zset \[down \[bar] + assert_equal {omega hill great foo elephant down} [r zrevrangebylex zset + \[d] + assert_equal 3 [r zlexcount zset \[ele \[h] + + # exclusive range + assert_equal {alpha bar} [r zrangebylex zset - (cool] + assert_equal {cool} [r zrangebylex zset (bar (down] + assert_equal {hill omega} [r zrangebylex zset (great +] + assert_equal {bar alpha} [r zrevrangebylex zset (cool -] + assert_equal {cool} [r zrevrangebylex zset (down (bar] + assert_equal {omega hill} [r zrevrangebylex zset + (great] + assert_equal 2 [r zlexcount zset (ele (great] + + # inclusive and exclusive + assert_equal {} [r zrangebylex zset (az (b] + assert_equal {} [r zrangebylex zset (z +] + assert_equal {} [r zrangebylex zset - \[aaaa] + assert_equal {} [r zrevrangebylex zset \[elez \[elex] + assert_equal {} [r zrevrangebylex zset (hill (omega] + } + + test "ZRANGEBYSLEX with LIMIT" { + create_default_lex_zset + assert_equal {alpha bar} [r zrangebylex zset - \[cool LIMIT 0 2] + assert_equal {bar cool} [r zrangebylex zset - \[cool LIMIT 1 2] + assert_equal {} [r zrangebylex zset \[bar \[down LIMIT 0 0] + assert_equal {} [r zrangebylex zset \[bar \[down LIMIT 2 0] + assert_equal {bar} [r zrangebylex zset \[bar \[down LIMIT 0 1] + assert_equal {cool} [r zrangebylex zset \[bar \[down LIMIT 1 1] + assert_equal {bar cool down} [r zrangebylex zset \[bar \[down LIMIT 0 100] + assert_equal {omega hill great foo elephant} [r zrevrangebylex zset + \[d LIMIT 0 5] + assert_equal {omega hill great foo} [r zrevrangebylex zset + \[d LIMIT 0 4] + } + + test "ZRANGEBYLEX with invalid lex range specifiers" { + assert_error "*not*string*" {r zrangebylex fooz foo bar} + assert_error "*not*string*" {r zrangebylex fooz \[foo bar} + assert_error "*not*string*" {r zrangebylex fooz foo \[bar} + assert_error "*not*string*" {r zrangebylex fooz +x \[bar} + assert_error "*not*string*" {r zrangebylex fooz -x \[bar} + } + test "ZREMRANGEBYSCORE basics" { proc remrangebyscore {min max} { create_zset zset {1 a 2 b 3 c 4 d 5 e} @@ -712,6 +768,111 @@ start_server {tags {"zset"}} { assert_equal {} $err } + test "ZRANGEBYLEX fuzzy test, 100 ranges in $elements element sorted set - $encoding" { + set lexset {} + r del zset + for {set j 0} {$j < $elements} {incr j} { + set e [randstring 0 30 alpha] + lappend lexset $e + r zadd zset 0 $e + } + set lexset [lsort -unique $lexset] + for {set j 0} {$j < 100} {incr j} { + set min [randstring 0 30 alpha] + set max [randstring 0 30 alpha] + set mininc [randomInt 2] + set maxinc [randomInt 2] + if {$mininc} {set cmin "\[$min"} else {set cmin "($min"} + if {$maxinc} {set cmax "\[$max"} else {set cmax "($max"} + set rev [randomInt 2] + if {$rev} { + set cmd zrevrangebylex + } else { + set cmd zrangebylex + } + + # Make sure data is the same in both sides + assert {[r zrange zset 0 -1] eq $lexset} + + # Get the Redis output + set output [r $cmd zset $cmin $cmax] + if {$rev} { + set outlen [r zlexcount zset $cmax $cmin] + } else { + set outlen [r zlexcount zset $cmin $cmax] + } + + # Compute the same output via Tcl + set o {} + set copy $lexset + if {(!$rev && [string compare $min $max] > 0) || + ($rev && [string compare $max $min] > 0)} { + # Empty output when ranges are inverted. + } else { + if {$rev} { + # Invert the Tcl array using Redis itself. + set copy [r zrevrange zset 0 -1] + # Invert min / max as well + lassign [list $min $max $mininc $maxinc] \ + max min maxinc mininc + } + foreach e $copy { + set mincmp [string compare $e $min] + set maxcmp [string compare $e $max] + if { + ($mininc && $mincmp >= 0 || !$mininc && $mincmp > 0) + && + ($maxinc && $maxcmp <= 0 || !$maxinc && $maxcmp < 0) + } { + lappend o $e + } + } + } + assert {$o eq $output} + assert {$outlen eq [llength $output]} + } + } + + test "ZREMRANGEBYLEX fuzzy test, 100 ranges in $elements element sorted set - $encoding" { + set lexset {} + r del zset zsetcopy + for {set j 0} {$j < $elements} {incr j} { + set e [randstring 0 30 alpha] + lappend lexset $e + r zadd zset 0 $e + } + set lexset [lsort -unique $lexset] + for {set j 0} {$j < 100} {incr j} { + # Copy... + r zunionstore zsetcopy 1 zset + set lexsetcopy $lexset + + set min [randstring 0 30 alpha] + set max [randstring 0 30 alpha] + set mininc [randomInt 2] + set maxinc [randomInt 2] + if {$mininc} {set cmin "\[$min"} else {set cmin "($min"} + if {$maxinc} {set cmax "\[$max"} else {set cmax "($max"} + + # Make sure data is the same in both sides + assert {[r zrange zset 0 -1] eq $lexset} + + # Get the range we are going to remove + set torem [r zrangebylex zset $cmin $cmax] + set toremlen [r zlexcount zset $cmin $cmax] + r zremrangebylex zsetcopy $cmin $cmax + set output [r zrange zsetcopy 0 -1] + + # Remove the range with Tcl from the original list + if {$toremlen} { + set first [lsearch -exact $lexsetcopy [lindex $torem 0]] + set last [expr {$first+$toremlen-1}] + set lexsetcopy [lreplace $lexsetcopy $first $last] + } + assert {$lexsetcopy eq $output} + } + } + test "ZSETs skiplist implementation backlink consistency test - $encoding" { set diff 0 for {set j 0} {$j < $elements} {incr j} { diff --git a/utils/hyperloglog/.gitignore b/utils/hyperloglog/.gitignore new file mode 100644 index 00000000..2211df63 --- /dev/null +++ b/utils/hyperloglog/.gitignore @@ -0,0 +1 @@ +*.txt diff --git a/utils/hyperloglog/hll-err.rb b/utils/hyperloglog/hll-err.rb new file mode 100644 index 00000000..75bb8e42 --- /dev/null +++ b/utils/hyperloglog/hll-err.rb @@ -0,0 +1,27 @@ +# hll-err.rb - Copyright (C) 2014 Salvatore Sanfilippo +# BSD license, See the COPYING file for more information. +# +# Check error of HyperLogLog Redis implementation for different set sizes. + +require 'rubygems' +require 'redis' +require 'digest/sha1' + +r = Redis.new +r.del('hll') +i = 0 +while true do + 100.times { + elements = [] + 1000.times { + ele = Digest::SHA1.hexdigest(i.to_s) + elements << ele + i += 1 + } + r.pfadd('hll',*elements) + } + approx = r.pfcount('hll') + abs_err = (approx-i).abs + rel_err = 100.to_f*abs_err/i + puts "#{i} vs #{approx}: #{rel_err}%" +end diff --git a/utils/hyperloglog/hll-gnuplot-graph.rb b/utils/hyperloglog/hll-gnuplot-graph.rb new file mode 100644 index 00000000..745baddc --- /dev/null +++ b/utils/hyperloglog/hll-gnuplot-graph.rb @@ -0,0 +1,88 @@ +# hll-err.rb - Copyright (C) 2014 Salvatore Sanfilippo +# BSD license, See the COPYING file for more information. +# +# This program is suited to output average and maximum errors of +# the Redis HyperLogLog implementation in a format suitable to print +# graphs using gnuplot. + +require 'rubygems' +require 'redis' +require 'digest/sha1' + +# Generate an array of [cardinality,relative_error] pairs +# in the 0 - max range, with the specified step. +# +# 'r' is the Redis object used to perform the queries. +# 'seed' must be different every time you want a test performed +# with a different set. The function guarantees that if 'seed' is the +# same, exactly the same dataset is used, and when it is different, +# a totally unrelated different data set is used (without any common +# element in practice). +def run_experiment(r,seed,max,step) + r.del('hll') + i = 0 + samples = [] + step = 1000 if step > 1000 + while i < max do + elements = [] + step.times { + ele = Digest::SHA1.hexdigest(i.to_s+seed.to_s) + elements << ele + i += 1 + } + r.pfadd('hll',*elements) + approx = r.pfcount('hll') + err = approx-i + rel_err = 100.to_f*err/i + samples << [i,rel_err] + end + samples +end + +def filter_samples(numsets,max,step,filter) + r = Redis.new + dataset = {} + (0...numsets).each{|i| + dataset[i] = run_experiment(r,i,max,step) + STDERR.puts "Set #{i}" + } + dataset[0].each_with_index{|ele,index| + if filter == :max + card=ele[0] + err=ele[1].abs + (1...numsets).each{|i| + err = dataset[i][index][1] if err < dataset[i][index][1] + } + puts "#{card} #{err}" + elsif filter == :avg + card=ele[0] + err = 0 + (0...numsets).each{|i| + err += dataset[i][index][1] + } + err /= numsets + puts "#{card} #{err}" + elsif filter == :absavg + card=ele[0] + err = 0 + (0...numsets).each{|i| + err += dataset[i][index][1].abs + } + err /= numsets + puts "#{card} #{err}" + elsif filter == :all + (0...numsets).each{|i| + card,err = dataset[i][index] + puts "#{card} #{err}" + } + else + raise "Unknown filter #{filter}" + end + } +end + +if ARGV.length != 4 + puts "Usage: hll-gnuplot-graph (max|avg|absavg|all)" + exit 1 +end +filter_samples(ARGV[0].to_i,ARGV[1].to_i,ARGV[2].to_i,ARGV[3].to_sym) diff --git a/utils/install_server.sh b/utils/install_server.sh index c5ca944e..15b60a08 100755 --- a/utils/install_server.sh +++ b/utils/install_server.sh @@ -30,46 +30,48 @@ # this scripts should be run as root die () { - echo "ERROR: $1. Aborting!" + echo "ERROR: $1. Aborting!" exit 1 } + +#Absolute path to this script +SCRIPT=$(readlink -f $0) +#Absolute path this script is in +SCRIPTPATH=$(dirname $SCRIPT) + #Initial defaults _REDIS_PORT=6379 echo "Welcome to the redis service installer" -echo "This script will help you easily set up a running redis server +echo "This script will help you easily set up a running redis server" +echo -" - -#check for root user TODO: replace this with a call to "id" -if [ `whoami` != "root" ] ; then +#check for root user +if [ "$(id -u)" -ne 0 ] ; then echo "You must run this script as root. Sorry!" exit 1 fi - #Read the redis port -read -p "Please select the redis port for this instance: [$_REDIS_PORT] " REDIS_PORT -if [ ! `echo $REDIS_PORT | egrep "^[0-9]+\$"` ] ; then +read -p "Please select the redis port for this instance: [$_REDIS_PORT] " REDIS_PORT +if ! echo $REDIS_PORT | egrep -q '^[0-9]+$' ; then echo "Selecting default: $_REDIS_PORT" - REDIS_PORT=$_REDIS_PORT + REDIS_PORT=$_REDIS_PORT fi #read the redis config file _REDIS_CONFIG_FILE="/etc/redis/$REDIS_PORT.conf" read -p "Please select the redis config file name [$_REDIS_CONFIG_FILE] " REDIS_CONFIG_FILE -if [ !"$REDIS_CONFIG_FILE" ] ; then +if [ -z "$REDIS_CONFIG_FILE" ] ; then REDIS_CONFIG_FILE=$_REDIS_CONFIG_FILE echo "Selected default - $REDIS_CONFIG_FILE" fi -#try and create it -mkdir -p `dirname "$REDIS_CONFIG_FILE"` || die "Could not create redis config directory" #read the redis log file path _REDIS_LOG_FILE="/var/log/redis_$REDIS_PORT.log" read -p "Please select the redis log file name [$_REDIS_LOG_FILE] " REDIS_LOG_FILE -if [ !"$REDIS_LOG_FILE" ] ; then +if [ -z "$REDIS_LOG_FILE" ] ; then REDIS_LOG_FILE=$_REDIS_LOG_FILE echo "Selected default - $REDIS_LOG_FILE" fi @@ -78,55 +80,71 @@ fi #get the redis data directory _REDIS_DATA_DIR="/var/lib/redis/$REDIS_PORT" read -p "Please select the data directory for this instance [$_REDIS_DATA_DIR] " REDIS_DATA_DIR -if [ !"$REDIS_DATA_DIR" ] ; then +if [ -z "$REDIS_DATA_DIR" ] ; then REDIS_DATA_DIR=$_REDIS_DATA_DIR echo "Selected default - $REDIS_DATA_DIR" fi -mkdir -p $REDIS_DATA_DIR || die "Could not create redis data directory" #get the redis executable path -_REDIS_EXECUTABLE=`which redis-server` +_REDIS_EXECUTABLE=`command -v redis-server` read -p "Please select the redis executable path [$_REDIS_EXECUTABLE] " REDIS_EXECUTABLE -if [ ! -f "$REDIS_EXECUTABLE" ] ; then +if [ ! -x "$REDIS_EXECUTABLE" ] ; then REDIS_EXECUTABLE=$_REDIS_EXECUTABLE - - if [ ! -f "$REDIS_EXECUTABLE" ] ; then + + if [ ! -x "$REDIS_EXECUTABLE" ] ; then echo "Mmmmm... it seems like you don't have a redis executable. Did you run make install yet?" exit 1 fi - fi - -#render the tmplates -TMP_FILE="/tmp/$REDIS_PORT.conf" -DEFAULT_CONFIG="../redis.conf" -INIT_TPL_FILE="./redis_init_script.tpl" -INIT_SCRIPT_DEST="/etc/init.d/redis_$REDIS_PORT" -PIDFILE="/var/run/redis_$REDIS_PORT.pid" - - - #check the default for redis cli -CLI_EXEC=`which redis-cli` -if [ ! "$CLI_EXEC" ] ; then +CLI_EXEC=`command -v redis-cli` +if [ -z "$CLI_EXEC" ] ; then CLI_EXEC=`dirname $REDIS_EXECUTABLE`"/redis-cli" fi +echo "Selected config:" + +echo "Port : $REDIS_PORT" +echo "Config file : $REDIS_CONFIG_FILE" +echo "Log file : $REDIS_LOG_FILE" +echo "Data dir : $REDIS_DATA_DIR" +echo "Executable : $REDIS_EXECUTABLE" +echo "Cli Executable : $CLI_EXEC" + +read -p "Is this ok? Then press ENTER to go on or Ctrl-C to abort." _UNUSED_ + +mkdir -p `dirname "$REDIS_CONFIG_FILE"` || die "Could not create redis config directory" +mkdir -p `dirname "$REDIS_LOG_FILE"` || die "Could not create redis log dir" +mkdir -p "$REDIS_DATA_DIR" || die "Could not create redis data directory" + +#render the templates +TMP_FILE="/tmp/${REDIS_PORT}.conf" +DEFAULT_CONFIG="${SCRIPTPATH}/../redis.conf" +INIT_TPL_FILE="${SCRIPTPATH}/redis_init_script.tpl" +INIT_SCRIPT_DEST="/etc/init.d/redis_${REDIS_PORT}" +PIDFILE="/var/run/redis_${REDIS_PORT}.pid" + +if [ ! -f "$DEFAULT_CONFIG" ]; then + echo "Mmmmm... the default config is missing. Did you switch to the utils directory?" + exit 1 +fi + #Generate config file from the default config file as template #changing only the stuff we're controlling from this script echo "## Generated by install_server.sh ##" > $TMP_FILE -SED_EXPR="s#^port [0-9]{4}\$#port ${REDIS_PORT}#;\ -s#^logfile .+\$#logfile ${REDIS_LOG_FILE}#;\ -s#^dir .+\$#dir ${REDIS_DATA_DIR}#;\ -s#^pidfile .+\$#pidfile ${PIDFILE}#;\ -s#^daemonize no\$#daemonize yes#;" -echo $SED_EXPR +read -r SED_EXPR <<-EOF +s#^port [0-9]{4}\$#port ${REDIS_PORT}#; \ +s#^logfile .+\$#logfile ${REDIS_LOG_FILE}#; \ +s#^dir .+\$#dir ${REDIS_DATA_DIR}#; \ +s#^pidfile .+\$#pidfile ${PIDFILE}#; \ +s#^daemonize no\$#daemonize yes#; +EOF sed -r "$SED_EXPR" $DEFAULT_CONFIG >> $TMP_FILE #cat $TPL_FILE | while read line; do eval "echo \"$line\"" >> $TMP_FILE; done -cp -f $TMP_FILE $REDIS_CONFIG_FILE || exit 1 +cp $TMP_FILE $REDIS_CONFIG_FILE || die "Could not write redis config file $REDIS_CONFIG_FILE" #Generate sample script from template file rm -f $TMP_FILE @@ -138,7 +156,7 @@ REDIS_INIT_HEADER=\ #Configurations injected by install_server below....\n\n EXEC=$REDIS_EXECUTABLE\n CLIEXEC=$CLI_EXEC\n -PIDFILE=$PIDFILE\n +PIDFILE=\"$PIDFILE\"\n CONF=\"$REDIS_CONFIG_FILE\"\n\n REDISPORT=\"$REDIS_PORT\"\n\n ###############\n\n" @@ -146,45 +164,82 @@ REDISPORT=\"$REDIS_PORT\"\n\n REDIS_CHKCONFIG_INFO=\ "# REDHAT chkconfig header\n\n # chkconfig: - 58 74\n -# description: redis_6379 is the redis daemon.\n +# description: redis_${REDIS_PORT} is the redis daemon.\n ### BEGIN INIT INFO\n # Provides: redis_6379\n -# Required-Start: $network $local_fs $remote_fs\n -# Required-Stop: $network $local_fs $remote_fs\n +# Required-Start: \$network \$local_fs \$remote_fs\n +# Required-Stop: \$network \$local_fs \$remote_fs\n # Default-Start: 2 3 4 5\n # Default-Stop: 0 1 6\n -# Should-Start: $syslog $named\n -# Should-Stop: $syslog $named\n -# Short-Description: start and stop redis_6379\n +# Should-Start: \$syslog \$named\n +# Should-Stop: \$syslog \$named\n +# Short-Description: start and stop redis_${REDIS_PORT}\n # Description: Redis daemon\n ### END INIT INFO\n\n" -if [ !`which chkconfig` ] ; then - #combine the header and the template (which is actually a static footer) - echo $REDIS_INIT_HEADER > $TMP_FILE && cat $INIT_TPL_FILE >> $TMP_FILE || die "Could not write init script to $TMP_FILE" -else +if command -v chkconfig >/dev/null; then #if we're a box with chkconfig on it we want to include info for chkconfig - echo -e $REDIS_INIT_HEADER $REDIS_CHKCONFIG_INFO > $TMP_FILE && cat $INIT_TPL_FILE >> $TMP_FILE || die "Could not write init script to $TMP_FILE" + echo "$REDIS_INIT_HEADER" "$REDIS_CHKCONFIG_INFO" > $TMP_FILE && cat $INIT_TPL_FILE >> $TMP_FILE || die "Could not write init script to $TMP_FILE" +else + #combine the header and the template (which is actually a static footer) + echo "$REDIS_INIT_HEADER" > $TMP_FILE && cat $INIT_TPL_FILE >> $TMP_FILE || die "Could not write init script to $TMP_FILE" fi +### +# Generate sample script from template file +# - No need to check which system we are on. The init info are comments and +# do not interfere with update_rc.d systems. Additionally: +# Ubuntu/debian by default does not come with chkconfig, but does issue a +# warning if init info is not available. + +cat > ${TMP_FILE} <> ${TMP_FILE} + #copy to /etc/init.d -cp -f $TMP_FILE $INIT_SCRIPT_DEST && chmod +x $INIT_SCRIPT_DEST || die "Could not copy redis init script to $INIT_SCRIPT_DEST" +cp $TMP_FILE $INIT_SCRIPT_DEST && \ + chmod +x $INIT_SCRIPT_DEST || die "Could not copy redis init script to $INIT_SCRIPT_DEST" echo "Copied $TMP_FILE => $INIT_SCRIPT_DEST" #Install the service echo "Installing service..." -if [ !`which chkconfig` ] ; then - #if we're not a chkconfig box assume we're able to use update-rc.d - update-rc.d redis_$REDIS_PORT defaults && echo "Success!" -else +if command -v chkconfig >/dev/null 2>&1; then # we're chkconfig, so lets add to chkconfig and put in runlevel 345 - chkconfig --add redis_$REDIS_PORT && echo "Successfully added to chkconfig!" - chkconfig --level 345 redis_$REDIS_PORT on && echo "Successfully added to runlevels 345!" + chkconfig --add redis_${REDIS_PORT} && echo "Successfully added to chkconfig!" + chkconfig --level 345 redis_${REDIS_PORT} on && echo "Successfully added to runlevels 345!" +elif command -v update-rc.d >/dev/null 2>&1; then + #if we're not a chkconfig box assume we're able to use update-rc.d + update-rc.d redis_${REDIS_PORT} defaults && echo "Success!" +else + echo "No supported init tool found." fi - + /etc/init.d/redis_$REDIS_PORT start || die "Failed starting service..." #tada echo "Installation successful!" exit 0 - diff --git a/utils/redis_init_script.tpl b/utils/redis_init_script.tpl index e2af1fe7..d6508631 100755 --- a/utils/redis_init_script.tpl +++ b/utils/redis_init_script.tpl @@ -3,29 +3,41 @@ case "$1" in start) if [ -f $PIDFILE ] then - echo "$PIDFILE exists, process is already running or crashed" + echo "$PIDFILE exists, process is already running or crashed" else - echo "Starting Redis server..." - $EXEC $CONF + echo "Starting Redis server..." + $EXEC $CONF fi ;; stop) if [ ! -f $PIDFILE ] then - echo "$PIDFILE does not exist, process is not running" + echo "$PIDFILE does not exist, process is not running" else - PID=$(cat $PIDFILE) - echo "Stopping ..." - $CLIEXEC -p $REDISPORT shutdown - while [ -x /proc/${PID} ] - do - echo "Waiting for Redis to shutdown ..." - sleep 1 - done - echo "Redis stopped" + PID=$(cat $PIDFILE) + echo "Stopping ..." + $CLIEXEC -p $REDISPORT shutdown + while [ -x /proc/${PID} ] + do + echo "Waiting for Redis to shutdown ..." + sleep 1 + done + echo "Redis stopped" fi ;; + status) + if [ ! -f $PIDFILE ] + then + echo 'Redis is not running' + else + echo "Redis is running ($(<$PIDFILE))" + fi + ;; + restart) + $0 stop + $0 start + ;; *) - echo "Please use start or stop as first argument" + echo "Please use start, stop, restart or status as first argument" ;; esac