Redis/redis-For-Windows
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 3 Wiki Activity

Compare commits

base: Redis/redis-For-Windows:3.0.1
Branches Tags
Redis/redis-For-Windows:develop Redis/redis-For-Windows:win-5.0 Redis/redis-For-Windows:win-6.2
Redis/redis-For-Windows:5.0.14.1 Redis/redis-For-Windows:v5.0.14.1 Redis/redis-For-Windows:v5.0.14 Redis/redis-For-Windows:v5.0.10 Redis/redis-For-Windows:v5.0.9 Redis/redis-For-Windows:v4.0.14.2 Redis/redis-For-Windows:v4.0.14.1 Redis/redis-For-Windows:v4.0.14 Redis/redis-For-Windows:v4.0.2.3-alpha Redis/redis-For-Windows:v4.0.2.2-alpha Redis/redis-For-Windows:v4.0.2.1-alpha Redis/redis-For-Windows:v4.0.2-alpha Redis/redis-For-Windows:win-3.2.100 Redis/redis-For-Windows:win-3.0.504 Redis/redis-For-Windows:win-3.0.503 Redis/redis-For-Windows:win-2.8.2402 Redis/redis-For-Windows:win-3.0.502 Redis/redis-For-Windows:win-2.8.2401 Redis/redis-For-Windows:3.2.1 Redis/redis-For-Windows:win-3.2.0000-preview Redis/redis-For-Windows:3.2.0 Redis/redis-For-Windows:3.2.0-rc3 Redis/redis-For-Windows:3.2.0-rc2 Redis/redis-For-Windows:win-2.8.2400 Redis/redis-For-Windows:win-3.0.501 Redis/redis-For-Windows:3.2-rc1 Redis/redis-For-Windows:3.0.6 Redis/redis-For-Windows:2.8.24 Redis/redis-For-Windows:win-3.0.500 Redis/redis-For-Windows:win-3.0.500-rc2 Redis/redis-For-Windows:win-3.0.500-rc1 Redis/redis-For-Windows:win-2.8.2104 Redis/redis-For-Windows:3.0.5 Redis/redis-For-Windows:2.8.23 Redis/redis-For-Windows:win-3.0.300-beta1 Redis/redis-For-Windows:win-2.8.2103 Redis/redis-For-Windows:2.8.22 Redis/redis-For-Windows:3.0.4 Redis/redis-For-Windows:win-3.0.300-alpha3 Redis/redis-For-Windows:win-2.8.2102 Redis/redis-For-Windows:win-3.0.300-alpha2 Redis/redis-For-Windows:win-3.0.100-alpha1 Redis/redis-For-Windows:3.0.3 Redis/redis-For-Windows:win-2.8.2101 Redis/redis-For-Windows:win-2.8.21 Redis/redis-For-Windows:3.0.2 Redis/redis-For-Windows:2.8.21 Redis/redis-For-Windows:2.8.20 Redis/redis-For-Windows:3.0.1 Redis/redis-For-Windows:win-2.8.19.1 Redis/redis-For-Windows:3.0.0 Redis/redis-For-Windows:3.0.0-rc6 Redis/redis-For-Windows:3.0.0-rc5 Redis/redis-For-Windows:win-2.8.19-rc1 Redis/redis-For-Windows:win-2.8.19 Redis/redis-For-Windows:3.0.0-rc4 Redis/redis-For-Windows:win-2.8.19-pre2 Redis/redis-For-Windows:win-2.8.17.4 Redis/redis-For-Windows:3.0.0-rc3 Redis/redis-For-Windows:3.0.0-rc2 Redis/redis-For-Windows:win-2.8.17.3 Redis/redis-For-Windows:win-2.8.19-pre1 Redis/redis-For-Windows:win-2.8.17.2 Redis/redis-For-Windows:win-2.8.17.1 Redis/redis-For-Windows:2.8.19 Redis/redis-For-Windows:2.8.18 Redis/redis-For-Windows:3.0.0-rc1 Redis/redis-For-Windows:2.8.17 Redis/redis-For-Windows:2.8.16 Redis/redis-For-Windows:2.8.15 Redis/redis-For-Windows:win2.8.12 Redis/redis-For-Windows:2.8.14 Redis/redis-For-Windows:3.0.0-beta8 Redis/redis-For-Windows:2.8.13 Redis/redis-For-Windows:3.0.0-beta7 Redis/redis-For-Windows:win_2.8.9 Redis/redis-For-Windows:2.8.12 Redis/redis-For-Windows:2.8.11 Redis/redis-For-Windows:3.0.0-beta6 Redis/redis-For-Windows:2.8.10 Redis/redis-For-Windows:3.0.0-beta5 Redis/redis-For-Windows:3.0.0-beta4 Redis/redis-For-Windows:2.6.14_msopentech Redis/redis-For-Windows:2.8.4_msopentech Redis/redis-For-Windows:msopentech_2.8.4 Redis/redis-For-Windows:3.0.0-beta3 Redis/redis-For-Windows:2.8.9 Redis/redis-For-Windows:2.8.8 Redis/redis-For-Windows:3.0.0-beta2 Redis/redis-For-Windows:2.8.7 Redis/redis-For-Windows:2.8.6 Redis/redis-For-Windows:3.0.0-beta1 Redis/redis-For-Windows:2.8.5 Redis/redis-For-Windows:2.8.4 Redis/redis-For-Windows:2.8.3 Redis/redis-For-Windows:2.8.2 Redis/redis-For-Windows:2.8.1 Redis/redis-For-Windows:2.8.0 Redis/redis-For-Windows:2.8.0-rc6 Redis/redis-For-Windows:2.8.0-rc5 Redis/redis-For-Windows:2.8.0-rc4 Redis/redis-For-Windows:2.8.0-rc3 Redis/redis-For-Windows:2.8.0-rc2 Redis/redis-For-Windows:2.8.0-rc1 Redis/redis-For-Windows:2.6.14 Redis/redis-For-Windows:Win2.6.8 Redis/redis-For-Windows:2.6.13 Redis/redis-For-Windows:2.6.12 Redis/redis-For-Windows:2.6.11 Redis/redis-For-Windows:2.6.10 Redis/redis-For-Windows:2.6.9 Redis/redis-For-Windows:2.6.8 Redis/redis-For-Windows:2.6.7 Redis/redis-For-Windows:2.6.6 Redis/redis-For-Windows:2.6.5 Redis/redis-For-Windows:2.6.4 Redis/redis-For-Windows:2.6.3 Redis/redis-For-Windows:2.6.2 Redis/redis-For-Windows:2.6.1 Redis/redis-For-Windows:2.6.0 Redis/redis-For-Windows:2.6.0-rc8 Redis/redis-For-Windows:2.6.0-rc7 Redis/redis-For-Windows:2.6.0-rc6 Redis/redis-For-Windows:2.6.0-rc5 Redis/redis-For-Windows:2.6.0-rc4 Redis/redis-For-Windows:2.6.0-rc3 Redis/redis-For-Windows:2.6.0-rc2 Redis/redis-For-Windows:2.6.0-rc1 Redis/redis-For-Windows:2.4.11 Redis/redis-For-Windows:2.4.10 Redis/redis-For-Windows:2.4.9 Redis/redis-For-Windows:2.4.8 Redis/redis-For-Windows:2.4.6_win32 Redis/redis-For-Windows:2.4.7 Redis/redis-For-Windows:2.4.6 Redis/redis-For-Windows:2.4.5 Redis/redis-For-Windows:2.4.4 Redis/redis-For-Windows:2.4.3 Redis/redis-For-Windows:2.4.2 Redis/redis-For-Windows:2.4.1 Redis/redis-For-Windows:2.4.0 Redis/redis-For-Windows:2.4.0-rc8 Redis/redis-For-Windows:2.4.0-rc7 Redis/redis-For-Windows:2.4.0-rc6 Redis/redis-For-Windows:2.4.0-rc5 Redis/redis-For-Windows:2.4.0-rc4 Redis/redis-For-Windows:2.4.0-rc3 Redis/redis-For-Windows:with-deprecated-diskstore Redis/redis-For-Windows:2.4.0-rc2 Redis/redis-For-Windows:2.4.0-rc1 Redis/redis-For-Windows:2.2.4 Redis/redis-For-Windows:2.2.3 Redis/redis-For-Windows:3.0-alpha0 Redis/redis-For-Windows:2.2.2 Redis/redis-For-Windows:2.2.1 Redis/redis-For-Windows:2.2.0 Redis/redis-For-Windows:2.2.0-rc4 Redis/redis-For-Windows:2.3-alpha0 Redis/redis-For-Windows:2.2.0-rc3 Redis/redis-For-Windows:2.2.0-rc2 Redis/redis-For-Windows:2.2.0-rc1 Redis/redis-For-Windows:2.2-alpha6 Redis/redis-For-Windows:2.2-alpha5 Redis/redis-For-Windows:2.2-alpha4 Redis/redis-For-Windows:2.2-alpha3 Redis/redis-For-Windows:2.2-alpha2 Redis/redis-For-Windows:2.2-alpha1 Redis/redis-For-Windows:2.2-alpha0 Redis/redis-For-Windows:v2.1.1-watch Redis/redis-For-Windows:v2.0.0-rc1 Redis/redis-For-Windows:v1.3.12 Redis/redis-For-Windows:v1.3.11 Redis/redis-For-Windows:v1.3.10 Redis/redis-For-Windows:v1.3.9 Redis/redis-For-Windows:v1.3.8 Redis/redis-For-Windows:v1.3.7 Redis/redis-For-Windows:1.3.6 Redis/redis-For-Windows:vm-playpen
..
compare: Redis/redis-For-Windows:2.8.22
Branches Tags
Redis/redis-For-Windows:win-5.0 Redis/redis-For-Windows:develop Redis/redis-For-Windows:win-6.2
Redis/redis-For-Windows:5.0.14.1 Redis/redis-For-Windows:v5.0.14.1 Redis/redis-For-Windows:v5.0.14 Redis/redis-For-Windows:v5.0.10 Redis/redis-For-Windows:v5.0.9 Redis/redis-For-Windows:v4.0.14.2 Redis/redis-For-Windows:v4.0.14.1 Redis/redis-For-Windows:v4.0.14 Redis/redis-For-Windows:v4.0.2.3-alpha Redis/redis-For-Windows:v4.0.2.2-alpha Redis/redis-For-Windows:v4.0.2.1-alpha Redis/redis-For-Windows:v4.0.2-alpha Redis/redis-For-Windows:win-3.2.100 Redis/redis-For-Windows:win-3.0.504 Redis/redis-For-Windows:win-3.0.503 Redis/redis-For-Windows:win-2.8.2402 Redis/redis-For-Windows:win-3.0.502 Redis/redis-For-Windows:win-2.8.2401 Redis/redis-For-Windows:3.2.1 Redis/redis-For-Windows:win-3.2.0000-preview Redis/redis-For-Windows:3.2.0 Redis/redis-For-Windows:3.2.0-rc3 Redis/redis-For-Windows:3.2.0-rc2 Redis/redis-For-Windows:win-2.8.2400 Redis/redis-For-Windows:win-3.0.501 Redis/redis-For-Windows:3.2-rc1 Redis/redis-For-Windows:3.0.6 Redis/redis-For-Windows:2.8.24 Redis/redis-For-Windows:win-3.0.500 Redis/redis-For-Windows:win-3.0.500-rc2 Redis/redis-For-Windows:win-3.0.500-rc1 Redis/redis-For-Windows:win-2.8.2104 Redis/redis-For-Windows:3.0.5 Redis/redis-For-Windows:2.8.23 Redis/redis-For-Windows:win-3.0.300-beta1 Redis/redis-For-Windows:win-2.8.2103 Redis/redis-For-Windows:2.8.22 Redis/redis-For-Windows:3.0.4 Redis/redis-For-Windows:win-3.0.300-alpha3 Redis/redis-For-Windows:win-2.8.2102 Redis/redis-For-Windows:win-3.0.300-alpha2 Redis/redis-For-Windows:win-3.0.100-alpha1 Redis/redis-For-Windows:3.0.3 Redis/redis-For-Windows:win-2.8.2101 Redis/redis-For-Windows:win-2.8.21 Redis/redis-For-Windows:3.0.2 Redis/redis-For-Windows:2.8.21 Redis/redis-For-Windows:2.8.20 Redis/redis-For-Windows:3.0.1 Redis/redis-For-Windows:win-2.8.19.1 Redis/redis-For-Windows:3.0.0 Redis/redis-For-Windows:3.0.0-rc6 Redis/redis-For-Windows:3.0.0-rc5 Redis/redis-For-Windows:win-2.8.19-rc1 Redis/redis-For-Windows:win-2.8.19 Redis/redis-For-Windows:3.0.0-rc4 Redis/redis-For-Windows:win-2.8.19-pre2 Redis/redis-For-Windows:win-2.8.17.4 Redis/redis-For-Windows:3.0.0-rc3 Redis/redis-For-Windows:3.0.0-rc2 Redis/redis-For-Windows:win-2.8.17.3 Redis/redis-For-Windows:win-2.8.19-pre1 Redis/redis-For-Windows:win-2.8.17.2 Redis/redis-For-Windows:win-2.8.17.1 Redis/redis-For-Windows:2.8.19 Redis/redis-For-Windows:2.8.18 Redis/redis-For-Windows:3.0.0-rc1 Redis/redis-For-Windows:2.8.17 Redis/redis-For-Windows:2.8.16 Redis/redis-For-Windows:2.8.15 Redis/redis-For-Windows:win2.8.12 Redis/redis-For-Windows:2.8.14 Redis/redis-For-Windows:3.0.0-beta8 Redis/redis-For-Windows:2.8.13 Redis/redis-For-Windows:3.0.0-beta7 Redis/redis-For-Windows:win_2.8.9 Redis/redis-For-Windows:2.8.12 Redis/redis-For-Windows:2.8.11 Redis/redis-For-Windows:3.0.0-beta6 Redis/redis-For-Windows:2.8.10 Redis/redis-For-Windows:3.0.0-beta5 Redis/redis-For-Windows:3.0.0-beta4 Redis/redis-For-Windows:2.6.14_msopentech Redis/redis-For-Windows:2.8.4_msopentech Redis/redis-For-Windows:msopentech_2.8.4 Redis/redis-For-Windows:3.0.0-beta3 Redis/redis-For-Windows:2.8.9 Redis/redis-For-Windows:2.8.8 Redis/redis-For-Windows:3.0.0-beta2 Redis/redis-For-Windows:2.8.7 Redis/redis-For-Windows:2.8.6 Redis/redis-For-Windows:3.0.0-beta1 Redis/redis-For-Windows:2.8.5 Redis/redis-For-Windows:2.8.4 Redis/redis-For-Windows:2.8.3 Redis/redis-For-Windows:2.8.2 Redis/redis-For-Windows:2.8.1 Redis/redis-For-Windows:2.8.0 Redis/redis-For-Windows:2.8.0-rc6 Redis/redis-For-Windows:2.8.0-rc5 Redis/redis-For-Windows:2.8.0-rc4 Redis/redis-For-Windows:2.8.0-rc3 Redis/redis-For-Windows:2.8.0-rc2 Redis/redis-For-Windows:2.8.0-rc1 Redis/redis-For-Windows:2.6.14 Redis/redis-For-Windows:Win2.6.8 Redis/redis-For-Windows:2.6.13 Redis/redis-For-Windows:2.6.12 Redis/redis-For-Windows:2.6.11 Redis/redis-For-Windows:2.6.10 Redis/redis-For-Windows:2.6.9 Redis/redis-For-Windows:2.6.8 Redis/redis-For-Windows:2.6.7 Redis/redis-For-Windows:2.6.6 Redis/redis-For-Windows:2.6.5 Redis/redis-For-Windows:2.6.4 Redis/redis-For-Windows:2.6.3 Redis/redis-For-Windows:2.6.2 Redis/redis-For-Windows:2.6.1 Redis/redis-For-Windows:2.6.0 Redis/redis-For-Windows:2.6.0-rc8 Redis/redis-For-Windows:2.6.0-rc7 Redis/redis-For-Windows:2.6.0-rc6 Redis/redis-For-Windows:2.6.0-rc5 Redis/redis-For-Windows:2.6.0-rc4 Redis/redis-For-Windows:2.6.0-rc3 Redis/redis-For-Windows:2.6.0-rc2 Redis/redis-For-Windows:2.6.0-rc1 Redis/redis-For-Windows:2.4.11 Redis/redis-For-Windows:2.4.10 Redis/redis-For-Windows:2.4.9 Redis/redis-For-Windows:2.4.8 Redis/redis-For-Windows:2.4.6_win32 Redis/redis-For-Windows:2.4.7 Redis/redis-For-Windows:2.4.6 Redis/redis-For-Windows:2.4.5 Redis/redis-For-Windows:2.4.4 Redis/redis-For-Windows:2.4.3 Redis/redis-For-Windows:2.4.2 Redis/redis-For-Windows:2.4.1 Redis/redis-For-Windows:2.4.0 Redis/redis-For-Windows:2.4.0-rc8 Redis/redis-For-Windows:2.4.0-rc7 Redis/redis-For-Windows:2.4.0-rc6 Redis/redis-For-Windows:2.4.0-rc5 Redis/redis-For-Windows:2.4.0-rc4 Redis/redis-For-Windows:2.4.0-rc3 Redis/redis-For-Windows:with-deprecated-diskstore Redis/redis-For-Windows:2.4.0-rc2 Redis/redis-For-Windows:2.4.0-rc1 Redis/redis-For-Windows:2.2.4 Redis/redis-For-Windows:2.2.3 Redis/redis-For-Windows:3.0-alpha0 Redis/redis-For-Windows:2.2.2 Redis/redis-For-Windows:2.2.1 Redis/redis-For-Windows:2.2.0 Redis/redis-For-Windows:2.2.0-rc4 Redis/redis-For-Windows:2.3-alpha0 Redis/redis-For-Windows:2.2.0-rc3 Redis/redis-For-Windows:2.2.0-rc2 Redis/redis-For-Windows:2.2.0-rc1 Redis/redis-For-Windows:2.2-alpha6 Redis/redis-For-Windows:2.2-alpha5 Redis/redis-For-Windows:2.2-alpha4 Redis/redis-For-Windows:2.2-alpha3 Redis/redis-For-Windows:2.2-alpha2 Redis/redis-For-Windows:2.2-alpha1 Redis/redis-For-Windows:2.2-alpha0 Redis/redis-For-Windows:v2.1.1-watch Redis/redis-For-Windows:v2.0.0-rc1 Redis/redis-For-Windows:v1.3.12 Redis/redis-For-Windows:v1.3.11 Redis/redis-For-Windows:v1.3.10 Redis/redis-For-Windows:v1.3.9 Redis/redis-For-Windows:v1.3.8 Redis/redis-For-Windows:v1.3.7 Redis/redis-For-Windows:1.3.6 Redis/redis-For-Windows:vm-playpen
1750 Commits
3.0.1 .. 2.8.22
Author SHA1 Message Date
antirez 54b368bb81 Redis 2.8.22. 2015-09-08 10:03:48 +02:00
antirez 35507e7871 Sentinel: command arity check added where missing. 2015-09-08 09:33:48 +02:00
Rogerio Goncalves fa81cc2624 Check args before run ckquorum. Fix issue #2635 2015-09-08 09:28:30 +02:00
MOON_CLJ 06894b8fe4 pfcount support multi keys 2015-09-08 09:22:11 +02:00
antirez 7bac7d37cb Fix merge issues in 490847c. 2015-09-07 17:29:43 +02:00
antirez 9155cdc3d6 Undo slaves state change on failed rdbSaveToSlavesSockets(). 
As Oran Agra suggested, in startBgsaveForReplication() when the BGSAVE
attempt returns an error, we scan the list of slaves in order to remove
them since there is no way to serve them currently.

However we check for the replication state BGSAVE_START, which was
modified by rdbSaveToSlaveSockets() before forking(). So when fork fails
the state of slaves remain BGSAVE_END and no cleanup is performed.

This commit fixes the problem by making rdbSaveToSlavesSockets() able to
undo the state change on fork failure.
 2015-09-07 16:21:33 +02:00
antirez 8c8a7cded9 Sentinel: fix bug in config rewriting during failover 
We have a check to rewrite the config properly when a failover is in
progress, in order to add the current (already failed over) master as
slave, and don't include in the slave list the promoted slave itself.

However there was an issue, the variable with the right address was
computed but never used when the code was modified, and no tests are
available for this feature for two reasons:

1. The Sentinel unit test currently does not test Sentinel ability to
persist its state at all.
2. It is a very hard to trigger state since it lasts for little time in
the context of the testing framework.

However this feature should be covered in the test in some way.

The bug was found by @badboy using the clang static analyzer.

Effects of the bug on safety of Sentinel
===

This bug results in severe issues in the following case:

1. A Sentinel is elected leader.
2. During the failover, it persists a wrong config with a known-slave
entry listing the master address.
3. The Sentinel crashes and restarts, reading invalid configuration from
disk.
4. It sees that the slave now does not obey the logical configuration
(should replicate from the current master), so it sends a SLAVEOF
command to the master (since the slave master is the same) creating a
replication loop (attempt to replicate from itself) which Redis is
currently unable to detect.
5. This means that the master is no longer available because of the bug.

However the lack of availability should be only transient (at least
in my tests, but other states could be possible where the problem
is not recovered automatically) because:

6. Sentinels treat masters reporting to be slaves as failing.
7. A new failover is triggered, and a slave is promoted to master.

Bug lifetime
===

The bug is there forever. Commit 16237d78 actually tried to fix the bug
but in the wrong way (the computed variable was never used! My fault).
So this bug is there basically since the start of Sentinel.

Since the bug is hard to trigger, I remember little reports matching
this condition, but I remember at least a few. Also in automated tests
where instances were stopped and restarted multiple times automatically
I remember hitting this issue, however I was not able to reproduce nor
to determine with the information I had at the time what was causing the
issue.
 2015-09-07 15:53:08 +02:00
antirez c47854615d Sentinel: clarify effect of resetting failover_start_time. 2015-09-07 15:53:08 +02:00
ubuntu 3ff2d65ffa SCAN iter parsing changed from atoi to chartoull 2015-09-07 13:25:38 +02:00
antirez 194b7e2186 Force slaves to resync after unsuccessful PSYNC. 
Using chained replication where C is slave of B which is in turn slave of
A, if B reconnects the replication link with A but discovers it is no
longer possible to PSYNC, slaves of B must be disconnected and PSYNC
not allowed, since the new B dataset may be completely different after
the synchronization with the master.

Note that there are varius semantical differences in the way this is
handled now compared to the past. In the past the semantics was:

1. When a slave lost connection with its master, disconnected the chained
slaves ASAP. Which is not needed since after a successful PSYNC with the
master, the slaves can continue and don't need to resync in turn.

2. However after a failed PSYNC the replication backlog was not reset, so a
slave was able to PSYNC successfully even if the instance did a full
sync with its master, containing now an entirely different data set.

Now instead chained slaves are not disconnected when the slave lose the
connection with its master, but only when it is forced to full SYNC with
its master. This means that if the slave having chained slaves does a
successful PSYNC all its slaves can continue without troubles.

See issue #2694 for more details.
 2015-08-21 15:32:04 +02:00
antirez 730f7c5f5e replicationHandleMasterDisconnection() belongs to replication.c. 2015-08-21 15:32:01 +02:00
antirez 12d2a89410 flushSlavesOutputBuffers(): details clarified via comments. 
Talking with @oranagra we had to reason a little bit to understand if
this function could ever flush the output buffers of the wrong slaves,
having online state but actually not being ready to receive writes
before the first ACK is received from them (this happens with diskless
replication).

Next time we'll just read this comment.
 2015-08-21 15:31:59 +02:00
antirez 7a02677097 Log client details on SLAVEOF command having an effect. 2015-08-21 15:31:11 +02:00
antirez 5630eeb12b startBgsaveForReplication(): handle waiting slaves state change. 
Before this commit, after triggering a BGSAVE it was up to the caller of
startBgsavForReplication() to handle slaves in WAIT_BGSAVE_START in
order to update them accordingly. However when the replication target is
the socket, this is not possible since the process of updating the
slaves and sending the FULLRESYNC reply must be coupled with the process
of starting an RDB save (the reason is, we need to send the FULLSYNC
command and spawn a child that will start to send RDB data to the slaves
ASAP).

This commit moves the responsibility of handling slaves in
WAIT_BGSAVE_START to startBgsavForReplication() so that for both
diskless and disk-based replication we have the same chain of
responsiblity. In order accomodate such change, the syncCommand() also
needs to put the client in the slave list ASAP (just after the initial
checks) and not at the end, so that startBgsavForReplication() can find
the new slave alrady in the list.

Another related change is what happens if the BGSAVE fails because of
fork() or other errors: we now remove the slave from the list of slaves
and send an error, scheduling the slave connection to be terminated.

As a side effect of this change the following errors found by
Oran Agra are fixed (thanks!):

1. rdbSaveToSlavesSockets() on failed fork will get the slaves cleaned
up, otherwise they remain in a wrong state forever since we setup them
for full resync before actually trying to fork.

2. updateSlavesWaitingBgsave() with replication target set as "socket"
was broken since the function changed the slaves state from
WAIT_BGSAVE_START to WAIT_BGSAVE_END via
replicationSetupSlaveForFullResync(), so later rdbSaveToSlavesSockets()
will not find any slave in the right state (WAIT_BGSAVE_START) to feed.
 2015-08-21 11:55:30 +02:00
antirez a7e9b38a48 slaveTryPartialResynchronization and syncWithMaster: better synergy. 
It is simpler if removing the read event handler from the FD is up to
slaveTryPartialResynchronization, after all it is only called in the
context of syncWithMaster.

This commit also makes sure that on error all the event handlers are
removed from the socket before closing it.
 2015-08-07 12:19:21 +02:00
antirez d4e4bd039a syncWithMaster(): non blocking state machine. 2015-08-07 12:18:29 +02:00
antirez c9df63c103 Fixed issues introduced during last merge. 2015-08-06 17:33:04 +02:00
antirez ce3a2d085b startBgsaveForReplication(): log what you really do. 2015-08-06 12:36:13 +02:00
antirez 6974e69f35 Replication: add REPLCONF CAPA EOF support. 
Add the concept of slaves capabilities to Redis, the slave now presents
to the Redis master with a set of capabilities in the form:

    REPLCONF capa SOMECAPA capa OTHERCAPA ...

This has the effect of setting slave->slave_capa with the corresponding
SLAVE_CAPA macros that the master can test later to understand if it
the slave will understand certain formats and protocols of the
replication process. This makes it much simpler to introduce new
replication capabilities in the future in a way that don't break old
slaves or masters.

This patch was designed and implemented together with Oran Agra
(@oranagra).
 2015-08-06 12:36:13 +02:00
antirez be56e4cf33 Fix synchronous readline "\n" handling. 
Our function to read a line with a timeout handles newlines as requests
to refresh the timeout, however the code kept subtracting the buffer
size left every time a newline was received, for a bug in the loop
logic. Fixed by this commit.
 2015-08-05 17:00:04 +02:00
antirez a67d67b561 Fix replication slave pings period. 
For PINGs we use the period configured by the user, but for the newlines
of slaves waiting for an RDB to be created (including slaves waiting for
the FULLRESYNC reply) we need to ping with frequency of 1 second, since
the timeout is fixed and needs to be refreshed.
 2015-08-05 17:00:04 +02:00
antirez 9a5560f4c3 Fix RDB encoding test for new csvdump format. 2015-08-05 14:05:34 +02:00
antirez 6da198cdf1 Remove slave state change handled by replicationSetupSlaveForFullResync(). 2015-08-05 14:00:25 +02:00
antirez 39994c2493 Make sure we re-emit SELECT after each new slave full sync setup. 
In previous commits we moved the FULLRESYNC to the moment we start the
BGSAVE, so that the offset we provide is the right one. However this
also means that we need to re-emit the SELECT statement every time a new
slave starts to accumulate the changes.

To obtian this effect in a more clean way, the function that sends the
FULLRESYNC reply was overloaded with a more important role of also doing
this and chanigng the slave state. So it was renamed to
replicationSetupSlaveForFullResync() to better reflect what it does now.
 2015-08-05 14:00:22 +02:00
antirez a89326f0f6 Test: csvdump now scans all DBs. 2015-08-05 14:00:18 +02:00
antirez b2ff48ef19 Don't send SELECT to slaves in WAIT_BGSAVE_START state. 2015-08-05 14:00:15 +02:00
antirez 7967f1bca6 PSYNC test: also test the vanilla SYNC. 2015-08-05 14:00:11 +02:00
antirez e684e7266c syncCommand() comments improved. 2015-08-05 14:00:08 +02:00
antirez 4b010572cd PSYNC initial offset fix. 
This commit attempts to fix a bug involving PSYNC and diskless
replication (currently experimental) found by Yuval Inbar from Redis Labs
and that was later found to have even more far reaching effects (the bug also
exists when diskstore is off).

The gist of the bug is that, a Redis master replies with +FULLRESYNC to
a PSYNC attempt that fails and requires a full resynchronization.
However, the baseline offset sent along with FULLRESYNC was always the
current master replication offset. This is not ok, because there are
many reasosn that may delay the RDB file creation. And... guess what,
the master offset we communicate must be the one of the time the RDB
was created. So for example:

1) When the BGSAVE for replication is delayed since there is one
   already but is not good for replication.
2) When the BGSAVE is not needed as we attach one currently ongoing.
3) When because of diskless replication the BGSAVE is delayed.

In all the above cases the PSYNC reply is wrong and the slave may
reconnect later claiming to need a wrong offset: this may cause
data curruption later.
 2015-08-05 14:00:04 +02:00
antirez dc4d24440f Test PSYNC with diskless replication. 
Thanks to Oran Agra from Redis Labs for providing this patch.
 2015-08-05 14:00:00 +02:00
Salvatore Sanfilippo d3688e8b68 Merge pull request #2666 from Abioy/2.8 
bugfix: errno might change before logging
 2015-07-17 10:44:18 +02:00
Salvatore Sanfilippo 88a38fe3d3 Merge pull request #2678 from Kiemes/fix-config-resetstat 
Fix: aof_delayed_fsync is not reset, fixes #2677
 2015-07-17 10:39:02 +02:00
Tom Kiemes cb98ae12f6 Fix: aof_delayed_fsync is not reset 
aof_delayed_fsync was not set to 0 when calling CONFIG RESETSTAT
 2015-07-16 11:57:07 +02:00
Yongyue Sun 9c7f98521e bugfix: errno might change before logging 
Signed-off-by: Yongyue Sun <abioy.sun@gmail.com>
 2015-07-13 14:59:05 +08:00
antirez fbb9d619f7 Fix 2.8.21 release notes to give full credits. 2015-06-04 12:00:21 +02:00
antirez 7f8b865a65 Redis 2.8.21 2015-06-04 11:32:24 +02:00
Ben Murphy 700b863f13 hide access to debug table 2015-06-03 13:36:02 +02:00
Ben Murphy 5a1b22ad7f disable loading lua bytecode 2015-06-03 13:36:02 +02:00
antirez 1eeb9bd714 Scripting: Lua cmsgpack lib updated to include str8 support 2015-06-03 08:45:56 +02:00
therealbill 22ee2f9cd5 adding a sentinel command: "flushconfig" 
This new command triggers a config flush to save the in-memory config to
disk. This is useful for cases of a configuration management system or a
package manager wiping out your sentinel config while the process is
still running - and has not yet been restarted. It can also be useful
for scripting a backup and migrate or clone of a running sentinel.
 2015-05-25 12:06:57 +02:00
antirez cdedad2322 Sentinel: CKQUORUM tests 2015-05-19 12:27:26 +02:00
antirez 76412ec9c0 Sentinel: SENTINEL CKQUORUM command 
A way for monitoring systems to check that Sentinel is technically able
to reach the quorum and failover, using the currently visible Sentinels.
 2015-05-19 12:27:26 +02:00
antirez f42fcff6d2 Rewrite smoveCommand test with ternary operator 2015-05-15 17:39:39 +02:00
Glenn Nethercutt 19382c8be6 uphold the smove contract to return 0 when the element is not a member of the source set, even if source=dest 2015-05-15 17:39:39 +02:00
Jungtaek Lim bea224308b protocol error log should be seen debug/verbose level 2015-05-15 17:06:45 +02:00
antirez 22979836e5 Fix 2.8.20 changelog typo 2015-05-05 11:50:30 +02:00
antirez 11934a9f32 Redis 2.8.20 2015-05-05 11:14:33 +02:00
FuGangqiang 8db3969971 sdsfree x and y 2015-05-04 13:03:06 +02:00
FuGangqiang 98756d4c0d fix doc example 2015-05-04 13:03:06 +02:00
FuGangqiang 7316fda33a fix typo 2015-05-04 13:03:06 +02:00
Itamar Haber a5bada19be update copyright year 2015-05-04 12:56:52 +02:00
therealbill f0ab4fd6fd Making sentinel flush config on +slave 
Originally, only the +slave event which occurs when a slave is
reconfigured during sentinelResetMasterAndChangeAddress triggers a flush
of the config to disk.  However, newly discovered slaves don't
apparently trigger this flush but do trigger the +slave event issuance.

So if you start up a sentinel, add a master, then add a slave to the
master (as a way to reproduce it) you'll see the +slave event issued,
but the sentinel config won't be updated with the known-slave entry.

This change makes sentinel do the flush of the config if a new slave is
deteted in sentinelRefreshInstanceInfo.
 2015-05-04 12:55:30 +02:00
antirez e4c54498db Sentinel: remove useless sentinelFlushConfig() call 
To rewrite the config in the loop that adds slaves back after a master
reset, in order to handle switching to another master, is useless: it
just adds latency since there is an fsync call in the inner loop,
without providing any additional guarantee, but the contrary, since if
after the first loop iteration the server crashes we end with just a
single slave entry losing all the other informations.

It is wiser to rewrite the config at the end when the full new
state is configured.
 2015-05-04 12:55:30 +02:00
Yossi Gottlieb 81a8ebd7f0 Fix Redis server crash when Lua command exceeds client output buffer 
limit.
 2015-05-04 12:20:28 +02:00
clark.kang c4d4c1ed12 fix sentinel memory leak 2015-05-04 12:18:13 +02:00
antirez 081a0c943f Fix Sentinel memory leak (hiredis bug) 
This fixes issue #2535, that was actually an hiredis library bug (I
submitted an issue and fix to the redis/hiredis repo as well).

When an asynchronous hiredis connection subscribes to a Pub/Sub channel
and gets an error, and in other related conditions, the function
redisProcessCallbacks() enters a code path where the link is
disconnected, however the function returns before freeing the allocated
reply object. This causes a memory leak. The memory leak was trivial to
trigger in Redis Sentinel, which uses hiredis, every time we tried to
subscribe to an instance that required a password, in case the Sentinel
was configured either with the wrong password or without password at
all. In this case, the -AUTH error caused the leaking code path to be
executed.

It was verified with Valgrind that after this change the leak no longer
happens in Sentinel with a misconfigured authentication password.
 2015-04-28 22:15:12 +02:00
antirez d593ae8ab6 sha1.c: use standard uint32_t. 2015-04-27 12:08:03 +02:00
Oran Agra c72253ec54 fixes to diskless replication. 
master was closing the connection if the RDB transfer took long time.
and also sent PINGs to the slave before it got the initial ACK, in which case the slave wouldn't be able to find the EOF marker.
 2015-04-01 15:21:08 +02:00
antirez 83eb5e7f54 Test: regression for issue #2473. 2015-03-27 12:11:46 +01:00
antirez 96aa610666 Replication: disconnect blocked clients when switching to slave role. 
Bug as old as Redis and blocking operations. It's hard to trigger since
only happens on instance role switch, but the results are quite bad
since an inconsistency between master and slave is created.

How to trigger the bug is a good description of the bug itself.

1. Client does "BLPOP mylist 0" in master.
2. Master is turned into slave, that replicates from New-Master.
3. Client does "LPUSH mylist foo" in New-Master.
4. New-Master propagates write to slave.
5. Slave receives the LPUSH, the blocked client get served.

Now Master "mylist" key has "foo", Slave "mylist" key is empty.

Highlights:

* At step "2" above, the client remains attached, basically escaping any
  check performed during command dispatch: read only slave, in that case.
* At step "5" the slave (that was the master), serves the blocked client
  consuming a list element, which is not consumed on the master side.

This scenario is technically likely to happen during failovers, however
since Redis Sentinel already disconnects clients using the CLIENT
command when changing the role of the instance, the bug is avoided in
Sentinel deployments.

Closes #2473.
 2015-03-24 16:14:49 +01:00
antirez 1f5e7b8863 Config: activerehashing option support in CONFIG SET. 2015-03-08 15:34:00 +01:00
antirez f8f40afd5d Fix iterator for issue #2438. 
Itereator misuse due to analyzeLatencyForEvent() accessing the
dictionary during the iteration, without the iterator being
reclared as safe.
 2015-03-04 11:49:53 -08:00
Charles Hooper 1f4074e957 override histfile from env - fixes #831 and copies #833 2015-02-10 14:55:17 +01:00
antirez 9e83d2d26f redis-cli --lru-test implemented (cache workload simulator). 2015-02-10 14:55:17 +01:00
antirez ccbec34fad redis-cli: interactive reconnection for latency modes. 
--stat mode already used to reconnect automatically if the server is no
longer available. This is useful since this is an interactive mode used
for debugging, however the same applies to --latency and --latency-dist
modes, so now both use the reconnecting command execution as well.

The reconnection code was modified to use basic VT100 escape sequences
in order to play better with different kinds of output on the screen
when the reconnection happens, and to hide the reconnection attempt
output when finally the reconnection happens.
 2015-02-10 14:55:17 +01:00
antirez d29eb68f20 redis-cli --latecy-dist reverted to gray scale. 
So far not able to find a color palette within the 256 colors which is
not confusing. However I believe it is a possible task, so will try
better later.
 2015-02-10 14:55:17 +01:00
antirez 657f719760 redis-cli --latency-dist now uses a color palette. 
Still not happy with the result but low grays are hard to see in certain
monitors with a non perfect gamma.
 2015-02-10 14:55:17 +01:00
antirez ff13fa787d redis-cli latency dist: add new top HL. 2015-02-10 14:55:17 +01:00
antirez f6f3cb9458 Add missing latency-dest legend symbol. 2015-02-10 14:55:17 +01:00
antirez 9d9be0e09d Initial implementation of redis-cli --latency-dist. 2015-02-10 14:55:17 +01:00
antirez d4047f7261 Faster memory efficiency test. 
This test on Linux was extremely slow, since in Tcl we can't enable
easily tcp-nodelay, so the busy loop used to take *a lot* with bigger
writes. Fixed using pipelining.
 2015-02-10 14:48:22 +01:00
mattcollier 61290d819d Update redis-cli.c 
Code was adding '\n'  (line 521) to the end of NIL values exlusively making csv output inconsistent.  Removed '\n'
 2015-02-02 22:50:53 +01:00
antirez 18dc769104 luaRedisGenericCommand(): log error at WARNING level when re-entered. 
Rationale is that when re-entering, it is likely due to Lua debugging
hooks. Returning an error will be ignored in most cases, going totally
unnoticed. With the log at least we leave a trace.

Related to issue #2302.
 2015-01-22 10:36:23 +01:00
antirez 1dea8b5a29 luaRedisGenericCommand() recursion: just return an error. 
Instead of calling redisPanic() to abort the server.

Related to issue #2302.
 2015-01-22 10:36:19 +01:00
antirez a0ba7b3f00 Panic on recursive calls to luaRedisGenericCommand(). 
Related to issue #2302.
 2015-01-22 10:36:15 +01:00
antirez e7aa5fb5ef AOF rewrite: set iterator var to NULL when freed. 
The cleanup code expects that if 'di' is not NULL, it is a valid
iterator that should be freed.

The result of this bug was a crash of the AOF rewriting process if an
error occurred after the DBs data are written and the iterator is no
longer valid.
 2015-01-21 16:42:56 +01:00
antirez 8a41dae49f Add "-lrt" in Makefile for Solaris. 
This fix is from @NanXiao, however I was not able to retain authorship
because the Pull Request original repository was removed.
 2015-01-09 11:54:52 +01:00
antirez 50cdedd127 Check for __sun macro in solarisfixes.h, not in includers. 2015-01-09 11:22:35 +01:00
antirez 77608ae5ac README section about make distclean reworded / extended. 2015-01-08 16:35:53 +01:00
Jungtaek Lim 5165fb98af Explain make distclean which seems not well known 2015-01-08 16:35:49 +01:00
antirez 22a0fe8de6 INFO loading stats: three fixes. 
1. Server unxtime may remain not updated while loading AOF, so ETA is
not updated correctly.

2. Number of processed byte was not initialized.

3. Possible division by zero condition (likely cause of issue #1932).
 2014-12-23 14:52:57 +01:00
Alon Diamant 585884da45 Fixed memory leaks in rdbSaveToSlavesSockets() 2014-12-22 11:01:21 +01:00
antirez ca3dd5e5ed Fix adjustOpenFilesLimit() logging to match real state. 
Fixes issue #2225.
 2014-12-19 10:05:09 +01:00
Rhommel Lamas 6f0a7353a6 Add redis-sentinel symlink to a relative path 2014-12-17 12:27:44 +01:00
antirez 902b877dc0 Redis 2.8.19. 2014-12-16 09:18:20 +01:00
antirez 02d465c8fe Don't log admin commands in MONITOR. 
Otherwise there are security risks, especially when providing Redis as a
service, the user may "sniff" for admin commands renamed to an
unguessable string via rename-command in redis.conf.
 2014-12-13 09:00:03 +01:00
antirez 4d8f4262a4 List of commands flagged as admin commands modified. 
The old list did not made much sense... and the flag is currently not
used at all, so no side effects.
 2014-12-13 08:59:43 +01:00
Salvatore Sanfilippo d4fc65309d Merge pull request #2092 from rhoml/rhoml-install-sentinel 
Adds a symlink for redis-sentinel when Make install
 2014-12-12 22:58:56 +01:00
antirez e47e460f1f Lua cmsgpack lib updated to latest version. 
It fixes a bad bug that crashes the server in certain conditions
as shown in issue #2210.
 2014-12-12 22:57:09 +01:00
Rhommel Lamas 5509c141f8 Add symlink to redis-sentinel during make install 2014-12-12 20:00:08 +01:00
antirez 7de1ef773e SORT: Don't sort Set elements if not needed. 
Related to #2094.
 2014-12-11 16:01:21 +01:00
Matt Stancliff e945a546af Fix zero-ordering SORT when called against lists 
People mostly use SORT against lists, but our prior
behavior was pretending lists were an unordered bag
requiring a forced-sort when no sort was requested.

We can just use the native list ordering to ensure
consistency across replicaion and scripting calls.

Closes #2079
Closes #545 (again)
 2014-12-11 16:01:09 +01:00
Ben Dowling d81c38316b Update redis_init_script.tpl 
status command currently reports success when redis has crashed and the pid file still exists. Changing to check the actual process is running.
 2014-12-11 15:21:53 +01:00
zhanghailei dba57ea910 FIXED redis-benchmark's idle mode.With idle mode shouldn't create write event 2014-12-11 15:21:53 +01:00
Sun He 888ea17548 zipmap.c: update comments above 2014-12-09 16:01:13 +01:00
Deepak Verma 86ebc139c2 replaced // comments #2150 2014-12-09 15:26:26 +01:00
azure provisioned user 3d73f088e1 redis-benchmark AUTH command to be discarded after the first send #2150 2014-12-09 15:26:26 +01:00
Sun He 76d53a6770 sds.c: Correct two spelling mistakes in comments 2014-12-09 15:14:14 +01:00
Sun He 4848cf9335 sds.c/sdscatvprintf: set va_end to finish va_list cpy 2014-12-09 15:14:14 +01:00
Sun He d2f584fa53 sds.c: Correct some comments 2014-12-09 15:14:13 +01:00
Serghei Iakovlev 2ed3f09cc3 Update whatisdoing.sh 
Improved getting pid
 2014-12-09 13:01:31 +01:00
antirez 77b997d429 Include stropts only if __sun is defined. 2014-12-09 12:59:43 +01:00
Jan-Erik Rediger d409371193 Fix implicit declaration of ioctl on Solaris 2014-12-09 12:57:59 +01:00
Johan Bergström 23b96c02a5 Silence _BSD_SOURCE warnings in glibc 2.20 and forward 
See https://sourceware.org/glibc/wiki/Release/2.20#Packaging_Changes
 2014-12-09 12:24:30 +01:00
antirez a47a042ea6 Mark whatisdoing.sh as deprecated in top-comment. 2014-12-09 12:18:47 +01:00
Serghei Iakovlev b5737d2d19 getting pid fixes 
```sh
$ ~ pidof redis-server
# nothing

$ ~ ps aux | grep [r]edis
redis      593  0.0  0.0  36900  5564 ?        Ssl  Dec02   1:37 /usr/bin/redis-server 127.0.0.1:6379
klay     15927  0.0  0.0  16772  6068 pts/6    S+   13:58   0:00 redis-cli

$ ~ uname -a
Linux edge 3.17.4-1-ARCH #1 SMP PREEMPT Fri Nov 21 21:14:42 CET 2014 x86_64 GNU/Linux
```
 2014-12-09 12:09:31 +01:00
Sun He a598e0894b sparkline.c: AddSample skip Empty label 2014-12-09 11:28:53 +01:00
Sun He 7d480abab4 sparkline.c: mov label-ini into the AddSample Function 2014-12-09 11:28:53 +01:00
Jan-Erik Rediger 2f3c860979 Only ignore sigpipe in interactive mode 
This allows shell pipes to correctly end redis-cli.

Ref #2066
 2014-12-09 09:01:10 +01:00
antirez 0c211a1953 Simplify lua_cmsgpack macro and fix build on old Linux distros. 
Thanks to @badboy for the help in checking the build after the fix.
 2014-12-05 10:52:56 +01:00
antirez d6ada4d1a8 Redis 2.8.18. 2014-12-04 11:36:52 +01:00
antirez 7c671c09a1 Check that tcp-backlog is matched by /proc/sys/net/core/somaxconn. 2014-12-04 11:07:55 +01:00
antirez d56ef62983 Network bandwidth tracking + refactoring. 
Track bandwidth used by clients and replication (but diskless
replication is not tracked since the actual transfer happens in the
child process).

This includes a refactoring that makes tracking new instantaneous
metrics simpler.
 2014-12-03 12:17:18 +01:00
antirez f60c8c0999 Don't show the ASCII logo if syslog is enabled. 
Closes issue #1935.
 2014-12-03 10:52:09 +01:00
antirez 04231cb7d7 Regression test for issue #2175. 2014-12-03 10:41:06 +01:00
antirez 6e0277769d Handle infinite explicitly in createStringObjectFromLongLong(). 2014-12-03 10:41:06 +01:00
antirez 42695fdfe6 Use exp format and more precision output for ZSCAN. 
Ref: issue #2175
 2014-12-03 10:41:06 +01:00
antirez 164ad2ead1 Over 80 chars comment trimmed in pfcountCommand(). 2014-12-02 17:03:15 +01:00
antirez 0e40124d0f Mark PFCOUNT as read-only, even if not true. 
PFCOUNT is technically speaking a write command, since the cached value
of the HLL is exposed in the data structure (design error, mea culpa), and
can be modified by PFCOUNT.

However if we flag PFCOUNT as "w", read only slaves can't execute the
command, which is a problem since there are environments where slaves
are used to scale PFCOUNT reads.

Nor it is possible to just prevent PFCOUNT to modify the data structure
in slaves, since without the cache we lose too much efficiency.

So while this commit allows slaves to create a temporary inconsistency
(the strings representing the HLLs in the master and slave can be
different in certain moments) it is actually harmless.

In the long run this should be probably fixed by turning the HLL into a
more opaque representation, for example by storing the cached value in
the part of the string which is not exposed (this should be possible
with SDS strings).
 2014-12-02 17:01:55 +01:00
antirez d0cc96f053 Mark diskless replication as experimental in redis.conf. 2014-12-02 14:13:44 +01:00
antirez a81043959a Test: wait for actual startup in start_server. 
start_server now uses return value from Tcl exec to get the server pid,
however this introduces errors that depend from timing: a lot of the
testing code base assumed the server to be actually up and running when
server_start returns.

So the old code that waits to see the pid in the log file was restored.
 2014-11-28 12:57:40 +01:00
antirez f0c00a14ed Test: try to cleanup still running Redis instances on exit. 
It's hard to run the Redis test continuously if it leaks processes on
exceptions / errors.
 2014-11-28 12:57:30 +01:00
antirez 3ac62dff23 Test framework: exit on timeout with report. 
There was no sane way to detect tests that may never end because of
Redis bugs or tests bugs.
 2014-11-28 12:54:17 +01:00
antirez 60c09dcf09 redis-benchmark: default num of requests is now 100000. 
10000 completes in a too short time and may easily provide unreliable
figures because of tiny duration.
 2014-11-28 10:38:58 +01:00
Matthias Petschick e689e93663 fix benchmark memleak in loop mode 2014-11-28 10:38:54 +01:00
antirez 01854f87e8 Document redis-cli --stat in --help output. 2014-11-25 18:25:58 +01:00
Matt Stancliff e305df099a Fix lua-cmsgpack 64 bit integer on 32 bit platform 
This syncs lua-cmsgpack with the mattsta/lua-cmsgpack upstream.

Fixes #2161
 2014-11-25 15:56:57 +01:00
antirez 9854d03fad Avoid valgrind memory leak false positive in processInlineBuffer(). 
zmalloc(0) cauesd to actually trigger a non-zero allocation since with
standard libc malloc we have our own zmalloc header for memory tracking,
but at the same time the returned pointer is at the end of the block and
not in the middle. This triggers a false positive when testing with
valgrind.

When the inline protocol args count is 0, we now avoid reallocating
c->argv, preventing the issue to happen.
 2014-11-25 14:51:04 +01:00
antirez 5ddf56f9b1 Attempt to prevent false positives in replication test. 2014-11-24 11:55:16 +01:00
antirez b19e745780 scripting.tcl tests order fixed to match unstable branch. 2014-11-14 17:28:58 +01:00
antirez ce96069676 lua_cjson.c Lua includes: angled -> quoted. 2014-11-14 17:28:54 +01:00
antirez ead8e31b28 Fix non-linux builds error introduced with THP checks. 2014-11-14 17:28:49 +01:00
Matt Stancliff 5d8c88fdc1 Lua: add cmsgpack scripting tests 
Basically: test to make sure we can load cmsgpack
and do some sanity checks to make sure pack/unpack works
properly.  We also have a bonus test for circular encoding
and decoding because I was curious how it worked.
 2014-11-14 17:28:45 +01:00
Matt Stancliff 381863be5c Lua: upgrade cmsgpack to 0.4.0 
Main reasons for upgrade:
  - Remove a warning when building Redis
  - Add multi pack/unpack
  - Improve memory usage and use Lua allocator properly
  - Fix some edge case encoding/decoding bugs
 2014-11-14 17:28:40 +01:00
Matt Stancliff a7b58f1569 Lua: remove new warning added by cjson header 
clang doesn't like "extern inline" when no definition
is given right away.
 2014-11-14 17:28:35 +01:00
Matt Stancliff 56e1861398 Lua: Use Redis solaris compatability for cjson too 
cjson calls isinf, but some Solaris versions don't have isinf
even with the attempted fix we have in deps/Makefile.

We can harmlessly include the Redis solarisfixes.h header to
give cjson proper isinf.

Note: cjson has a compile-time setting for using their own defined
isinf, but the Redis definition in solarisfixes.h is more complete.

Fixes antirez#1620
 2014-11-14 17:28:32 +01:00
Matt Stancliff e2983dd03f Lua: Upgrade cjson to 2.1.0 (2012-03-01) 
The new cjson has some improvements over our current version including
increased platform compatability, a new resource limit to restrict
decode depth, and better invalid number handling.

One minor change was required to deps/Makefile because this version
of cjson doesn't export itself globally, so we added a quick little
define of -DENABLE_CJSON_GLOBAL.

cjson now has an optional higher performing float parsing interface,
but we are not including it (g_fmt.c, dtoa.c) because it requires
endianness declaration during compile time.

This commit is exactly lua_cjson.c from 2.1.0 with one minor
change of altering the two Lua includes for local search
instead of system-wide importing.
 2014-11-14 17:28:27 +01:00
Matt Stancliff 2828bbd511 Lua: add cjson scripting test 
Two simple decode tests added mainly to check that
the 'cjson' global gets registered and is usable.
 2014-11-14 17:28:23 +01:00
Matt Stancliff 1e501d9fd6 Lua: Add bitop 
A few people have written custom C commands because bit
manipulation isn't exposed through Lua.  Let's give
them Mike Pall's bitop.

This adds bitop 1.0.2 (2012-05-08) from http://bitop.luajit.org/

bitop is imported as "bit" into the global namespace.

New Lua commands: bit.tobit, bit.tohex, bit.bnot, bit.band, bit.bor, bit.bxor,
bit.lshift, bit.rshift, bit.arshift, bit.rol, bit.ror, bit.bswap

Verification of working (the asserts would abort on error, so (nil) is correct):
127.0.0.1:6379> eval "assert(bit.tobit(1) == 1); assert(bit.band(1) == 1); assert(bit.bxor(1,2) == 3); assert(bit.bor(1,2,4,8,16,32,64,128) == 255)" 0
(nil)
127.0.0.1:6379> eval 'assert(0x7fffffff == 2147483647, "broken hex literals"); assert(0xffffffff == -1 or 0xffffffff == 2^32-1, "broken hex literals"); assert(tostring(-1) == "-1", "broken tostring()"); assert(tostring(0xffffffff) == "-1" or tostring(0xffffffff) == "4294967295", "broken tostring()")' 0
(nil)

Tests also integrated into the scripting tests and can be run with:
./runtest --single unit/scripting

Tests are excerpted from `bittest.lua` included in the bitop distribution.
 2014-11-14 17:28:19 +01:00
antirez d9326cb83d THP detection for LATENCY DOCTOR. 2014-11-12 11:17:31 +01:00
antirez 011cfdfb6b Check THP support at startup and warn about it. 2014-11-12 11:17:27 +01:00
antirez c0266d87ae THP detection / reporting functions added. 2014-11-12 11:17:24 +01:00
antirez a5fcf44fcc Diskless SYNC: fix RDB EOF detection. 
RDB EOF detection was relying on the final part of the RDB transfer to
be a magic 40 bytes EOF marker. However as the slave is put online
immediately, and because of sockets timeouts, the replication stream is
actually contiguous with the RDB file.

This means that to detect the EOF correctly we should either:

1) Scan all the stream searching for the mark. Sucks CPU-wise.
2) Start to send the replication stream only after an acknowledge.
3) Implement a proper chunked encoding.

For now solution "2" was picked, so the master does not start to send
ASAP the stream of commands in the case of diskless replication. We wait
for the first REPLCONF ACK command from the slave, that certifies us
that the slave correctly loaded the RDB file and is ready to get more
data.
 2014-11-11 17:27:22 +01:00
antirez ec1ab0f18b Disconnect timedout slave: regression introduced with diskless repl. 2014-11-11 17:25:58 +01:00
Matt Stancliff eb67ca0f00 redis-cli: ignore SIGPIPE network errors 
Closes #2066
 2014-10-30 12:02:32 +01:00
antirez bb9ad32cb0 Fix DEBUG POPULATE warning for lack of casting. 2014-10-29 14:36:55 +01:00
antirez a37f8ec8b4 DEBUG POPULATE two args form implemented. 
The old DEBUG POPULATE form for automatic creation of test keys is:

    DEBUG POPULATE <count>

Now an additional form is available:

    DEBUG POPULATE <count> <prefix>

When prefix is not specified, it defaults to "key", so the keys are
named incrementally from key:0 to key:<count-1>. Otherwise the specified
prefix is used instead of "key".

The command is useful in order to populate different Redis instances
with key names guaranteed to don't collide. There are other debugging
uses, for example it is possible to add additional N keys using a count
of N and a random prefix at every call.
 2014-10-29 14:36:55 +01:00
antirez b2509d7759 redis-cli: add missing newline in error message. 2014-10-29 14:35:48 +01:00
antirez 9b7a190c82 Diskless replication: missing listRewind() added. 
This caused BGSAVE to be triggered a second time without any need when
we switch from socket to disk target via the command

    CONFIG SET repl-diskless-sync no

and there is already a slave waiting for the BGSAVE to start.
Also comments clarified about what is happening.
 2014-10-29 14:34:58 +01:00
antirez 49629a2e34 Log slave ip:port in more log messages. 2014-10-29 14:34:58 +01:00
antirez 081eeb9f6b Use new slave name function for diskless repl reporting. 2014-10-29 14:34:58 +01:00
antirez 72ea77af0b Added a function to get slave name for logs. 2014-10-29 14:34:51 +01:00
antirez 16319c69b1 Diskless replication: log BGSAVE delay only when it is non-zero. 2014-10-29 14:34:25 +01:00
antirez bafd3f646f Document repl-diskless-sync-delay in redis.conf. 2014-10-29 14:34:25 +01:00
antirez f71c4d54d1 Diskless sync delay is now configurable. 2014-10-29 14:34:18 +01:00
antirez 4b520177b3 Remove duplicated log message about starting BGSAVE. 2014-10-29 14:33:50 +01:00
antirez 7e8a471baf Diskless replication documented inside example redis.conf. 2014-10-29 14:33:50 +01:00
antirez d57ed95222 Diskless replication tested with the multiple slaves consistency test. 2014-10-29 14:33:50 +01:00
antirez 7d32dfc69f Diskless replication: child -> parent communication improved. 
Child now reports full info to the parent including IDs of slaves in
failure state and exit code.
 2014-10-29 14:33:50 +01:00
antirez 7728ee809d Translate rio fdset target EWOULDBLOCK error into ETIMEDOUT. 
EWOULDBLOCK with the fdset rio target is returned when we try to write
but the send timeout socket option triggered an error. Better to
translate the error in something the user can actually recognize as a
timeout.
 2014-10-29 14:33:50 +01:00
antirez 7cffd74b38 Diskless replication: set / reset socket send timeout. 
We need to avoid that a child -> slaves transfer can continue forever.
We use the same timeout used as global replication timeout, which is
documented to also affect I/O operations during bulk transfers.
 2014-10-29 14:33:50 +01:00
antirez eff8fb6f39 anet.c: new API anetSendTimeout(). 2014-10-29 14:33:50 +01:00
antirez c38bb5b2ea Diskless replication: less debugging printfs around. 2014-10-29 14:33:50 +01:00
antirez 348c4f6ce3 rio.c fdset write() method fixed: wrong type for return value. 2014-10-29 14:33:50 +01:00
antirez a7d0137075 rio fdset target: handle short writes. 
While the socket is set in blocking mode, we still can get short writes
writing to a socket.
 2014-10-29 14:33:50 +01:00
antirez 23bf435f54 anet.c: API to set sockets back to blocking mode. 2014-10-29 14:33:50 +01:00
antirez 9a521e6c74 Diskless replication: rio fdset target new supports buffering. 
To perform a socket write() for each RDB rio API write call was
extremely unefficient, so now rio has minimal buffering capabilities.
Writes are accumulated into a buffer and only when a given limit is
reacehd are actually wrote to the N slaves FDs.

Trivia: rio lacked support for buffering since our targets were:

1) Memory buffers.
2) C standard I/O.

Both were buffered already.
 2014-10-29 14:33:50 +01:00
antirez 9588973fda Diskless replication: Various fixes to backgroundSaveDoneHandlerSocket() 2014-10-29 14:33:50 +01:00
antirez 8a618971bd Diskless replication: don't send "\n" pings to slaves. 
This is useful for normal replication in order to refresh the slave
when we are persisting on disk, but for diskless replication the
child is already receiving data while in WAIT_BGSAVE_END state.
 2014-10-29 14:33:50 +01:00
antirez b794b94797 Diskless replication: remove 40 bytes EOF mark from end of RDB file. 2014-10-29 14:33:50 +01:00
antirez 3f1f29fdf3 Diskless replication: swap inverted branches to compute read len. 2014-10-29 14:33:50 +01:00
antirez dbc6e9a1ef Diskless replication: don't enter the read-payload branch forever. 2014-10-29 14:33:49 +01:00
antirez adf55b2337 syncReadLine(): actually enforce buffer size limits. 2014-10-29 14:33:49 +01:00
antirez 89f82a25e1 Diskless replication: EOF:<mark> streaming support slave side. 2014-10-29 14:33:49 +01:00
antirez 583a762cd9 Diskless replication: redis.conf and CONFIG SET/GET support. 2014-10-29 14:33:49 +01:00
antirez 31b0e13268 Diskless replication: trigger a BGSAVE after a config change. 
If we turn from diskless to disk-based replication via CONFIG SET, we
need a way to start a BGSAVE if there are slaves alerady waiting for a
BGSAVE to start. Normally with disk-based replication we do it as soon
as the previous child exits, but when there is a configuration change
via CONFIG SET, we may have slaves in WAIT_BGSAVE_START state without
an RDB background process currently active.
 2014-10-29 14:33:49 +01:00
antirez d403e4ae63 Diskless replication flag renamed repl_diskless -> repl_diskless_sync. 2014-10-29 14:33:49 +01:00
antirez 2f0b58e4ed Diskless replication: trigger diskless RDB transfer if needed. 2014-10-29 14:33:49 +01:00
antirez 82bfae5b70 Diskless replication: handle putting the slave online. 2014-10-29 14:33:49 +01:00
antirez 132c108f17 Diskless replication: read report from child. 2014-10-29 14:33:49 +01:00
antirez e82ec2c5d8 Diskless replication: child writes report to parent. 2014-10-29 14:33:49 +01:00
antirez 2ed67e4a9e rio.c fdset target: tolerate (and report) a subset of FDs in error. 
Fdset target is used when we want to write an RDB file directly to
slave's sockets. In this setup as long as there is a single slave that
is still receiving our payload, we want to continue sennding instead of
aborting. However rio calls should abort of no FD is ok.

Also we want the errors reported so that we can signal the parent who is
ok and who is broken, so there is a new set integers with the state of
each fd. Zero is ok, non-zero is the errno of the failure, if avaialble,
or a generic EIO.
 2014-10-29 14:33:49 +01:00
antirez 51864fcdd7 Diskless replication: parent-child pipe and a few TODOs. 2014-10-29 14:33:49 +01:00
antirez 1b4cadb664 Diskless replication: RDB -> slaves transfer draft implementation. 2014-10-29 14:33:49 +01:00
antirez 38dee9dc51 rio.c: draft implementation of fdset target implemented. 2014-10-29 14:33:49 +01:00
antirez 5d295fea12 rio.c refactoring before adding a new target. 2014-10-29 14:33:49 +01:00
antirez aef4c60c78 Add some comments in syncCommand() to clarify RDB target. 2014-10-29 14:33:49 +01:00
antirez 81abd1e178 Define different types of RDB childs. 
We need to remember what is the saving strategy of the current RDB child
process, since the configuration may be modified at runtime via CONFIG
SET and still we'll need to understand, when the child exists, what to
do and for what goal the process was initiated: to create an RDB file
on disk or to write stuff directly to slave's sockets.
 2014-10-29 14:33:49 +01:00
antirez 1e633f21c5 RDB file creation refactored to target non-disk target. 2014-10-29 14:33:49 +01:00
antirez ff8a3baaf7 Replication: better way to send a preamble before RDB payload. 
During the replication full resynchronization process, the RDB file is
transfered from the master to the slave. However there is a short
preamble to send, that is currently just the bulk payload length of the
file in the usual Redis form $..length..<CR><LF>.

This preamble used to be sent with a direct write call, assuming that
there was alway room in the socket output buffer to hold the few bytes
needed, however this does not scale in case we'll need to send more
stuff, and is not very robust code in general.

This commit introduces a more general mechanism to send a preamble up to
2GB in size (the max length of an sds string) in a non blocking way.
 2014-10-29 14:33:30 +01:00
antirez b6203138d3 Linenoise README updated to match source code. 2014-10-06 10:02:08 +02:00
Miguel Parramon 85b15acb44 redis.conf small grammar fix 
😄

Closes #2034
 2014-10-06 10:02:08 +02:00
Ezequiel Lovelle 09fc0fbbe4 Fix typo 
Closes #2029
 2014-10-06 10:02:08 +02:00
Gregory Petrosyan 03b36eef72 Fix typos in comments 
Closes #2002
 2014-10-06 10:01:36 +02:00
Matt Stancliff e1ccd7243f Add missing 'by' 2014-10-06 10:01:36 +02:00
Juarez Bochi 364aeed96d Fix typo in scripting.c comment 
Closes #1960
 2014-10-06 10:01:36 +02:00
T.J. Schuck a7a2f69bdb Fix typo radis -> redis 
Closes #1938
 2014-10-06 10:01:36 +02:00
Matt Stancliff 1adc4a7888 Cleanup wording of dictScan() comment 
Some language in the comment was difficult
to understand, so this commit: clarifies wording, removes
unnecessary words, and relocates some dependent clauses
closer to what they actually describe.

I also tried to break up longer chains of thought
(if X, then Y, and Q, and also F, so obviously M)
into more manageable chunks for ease of understanding.
 2014-10-06 10:01:36 +02:00
Matt Stancliff c3375fbdff Remove trailing spaces from tests 2014-10-06 10:01:27 +02:00
Matt Stancliff c4d32c03d0 Fix spelling in some test cases 2014-10-06 10:01:05 +02:00
Jan-Erik Rediger 919a4c3126 Fix sentinel.conf typo 
Closes #1739
 2014-10-06 10:00:48 +02:00
Xiaojie Zhang 8a1d16fe5c Fix comment inconsistencies in ziplist.c 
Closes #1523
 2014-10-06 10:00:48 +02:00
Matt Stancliff e0ffc863c6 Clean up text throughout project 
- Remove trailing newlines from redis.conf
  - Fix comment misspelling
  - Clarifies zipEncodeLength usage and a C API mention (#1243, #1242)
  - Fix cluster typos (inspired by @papanikge #1507)
  - Fix rewite -> rewrite in a few places (inspired by #682)

Closes #1243, #1242, #1507
 2014-10-06 10:00:43 +02:00
Matt Stancliff 22e6f434f8 Spell software correctly 2014-10-06 09:59:52 +02:00
Aaron Rutkovsky 3101760937 Fix typos 
Closes #1513
 2014-10-06 09:59:02 +02:00
Ben d3005539c5 Grammar and typo fixes in redis.conf 
Closes #1441
 2014-10-06 09:58:56 +02:00
xuxiang 3c99e429cf Fix typo in redis.h 
Closes #1386
 2014-10-06 09:58:18 +02:00
Ted Nyman 55597bbd67 Fix on-place -> in-place 
Closes #1373
 2014-10-06 09:58:18 +02:00
Michael Parker a5044c254d Fix hash table size in comment for dictScan 
Closes #1351
 2014-10-06 09:58:18 +02:00
Jan-Erik Rediger 33daf8e9a9 Fix many small typos 
Closes #1871
 2014-10-06 09:58:11 +02:00
zionwu b72422e100 Fix incorrect comments 
error != success; and 0 != number of bytes written

Closes #1806
 2014-10-06 09:57:01 +02:00
Jan-Erik Rediger 6cdd38f947 Fix typo in redis.conf 
Closes #1713
 2014-10-06 09:57:01 +02:00
Juarez Bochi 6b64fa49f8 Fix typo 
Closes #1682
 2014-10-06 09:57:01 +02:00
Agis Anastasopoulos fc9bce45c5 Fix typo 
Closes #1544
 2014-10-06 09:57:01 +02:00
Jan-Erik Rediger 575e293ddd Fix typo: ad -> and 
Closes #1537
 2014-10-06 09:57:01 +02:00
Matt Stancliff d05fcaffa0 Update linenoise accessory files 
The core linenoise code was being backported, but not
the README or example.  It's less confusing for users
if everything matches across directories.

Fix inspired by @thrig

Closes #1872
 2014-10-06 09:57:01 +02:00
Manuel Meurer a99d187bc4 Improve disable save comment in redis.conf 
Closes #1897
 2014-10-06 09:57:01 +02:00
Lynn 904bd0400c Remove surplus double quotes 
Closes #1877
 2014-10-06 09:57:01 +02:00
antirez 04b6945ebc Linenoise lib updated. 2014-09-22 14:56:01 +02:00
antirez ade6c5e782 Redis 2.8.17 2014-09-19 17:49:16 +02:00
Matt Stancliff cf143b571e Fix hiredis getaddrinfo leak 
Fixed in Redis by 1a5e5b6, but since that part of code
is largely copy/paste from Redis, the fix needs to be
ported over too.

Closes #2012
 2014-09-19 14:21:36 +02:00
antirez a69534967b Redis 2.8.16. 2014-09-16 15:15:56 +02:00
antirez 5e38bc389c Better truncated AOF loading tests. 
Now there are tests to write more data after loading a truncated AOF,
testing that the loaded data is correct, appending more, and testing
again.
 2014-09-16 11:05:50 +02:00
antirez 01f7db608d Seek at the end of AOF after truncate call. 
It is not clear if files open in append only mode will automatically fix
their offset after a truncate(2) operation. This commit makes sure that
we reposition the AOF file descriptor offset at the end of the file
after a truncated AOF is loaded and trimmed to the last valid command.
 2014-09-16 11:05:50 +02:00
antirez 9faeff0215 On AOF end of file, truncate the AOF to last valid command. 
Recently we introduced the ability to load truncated AOFs, but
unfortuantely the support was broken since the server, after loading the
truncated AOF, continues appending to the file that is corrupted at the
end. The problem is fixed only in the next AOF rewrite.

This commit fixes the issue by truncating the AOF to the last valid
opcode, and aborting if it is not possible to truncate the file
correctly.
 2014-09-16 10:41:15 +02:00
antirez fbd9dc6039 Don't propagate SAVE. 
This is a general fix (check that dirty delta is positive) but actually
should have as the only effect fixing the SAVE propagation to
AOF and slaves.
 2014-09-16 10:41:15 +02:00
antirez 551ffc486c version.h updated to 2.8.15. 2014-09-12 16:16:58 +02:00
antirez 78dca8977f Redis 2.8.15. 2014-09-12 16:15:48 +02:00
antirez a6695e341b Sentinel sentinelGetLeader() top comment improved. 2014-09-11 19:28:10 +02:00
antirez 9fe4429a09 Sentinel: fix computation of total number of votes. 
The code to check the number of voters was never updated to follow the new
Sentinel specification, so the number of voters was computed using only
the set of Sentinels that provided a vote.

This means that there is a changing majority on partitions, even if
usually the issue is not triggered because of the configured quorum
check (what was broken was the other implicit check that requires anyway
half of the known sentinels to agree in order to start a failover).
 2014-09-11 19:28:09 +02:00
antirez 6fbcb19275 luaRedisGenericCommand() cached argv handling simplified. 
As discussed in issue #1945.
 2014-09-10 15:47:02 +02:00
xiaost 8375a6d0aa Limit the *SCAN command dictScan iterations 
*SCAN will cause redis server to hang for seconds
after millions of keys was deleted by SCAN/DEL pairs
 2014-09-10 09:58:20 +02:00
antirez 55024e9a9a Sentinel: don't set announce-ip if is empty. 2014-09-10 09:26:29 +02:00
antirez 3cc31da349 Sentinel: clarify announce-ip/port options in sentinel.conf. 2014-09-10 09:26:29 +02:00
antirez 8c3507aa8f Sentinel: announce ip/port changes + rewrite. 
The original implementation was modified in order to allow to
selectively announce a different IP or port, and to rewrite the two
options in the config file after a rewrite.
 2014-09-10 09:26:29 +02:00
Dara Kong 9f3abf436b sentinel: Decouple bind address from address sent to other sentinels 
There are instances such as EC2 where the bind address is private
(behind a NAT) and cannot be accessible from WAN.

https://groups.google.com/d/msg/redis-db/PVVvjO4nMd0/P3oWC036v3cJ
 2014-09-10 09:26:29 +02:00
antirez ff47d5ebf6 Tests for aof-load-truncated = yes. 2014-09-08 10:58:00 +02:00
antirez b3fdddfd1f AOF tests fixed turning aof-load-truncated to no. 
When aof-load-truncated option was introduced, with a default of "yes",
the past behavior of the server to abort with trunncated AOF changed, so
we need to explicitly configure the tests to abort with truncated AOF
by setting the option to no.
 2014-09-08 10:58:00 +02:00
antirez 670084b724 Make aof-load-truncated option actually configurable. 2014-09-08 10:58:00 +02:00
antirez 03a5c0ecb5 Correct cleanup when aborting AOF loading. 
Because of the new ability to start with a truncated AOF, we need
to correctly release all the memory on EOF error. Otherwise there is a
small leak, that is not really a problem, but causes a false positive in
the tests that detect memory leaks.
 2014-09-08 10:58:00 +02:00
antirez 683cf5b872 AOF ability to load truncated files. 2014-09-08 10:57:57 +02:00
antirez 94f7979414 Test AOF format error detection. 2014-09-08 10:57:11 +02:00
antirez 537f3053fa AOF loading: split handling of format errors from unexpected EOF. 2014-09-08 10:57:10 +02:00
Matt Stancliff 667e2991dc Return empty string if GETRANGE of empty string 
Previously, GETRANGE of a key containing nothing ("")
would allocate a large (size_t)-1 return value causing
crashes on 32bit builds when it tried to allocate the
4 GB return string.
 2014-09-04 11:47:21 +02:00
Matt Stancliff 4284b07220 Increase size of range request in getrange 
32 bit builds don't have a big enough long to capture
the same range as a 64 bit build.  If we use "long long"
we get proper size limits everywhere.

Also updates size of unsigned comparison to fit new size of `end`.

Fixes #1981
 2014-09-04 11:46:57 +02:00
antirez db6e874a25 Redis 2.8.14. 2014-09-01 17:00:38 +02:00
Matt Stancliff dd36f18c76 Sentinel: Abort Hello quicker if not connected 
We can save a little work by aborting when we enter the function
if we're disconnected.
 2014-09-01 16:35:04 +02:00
Paddy Byers d1f89bdc2b Add regression test for issue #1939 2014-09-01 10:41:15 +02:00
Paddy Byers 59ad68ca4f Store the length of the static argv when first allocated. 2014-09-01 10:40:04 +02:00
antirez 8fb8a474fd Fix dictRehash assert casting type. 
Also related to #1929.
 2014-08-27 10:31:52 +02:00
antirez 85b3db4065 Cast to right type in dictNext(). 
This closes issue #1929, the other part was fixed in the context of issue
 2014-08-27 10:31:48 +02:00
antirez 3a706ba792 Use long for rehash and iterator index in dict.h. 
This allows to support datasets with more than 2 billion of keys
(possible in very large memory instances, this bug was actually
reported).

Closes issue #1814.
 2014-08-27 10:31:44 +02:00
antirez 97fdcd8746 Missing va_end also added in hiredis copy of sds.c. 2014-08-27 10:31:41 +02:00
Mariano Pérez Rodríguez 45fac9481a Missing assert removal from sdsIncrLen() 
Companion for 8eeb1802ec, but dealing with hiredis.
 2014-08-27 10:31:35 +02:00
antirez c3eac3ddff No longer useful assert removed from sdsIncrLen(). 2014-08-27 10:31:32 +02:00
Matt Stancliff fdcadd16c6 Fix redis-benchmark abort output 2014-08-27 10:31:29 +02:00
Matt Stancliff ac48aada30 Deny CLIENT command in scripts 
We don't want scripts doing CLIENT SETNAME
or CLIENT KILL or CLIENT LIST or CLIENT PAUSE.

Originally reported by Chris Wj then proper
action inspired by Itamar Haber.

Reference: https://groups.google.com/forum/#!topic/redis-db/09B2EYwyVgk
 2014-08-27 10:31:20 +02:00
Matt Stancliff abe29c83e6 redis-benchmark: Remove unused buffer 
This just deletes old code that didn't get removed when
logic changed.  We were setting offsets that never
got read anywhere.

Since clients are now just cloned, we don't need to track
per-client buffer offsets anywhere because they are all
the same from the original client.
 2014-08-27 10:30:49 +02:00
CharSyam 8fcce90b25 redis-benchmark: add auth option 
Closes #1097
 2014-08-27 10:30:49 +02:00
Matt Stancliff 4ac8472cac Rename two 'buf' vars to 'ip' for better clarity 
Clearly ip[32] is wrong, but it's less clear that buf[32] was wrong
without further reading.
 2014-08-27 10:30:42 +02:00
Eiichi Sato f7b5e2d11e Sentinel: fix bufsize to support IPv6 address 
Closes #1914
 2014-08-27 10:30:37 +02:00
Jan-Erik Rediger 32a94beca4 Use correct github url to find commands.json 
Once this is merged:
  - merge the latest changes to commands.json in antirez/redis-doc
  - re-run: utils/generate-command-help.rb > src/help.h

Then we'll have nice and easy tab-completed help in redis-cli again.

Closes #1909
 2014-08-27 10:30:34 +02:00
Mariano Pérez Rodríguez f862e5ddfa Stop tests from leaving a black background 
Uses ANSI "default background" color code after closing tests
so any non-black terminals don't remain polluted.

Fixes #1649
Closes #1912
 2014-08-27 10:30:29 +02:00
Matt Stancliff 32168a5e8a Sentinel: Reject config from STDIN 
Sentinel needs to die with a more accurate error message
when attempted to open a config from STDIN.

See: https://groups.google.com/forum/#!topic/redis-db/sYx7VNMWaNM
 2014-08-27 10:30:26 +02:00
Matt Stancliff 4bf7c5d0f8 Remove unused global variable 
It has an important name, but nothing uses it...
 2014-08-27 10:30:22 +02:00
antirez 33b37b1400 Fix invalid expire error for SET family commands. 2014-08-27 10:30:19 +02:00
Cong Ding ae2cd2e16f Remove unused function 
Closes #878
 2014-08-27 10:30:11 +02:00
David Palm 8c0743d80e Extend range of bytesToHuman to TB and PB 
Also adds a fallthrough case for when given
large values (like overflow numbers of 2^64 by mistake).

Closes #858
 2014-08-27 10:30:08 +02:00
yoav ef12b4c3eb Add error check for writing RDB checksum 
Closes #857
 2014-08-27 10:30:04 +02:00
Dov Murik 1580a378b8 redis-cli: fix prompt after shutdown command 
Fix redis-cli prompt to state "not connected" after a SHUTDOWN command
is sent.
 2014-08-27 10:30:01 +02:00
xiaoyu 467848460a Clarify argument to dict macro 
d is more clear because the type of argument is dict not dictht

Closes #513
 2014-08-27 10:29:57 +02:00
antirez 5e707c66dd Fix undefined behavior in ll2string(). 
The bug was found by @CAFxX, thanks!
See issue #1940.
 2014-08-27 10:29:53 +02:00
antirez 65d47452f8 Remove warnings and improve integer sign correctness. 2014-08-27 10:29:26 +02:00
antirez 09757420a4 Add -W to compilation flags. 2014-08-27 10:25:57 +02:00
antirez 134345e332 Use unsigned integers in SDS header. 
This raises the max string to 4GB without any downside.
 2014-08-27 10:25:54 +02:00
Pieter Noordhuis f89a7bbe11 va_copy must be matched by va_end 
Hat tip to @rfuchs. See: https://github.com/redis/hiredis/pull/178.

Fixes #1187
 2014-08-27 10:25:44 +02:00
charsyam dbcf381099 Remove duplicate prototypes in redis.h 
Also moves acceptHandler() to be near the other accept...() functions.

Closes #1105
 2014-08-27 10:25:40 +02:00
NanXiao 6bc74b4393 Use resolv library in Solaris 
For some Solaris flavours, the inet_aton in in resolv library.
Not linking this library will introduce link error.

Improves compatability with older Solaris and still
works on new Solaris.

Closes #1092
 2014-08-27 10:25:37 +02:00
Daniel Price 1aff711d90 Define AR to help with lua cross-compilation 
Closes #997
 2014-08-27 10:25:34 +02:00
Matt Stancliff 866f6bcc80 Cleanup double semicolons 
Closes #1161
 2014-08-27 10:25:31 +02:00
Chris Lamb 29ed617ac4 redis-check-dump: Prevent segfault if can't malloc 
Found by The Mayhem Team (Alexandre Rebert, Thanassis Avgerinos,
Sang Kil Cha, David Brumley, Manuel Egele) Cylab, Carnegie Mellon
University. See http://bugs.debian.org/716259 for more.

Signed-off-by: Chris Lamb <lamby@debian.org>

Fixes #1191
 2014-08-27 10:25:28 +02:00
charsyam a43e45e391 Remove unused LINE_BUFLEN definition 
Closes #1129
 2014-08-27 10:25:25 +02:00
Matt Stancliff 25a4699a9e Reject MOVE to non-integer DBs 
Previously, "MOVE key somestring" would move the key to
DB 0 which is just unexpected and wrong.
String as DB == error.

Test added too.

Modified by @antirez in order to use the getLongLongFromObject() API
instead of strtol().

Fixes #1428
 2014-08-27 10:25:22 +02:00
Matt Stancliff 09a3976478 pubsub: Return integers for NUMSUB, not strings 
Also adds test for numsub — due to tcl being tcl,
it doesn't capture the "numberness" of the fix,
but now we at least have one test case for numsub.

Closes #1561
 2014-08-27 10:25:15 +02:00
Xiaojie Zhang 60c748628b Avoid unnecessary decoding in ziplist.c 
Closes #1519
 2014-08-27 10:25:12 +02:00
Samuel Chou 96614f5866 Improve accuracy of HAVE_ATOMIC dependency check 
[I had to split out the clang check due to
 clang *really* not liking the __GLIBC_PREREQ macro; -matt]

Closes #1456
 2014-08-27 10:25:07 +02:00
Jim Howard d1df50ad59 Remove redundant event loop fd processing 
Reason:
https://github.com/antirez/redis/pull/1450#issuecomment-30432925

Closes #1450
 2014-08-27 10:25:04 +02:00
Xian Li 282c6106ce memtest: Add missing free() 
Closes #1614
 2014-08-27 10:25:00 +02:00
Joshua Anderson 25b5abe8cf src/Makefile: Reword "to run make test" message 
Closes #1610
 2014-08-27 10:24:56 +02:00
cubicdaiya a7c46df612 Use 'void' for zero-argument functions 
According to the C standard,
it is desirable to give the type 'void'
to functions have no argument.

Closes #1631
 2014-08-27 10:24:52 +02:00
Matt Stancliff 8dbd0134e5 Fix assert technical correctness 
dictAdd returns DICT_OK, not REDIS_OK. They both
have the same underlying values, so it works even though
the code is technically wrong.

Fixes #1512
 2014-08-27 10:24:46 +02:00
Jan-Erik Rediger 0c8999dfeb install_server.sh: add missing bang 
This was discovered by _bodya and reported in the IRC channel.
Everything worked fine as these scripts are always executed as shell
scripts.

Closes #1728
 2014-08-27 10:24:42 +02:00
Edgars Irmejs 0f9ab49d58 Change unixsocketperm comment to 700 from 755 
According to unix manuals, "Connecting to the socket object requires
read/write permission." -- mode 755 is useless for anybody
other than the owner.

Fixes #1696
 2014-08-27 10:24:39 +02:00
ripcurld00d 4be50f7fbb redis-check-dump: use names instead of magic nums 
Use constants to avoid magic numbers in `types`, which is an array
that stores the names of `REDIS` types.

Closes #1681
 2014-08-27 10:24:36 +02:00
Kashif Rasul 8fc2247c9b Fix issues raised by clang analyzer 
Modified by @antirez since the original fix to genInfoString() looked
weak. Probably the clang analyzer complained about `section` being
possibly NULL, and strcasecmp() called with a NULL pointer. In the
practice this can never happen, still for the sake of correctness
the right fix is not to modify only the first call, but to set `section`
to the value of "default" if it happens to be NULL.

Closes #1660
 2014-08-27 10:24:33 +02:00
antirez ebd25710dd Force quit when receiving a second SIGINT. 
Also quit ASAP when we are still loading a DB, since care is not needed
in this special condition, especially for a SIGINT.
 2014-08-27 10:24:29 +02:00
Matt Stancliff 524b2626f7 Add graceful exit when Ctrl-C is received 2014-08-27 10:24:26 +02:00
Matt Stancliff 78bde472c0 Fix intset midpoint selection 
The classic (min+max)/2 is provably unsafe.  Fixed
as recommended in research:
http://googleresearch.blogspot.com/2006/06/extra-extra-read-all-about-it-nearly.html

Fix inspired by @wjin, but I used a different approach.
(later, I found @kuebler fixed the same issue too).

Fixes #1741, #1602
 2014-08-27 10:24:22 +02:00
Jan-Erik Rediger 07b2779bff Handle large getrange requests 
Previously the end was casted to a smaller type
which resulted in a wrong check and failed
with values larger than handled by unsigned.

Closes #1847, #1844
 2014-08-27 10:24:19 +02:00
Matt Stancliff 4024220fe6 scripting: no eval with negative key count 
Negative key count causes segfault in Lua functions.

Fixes #1842
Closes #1843
 2014-08-27 10:24:15 +02:00
Jan-Erik Rediger 50669f7b66 redis-cli: fix latency result output 
(Cleaned up a little by @mattsta)

Closes #1774
 2014-08-27 10:24:11 +02:00
Wei Jin 9b53f79e00 Remove redundant else/return block 
Fixes #1741
 2014-08-27 10:24:08 +02:00
antirez 9840df7bad Clarify AIX "#undef hz". 2014-08-27 10:22:59 +02:00
siahl b5dc2c5b91 Add support for compiling on AIX 
Closes #1900
 2014-08-27 10:22:28 +02:00
Matt Stancliff cbd1607c6a redis-cli: Add --no-raw option 
Some people need formatted output even when they have no
interactive tty.

Fixes #760
 2014-08-27 10:21:19 +02:00
Matt Stancliff 9a039f76c2 redis-cli: stop showing incorrectly selected DB 
Previously redis-cli would happily show "-1" or "99999"
as valid DB choices.

Now, if the SELECT call returned an error, we don't update
the DB number in the CLI.

Inspired by @anupshendkar in #1313

Fixes #566, #1313
 2014-08-27 10:21:19 +02:00
Matt Stancliff 28b54b5d94 redis-cli: Re-attach selected DB after auth 
Previously, if you did SELECT then AUTH, redis-cli
would show your SELECT'd db even though it didn't
happen.

Note: running into this situation is a (hopefully) very limited
used case of people using multiple DBs and AUTH all at the same
time.

Fixes antirez#1639
 2014-08-27 10:21:19 +02:00
Yossi Gottlieb ff58d5a380 Fail SYNC if background save child aborted due to a signal. 2014-07-28 14:46:03 +02:00
Salvatore Sanfilippo eb7b3427e7 Merge pull request #1894 from veselov/2.8 
* fixed doc URL for keyspace events
 2014-07-28 09:38:33 +02:00
vps 4d7b3815d5 * fixed doc URL for keyspace events 2014-07-27 20:14:39 -07:00
antirez 9fa51dd4d1 Example redis.conf: improve slaveof description. 2014-07-25 14:20:31 +02:00
antirez d558af8e6f Fix util.c compilation by including stdint.h. 2014-07-23 18:01:51 +02:00
antirez e426e77896 Faster ll2string() implementation. 
Based on ideas documented in this blog post:

https://www.facebook.com/notes/facebook-engineering/three-optimization-tips-for-c/10151361643253920

The original code was modified to handle signed integers, reformetted to
fit inside the Redis code base, and was stress-tested with a program
in order to validate the implementation against snprintf().

Redis was measured to be measurably faster from the point of view of
clients in real-world operations because of this change, since sometimes
number to string conversion is used extensively (for example every time
a GET results into an integer encoded object to be returned to the
user).
 2014-07-23 14:58:26 +02:00
antirez 8a44e10848 PFSELFTEST: less false positives. 
This is just a quickfix, for the nature of the test the right way to fix
it is to average the error of N runs, since otherwise it is always
possible to get a false positive with a bad run, or to minimize too much
this possibility we may end testing with too much "large" error ranges.
 2014-07-23 11:45:03 +02:00
antirez 0bfdd4bb49 Test: check sorted set elements order after union. 
This is not a regression but issue #1786 showed the need for this test.
 2014-07-22 17:58:41 +02:00
antirez 098bb97703 Remove useless var and check in zunionInterGenericCommand(). 2014-07-22 17:58:26 +02:00
antirez 316e1db970 ZUNIONSTORE reimplemented for speed. 
The user @kjmph provided excellent ideas to improve speed of ZUNIONSTORE
(in certain cases by many order of magnitude), together with an
implementation of the ideas.

While the ideas were sounding, the implementation could be improved both
in terms of speed and clearness, so that's my attempt at reimplementing
the speedup proposed, trying to improve by directly using just a
dictionary with an embedded score inside, and reusing the single-pass
aggregate + order-later approach.

Note that you can't apply this commit without applying the previous
commit in this branch that adds a double in the dictEntry value union.

Issue #1786.
 2014-07-22 17:58:05 +02:00
antirez c8a6cd94b3 Add double field in dict.c entry value union. 2014-07-22 17:53:06 +02:00
antirez 7b11dfe05e Test: Pub/Sub PING. 2014-07-18 12:03:38 +02:00
antirez 156029b336 Test: small integer sharing depends on maxmemory policy. 2014-07-18 10:55:19 +02:00
antirez 06e9b3cad8 PING: backward compatible error for wrong number of args. 2014-07-18 10:46:21 +02:00
antirez ab5f20d19c tryObjectEncoding(): use shared objects with maxmemory and non-LRU policy. 
In order to make sure every object has its own private LRU counter, when
maxmemory is enabled tryObjectEncoding() does not use the pool of shared
integers. However when the policy is not LRU-based, it does not make
sense to do so, and it is much better to save memory using shared
integers.
 2014-07-18 10:46:05 +02:00
antirez 70e39481ab Variadic PING with support for Pub/Sub. 
PING can now be called with an additional arugment, behaving exactly
like the ECHO command. PING can now also be called in Pub/Sub mode (with
one more more subscriptions to channels / patterns) in order to trigger
the delivery of an asynchronous pong message with the optional payload.

This fixes issue #420.
 2014-07-18 10:41:47 +02:00
antirez 294bcfc4d2 PubSub clients refactoring and new PUBSUB flag. 
The code tested many times if a client had active Pub/Sub subscriptions
by checking the length of a list and dictionary where the patterns and
channels are stored. This was substituted with a client flag called
REDIS_PUBSUB that is simpler to test for. Moreover in order to manage
this flag some code was refactored.

This commit is believed to have no effects in the behavior of the
server.
 2014-07-18 10:41:22 +02:00
antirez 0c29cadd46 Redis 2.8.13. 2014-07-14 17:48:14 +02:00
michael-grunder 3df2ab673c Fix OBJECT arity 
Previously, the command definition for the OBJECT command specified
a minimum of two args (and that it was variadic), which meant that
if you sent this:

OBJECT foo

When cluster was enabled, it would result in an assertion/SEGFAULT
when Redis was attempting to extract keys.

It appears that OBJECT is not variadic, and only ever takes 3 args.

https://gist.github.com/michael-grunder/25960ce1508396d0d36a
 2014-07-14 15:38:32 +02:00
antirez c970507478 Fix typo in LATENCY DOCTOR output. 2014-07-11 10:57:34 +02:00
antirez bb10ad9c13 Test: more reliable AOF rewrite test under write load. 2014-07-10 16:42:52 +02:00
antirez c785d4cccc LATENCY command / monitor basic tests. 2014-07-10 16:24:24 +02:00
antirez e1fd73f634 LATENCY HISTORY returns empty array if no data is available. 
Previously we returned an error.
 2014-07-10 16:24:24 +02:00
antirez 315536d012 Test: enable latency monitor in the default config. 
This way as a side effect of running the test we also stress the latency
monitor data collection.
 2014-07-10 15:04:59 +02:00
antirez f0e67afd94 Test: AOF rewrite during write load. 2014-07-10 11:24:59 +02:00
antirez 1f3c097b1c Fixed a few missing newline in createLatencyReport(). 2014-07-09 20:02:01 +02:00
antirez e4db34e97c createLatencyReport(): compile before commit avoids commits. 2014-07-09 20:01:57 +02:00
antirez f9765990b2 createLatencyReport(), fix mount option name. 2014-07-09 20:01:53 +02:00
antirez 54ca682887 createLatencyReport(): initialize all advices to zero. 2014-07-09 20:01:48 +02:00
antirez 95d880a7bf LATENCY DOCTOR first implementation complete. 2014-07-09 20:01:44 +02:00
antirez 2e581217d9 Fix analyzeLatencyForEvent() MAD computation. 2014-07-09 20:01:41 +02:00
antirez 121a9a3579 LATENCY DOCTOR: initial draft and events summary output. 2014-07-09 20:01:37 +02:00
antirez 3dd113bf33 Latency: low level time series analysis implemented. 2014-07-09 20:01:33 +02:00
antirez 3e8f5113fa LATENCY SAMPLES renamed LATENCY HISTORY. 2014-07-09 20:01:30 +02:00
antirez c145854bac LATENCY RESET implemented. 2014-07-09 20:01:27 +02:00
antirez 65dbb3546a redis-benchmark: abort when all clients are disconnected. 2014-07-09 20:01:18 +02:00
antirez 4c68319370 Use fsync instead of aof_fsync in final AOF sync. 
This happens in the child process so we don't care about latency:
better to sync metadata as well.
 2014-07-09 20:01:01 +02:00
antirez f3eab7a9b4 Fixed conditional for aof-write-pending-fsync latency event selection. 2014-07-09 20:00:57 +02:00
antirez f3c4111edd Dependencies updated. 2014-07-09 20:00:50 +02:00
antirez 07b4d33734 Cast void* to char* to avoid waring in latencyCommand(). 2014-07-09 19:58:06 +02:00
antirez 58fb90fdac Properly initialize min/max in latency.c. 2014-07-09 19:57:59 +02:00
antirez c005315a64 latencyStartMonitor() modified to avoid warnings. 2014-07-09 19:57:53 +02:00
antirez e81b51ad8c Latency monitor: specialize delayed aof writes events. 2014-07-09 19:57:48 +02:00
antirez 8cc912db43 LATENCY GRAPH: filling under the curve is more readable. 2014-07-09 19:57:42 +02:00
antirez e905588062 LATENCY GRAPH implemented. 2014-07-09 19:57:37 +02:00
antirez b38996aab1 latencyTimeSeries structure max field type fixed. 2014-07-09 19:57:34 +02:00
antirez 062564c261 Free labels in freeSparklineSequence(). 2014-07-09 19:57:28 +02:00
antirez d20d89e630 LATENCY LATEST: add the max field. 2014-07-09 19:57:22 +02:00
antirez dc1c91a8bf Latency monitor trheshold value is now configurable. 
This commit adds both support for redis.conf and CONFIG SET/GET.
 2014-07-09 19:57:17 +02:00
antirez 350792bd72 ASCII sparklines generation API. 2014-07-09 19:57:08 +02:00
antirez ed95915d33 License added to latency.h. 2014-07-09 19:56:47 +02:00
antirez 34afc91030 Latency monitor turned off by default. 
It is not a good idea to bloat the code with gettimeofday() calls if the
instance is working well, and turning monitoring on at runtime is a
joke.
 2014-07-09 19:56:43 +02:00
antirez 35a62f7408 Latency monitor: more hooks around the code. 2014-07-09 19:56:43 +02:00
antirez 855c97ce1b Latency monitor: don't add new samples in the same second. 
Instead we update the old sample with the new latency if it is greater.
 2014-07-09 19:56:35 +02:00
antirez 3cb03b68a4 LATENCY LATEST implemented. 2014-07-09 19:56:30 +02:00
antirez e400ae60f5 Latency monitor: command duration is in useconds. Convert. 2014-07-09 19:04:02 +02:00
antirez a57eb3272d LATENCY SAMPLES implemented. 2014-07-09 19:03:59 +02:00
antirez 72c84acc99 Latency monitor: collect slow commands. 
We introduce the distinction between slow and fast commands since those
are two different sources of latency. An O(1) or O(log N) command without
side effects (can't trigger deletion of large objects as a side effect of
its execution) if delayed is a symptom of inherent latency of the system.

A non-fast command (commands that may run large O(N) computations) if
delayed may just mean that the user is executing slow operations.

The advices LATENCY should provide in this two different cases are
different, so we log the two classes of commands in a separated way.
 2014-07-09 19:03:26 +02:00
antirez 58ecc0bd68 Latency monitor: basic samples collection. 2014-07-09 18:57:57 +02:00
antirez 0baab66d16 Fixed 2.8 build breakage introduced back porting COMMAND. 2014-07-02 22:42:00 +02:00
antirez 670cf9598e Fix Solaris compilation due to ctime_r() call. 
Introduced in Redis 2.8.10 because of a change in Sentinel.
This closes issue #1837.
 2014-06-30 16:29:14 +02:00
antirez 41d7c90417 Test: fix instances.tcl restart_instance abort error. 2014-06-30 12:06:24 +02:00
antirez 63feb93a9f COMMAND COUNT subcommand added. 2014-06-27 18:38:08 +02:00
antirez 0a6649a7c0 COMMAND: fix argument parsing. 
This fixes detection of wrong subcommand (that resulted in the default
all-commands output instead) and allows COMMAND INFO to be called
without arguments (resulting into an empty array) which is useful in
programmtically generated calls like the following (in Ruby):

    redis.commands("command","info",*mycommands)

Note: mycommands may be empty.
 2014-06-27 18:38:08 +02:00
antirez 53377f8c6d COMMANDS command renamed COMMAND. 2014-06-27 18:38:03 +02:00
antirez f3efd529df COMMANDS command: remove static + aesthetic changes. 
Static was removed since it is needed in order to get symbols in stack
traces. Minor changes in the source code were operated to make it more
similar to the existing Redis code base.
 2014-06-27 18:37:34 +02:00
Matt Stancliff fdc5dbd5b1 Cluster: Add COMMANDS command 
COMMANDS returns a nested multibulk reply for each
command in the command table.  The reply for each
command contains:
  - command name
  - arity
  - array of command flags
  - start key position
  - end key position
  - key offset step
  - optional: if the keys are not deterministic and
    Redis uses an internal key evaluation function,
    the 6th field appears and is defined as a status
    reply of: REQUIRES ARGUMENT PARSING

Cluster clients need to know where the keys are in each
command to implement proper routing to cluster nodes.

Redis commands can have multiple keys, keys at offset steps, or other
issues where you can't always assume the first element after
the command name is the cluster routing key.

Using the information exposed by COMMANDS, client implementations
can have live, accurate key extraction details for all commands.

Also implements COMMANDS INFO [commands...] to return only a
specific set of commands instead of all 160+ commands live in Redis.
 2014-06-27 18:37:29 +02:00
antirez afe949efcb Remove infinite loop from PSYNC test. 
Added for debugging and forgot there.
 2014-06-27 18:36:59 +02:00
antirez 1066fed769 Test: hopefully more robust PSYNC test. 
This is supposed to fix issue #1417, but we'll know if this is enough
only after a couple of runs of the CI test without false positives.
 2014-06-27 18:36:59 +02:00
Matt Stancliff cd4f0dc775 Allow __powerpc__ to define HAVE_ATOMIC too 
From mailing list post https://groups.google.com/forum/#!topic/redis-db/D3k7KmJmYgM

In the file “config.h”, the definition HAVE_ATOMIC is used to indicate
if an architecture on which redis is implemented supports atomic
synchronization primitives.  Powerpc  supports atomic synchronization
primitives, however, it is not listed as one of the architectures
supported in config.h. This patch  adds the __powerpc__ to the list of
architectures supporting these primitives. The improvement of redis
due to the atomic synchronization on powerpc is significant,
around 30% to 40%, over the default implementation using pthreads.

This proposal adds __powerpc__ to the list of architectures designated
to support atomic builtins.
 2014-06-27 18:36:59 +02:00
Matt Stancliff 58fea46976 Allow atomic memory count update with C11 builtins 
From mailing list post https://groups.google.com/forum/#!topic/redis-db/QLjiQe4D7LA

In zmalloc.c the following primitives are currently used
to synchronize access to single global variable:
__sync_add_and_fetch
__sync_sub_and_fetch

In some architectures such as powerpc these primitives are overhead
intensive. More efficient C11 __atomic builtins are available with
newer GCC versions, see
http://gcc.gnu.org/onlinedocs/gcc-4.8.2/gcc/_005f_005fatomic-Builtins.html#_005f_005fatomic-Builtins

By substituting the following  __atomic… builtins:
__atomic_add_fetch
__atomic_sub_fetch

the performance improvement on certain architectures such as powerpc can be significant,
around 10% to 15%, over the implementation using __sync builtins while there is only slight uptick on
Intel architectures because it was already enforcing Intel Strongly ordered memory semantics.

The selection of __atomic built-ins can be predicated on the definition of ATOMIC_RELAXED
which Is available on in gcc 4.8.2 and later versions.
 2014-06-27 18:36:59 +02:00
Matt Stancliff ff3ca17bac Use predefined macro for used_memory() update 2014-06-27 18:36:59 +02:00
antirez 04749ee015 Fixed assert conditional in ROLE command test. 2014-06-26 22:13:57 +02:00
antirez 60ff8095d6 No more trailing spaces in Redis source code. 2014-06-26 18:52:16 +02:00
antirez e9cd75dcd1 CLIENT KILL: don't kill the master as a normal client. 
Technically the problem is due to the client type API that does not
return a special value for the master, however fixing it locally in the
CLIENT KILL command is better currently because otherwise we would
introduce a new output buffer limit class as a side effect.
 2014-06-26 18:43:42 +02:00
antirez ac6b656fd6 Old form of CLIENT KILL should still allow suicide. 2014-06-24 12:49:18 +02:00
antirez 32ebb3e651 Redis 2.8.12. 2014-06-23 16:56:18 +02:00
antirez 5d16a838db Sentinel test: more correct sentinels config reset. 
In the initialization test for each instance we used to unregister the
old master and register it again to clear the config.
However there is a race condition doing this: as soon as we unregister
and re-register "mymaster", another Sentinel can update the new
configuration with the old state because of gossip "hello" messages.

So the correct procedure is instead, unregister "mymaster" from all the
sentinel instances, and re-register it everywhere again.
 2014-06-23 14:04:09 +02:00
antirez d3c2918434 Sentinel implementation of ROLE. 2014-06-23 12:07:55 +02:00
antirez 95f492ee57 Silence different signs comparison warning in sds.c. 2014-06-23 11:53:50 +02:00
antirez 364464e790 Use Redis updated sds.c for deps/hiredis. 2014-06-23 11:53:45 +02:00
Matt Stancliff 8066334986 Sentinel: bind source address 
Some deployments need traffic sent from a specific address.  This
change uses the same policy as Cluster where the first listed bindaddr
becomes the source address for outgoing Sentinel communication.

Fixes #1667
 2014-06-23 11:53:40 +02:00
Matt Stancliff e59eef246b Add REDIS_BIND_ADDR access macro 
We need to access (bindaddr[0] || NULL) in a few places, so centralize
access with a nice macro.
 2014-06-23 11:53:25 +02:00
antirez a8ecd3f040 Dummy zmalloc.h restored into deps/hiredis. 2014-06-23 11:53:15 +02:00
Matt Stancliff 6ee25dc439 hiredis: Update to latest version 
This is hiredis f225c276be7fd0646019b51023e3f41566633dfe

This update includes all changes that diverged inside of Redis since
the last update.  This version also allows optional source address
binding for connections which we need for some Sentinel deployments.
 2014-06-23 11:53:12 +02:00
Matt Stancliff ed705f4789 Cancel SHUTDOWN if initial AOF is being written 
Fixes #1826 (and many other reports of the same problem)
 2014-06-23 10:09:25 +02:00
antirez 9a5408f23e Basic tests for the ROLE command. 2014-06-23 09:09:02 +02:00
antirez 061fd99767 Test: AOF test false positive when running in slow hosts. 
The bug was triggered by running the test with Valgrind (which is a lot
slower and more sensible to timing issues) after the recent changes
that made Redis more promptly able to reply with the -LOADING error.
 2014-06-21 16:04:20 +02:00
antirez 2b805ce119 Allow to call ROLE in LOADING state. 2014-06-21 15:29:12 +02:00
antirez 8c460a2872 ROLE command: array len fixed for slave output. 2014-06-21 15:29:11 +02:00
antirez 7d0992dad6 Sentinel: send SLAVEOF with MULTI, CLIENT KILL, CONFIG REWRITE. 
This implements the new Sentinel-Client protocol for the Sentinel part:
now instances are reconfigured using a transaction that ensures that the
config is rewritten in the target instance, and that clients lose the
connection with the instance, in order to be forced to: ask Sentinel,
reconnect to the instance, and verify the instance role with the new
ROLE command.
 2014-06-21 15:27:57 +02:00
antirez 674194ad42 CLIENT KILL API modified. 
Added a new SKIPME option that is true by default, that prevents the
client sending the command to be killed, unless SKIPME NO is sent.
 2014-06-21 15:27:57 +02:00
antirez 61d9a73d87 CLIENT KILL: fix closing link of the current client. 2014-06-21 15:27:57 +02:00
antirez 09dc6dadc1 New features for CLIENT KILL. 2014-06-21 15:27:57 +02:00
antirez cad13223f0 Assign an unique non-repeating ID to each new client. 
This will be used by CLIENT KILL and is also a good way to ensure a
given client is still the same across CLIENT LIST calls.

The output of CLIENT LIST was modified to include the new ID, but this
change is considered to be backward compatible as the API does not imply
you can do positional parsing, since each filed as a different name.
 2014-06-21 15:27:53 +02:00
antirez b6a26b52bf Client types generalized. 
Because of output buffer limits Redis internals had this idea of type of
clients: normal, pubsub, slave. It is possible to set different output
buffer limits for the three kinds of clients.

However all the macros and API were named after output buffer limit
classes, while the idea of a client type is a generic one that can be
reused.

This commit does two things:

1) Rename the API and defines with more general names.
2) Change the class of clients executing the MONITOR command from "slave"
   to "normal".

"2" is a good idea because you want to have very special settings for
slaves, that are not a good idea for MONITOR clients that are instead
normal clients even if they are conceptually slave-alike (since it is a
push protocol).

The backward-compatibility breakage resulting from "2" is considered to
be minimal to care, since MONITOR is a debugging command, and because
anyway this change is not going to break the format or the behavior, but
just when a connection is closed on big output buffer issues.
 2014-06-21 15:27:12 +02:00
antirez 8060de98f8 ROLE output improved for slaves. 
Info about the replication state with the master added.
 2014-06-21 15:27:12 +02:00
antirez 41a15205b6 ROLE command added. 
The new ROLE command is designed in order to provide a client with
informations about the replication in a fast and easy to use way
compared to the INFO command where the same information is also
available.
 2014-06-21 15:27:12 +02:00
antirez d8d415e717 CLIENT LIST speedup via peerid caching + smart allocation. 
This commit adds peer ID caching in the client structure plus an API
change and the use of sdsMakeRoomFor() in order to improve the
reallocation pattern to generate the CLIENT LIST output.

Both the changes account for a very significant speedup.
 2014-06-21 15:27:08 +02:00
antirez 52189cb9c3 Use sdscatfmt() in getClientInfoString() to make it faster. 2014-06-21 15:27:02 +02:00
antirez 4acc3daa92 Added new sdscatfmt() %u and %U format specifiers. 
This commit also fixes a bug in the implementation of sdscatfmt()
resulting from stale references to the SDS string header after
sdsMakeRoomFor() calls.
 2014-06-21 15:26:58 +02:00
antirez 3a915ace44 sdscatfmt() added to SDS library. 
sdscatprintf() relies on printf() family libc functions and is sometimes
too slow in critical code paths. sdscatfmt() is an alternative which is:

1) Far less capable.
2) Format specifier uncompatible.
3) Faster.

It is suitable to be used in those speed critical code paths such as
CLIENT LIST output generation.
 2014-06-21 15:26:54 +02:00
antirez ebc62a6f35 Jemalloc updated to 3.6.0. 
Not a single bug in about 3 months, and our previous version was
too old (3.2.0).
 2014-06-20 15:01:36 +02:00
antirez a0e0fba7fa Jemalloc update script added. 2014-06-20 14:53:47 +02:00
antirez 2355ae9bc6 Sentinel test: tolerate larger delays in init tests. 2014-06-19 16:10:20 +02:00
antirez fd2faee5de Sentinel test: unit 02, avoid some time related false positives. 2014-06-19 16:10:17 +02:00
antirez 93ee0f2666 Sentinel: send hello messages ASAP after config change. 
Eventual configuration convergence is guaranteed by our periodic hello
messages to all the instances, however when there are important notices
to share, better make a phone call. With this commit we force an hello
message to other Sentinal and Redis instances within the next 100
milliseconds of a config update, which is practically better than
waiting a few seconds.
 2014-06-19 15:25:57 +02:00
antirez 9b883974af Sentinel: handle SRI_PROMOTED flag correctly. 
Lack of check of the SRI_PROMOTED flag caused Sentienl to act with the
promoted slave turned into a master during failover like if it was a
normal instance.

Normally this problem was not apparent because during real failovers the
old master is down so the bugged code path was not entered, however with
manual failovers via the SENTINEL FAILOVER command, the problem was
easily triggered.

This commit prevents promoted slaves from getting reconfigured, moreover
we now explicitly check that during a failover the slave turning into a
master is the one we selected for promotion and not a different one.
 2014-06-19 10:35:03 +02:00
antirez 7665cc67b3 Sentinel test: add manual failover test. 2014-06-19 10:32:38 +02:00
antirez 1267e28edc Test: use higher level redis.tcl proc to read replies. 2014-06-18 16:25:45 +02:00
antirez 2cff0c041f Test: Tcl client initial support for automatic reconnection. 2014-06-18 16:19:22 +02:00
antirez 9a39db3bb6 Test: tcl client, unset deferred var on close. 2014-06-18 16:19:19 +02:00
antirez b0964b7dd9 redis.tcl: return I/O error message when peer closes connection. 2014-06-18 16:18:45 +02:00
antirez eef369b66a Sentinel framework new layout back ported 2.8. 
Functionally the same, but makes cherry picking simpler.
 2014-06-18 16:14:33 +02:00
Matt Stancliff f392c7737c Add correct exit value to failed tests 
(Note from @antirez: modified to apply to 2.8)
 2014-06-18 15:03:06 +02:00
Alex Suraci d178cd03ed add missing signal.h include 2014-06-18 09:34:23 +02:00
Matt Stancliff 1b044af402 Add SIGINT handler to cli --intrinsic-latency 
If we run a long latency session and want to Ctrl-C out of it,
it's nice to still get the summary output at the end.
 2014-06-17 23:40:05 +02:00
antirez 258f3cd360 Scripting: regression test for issue #1811. 2014-06-16 10:30:42 +02:00
antirez ca6b95dfdd Fix semantics of Lua calls to SELECT. 
Lua scripts are executed in the context of the currently selected
database (as selected by the caller of the script).

However Lua scripts are also free to use the SELECT command in order to
affect other DBs. When SELECT is called frm Lua, the old behavior, before
this commit, was to automatically set the Lua caller selected DB to the
last DB selected by Lua. See for example the following sequence of
commands:

    SELECT 0
    SET x 10
    EVAL "redis.call('select','1')" 0
    SET x 20

Before this commit after the execution of this sequence of commands,
we'll have x=10 in DB 0, and x=20 in DB 1.

Because of the problem above, there was a bug affecting replication of
Lua scripts, because of the actual implementation of replication. It was
possible to fix the implementation of Lua scripts in order to fix the
issue, but looking closely, the bug is the consequence of the behavior
of Lua ability to set the caller's DB.

Under the old semantics, a script selecting a different DB, has no simple
ways to restore the state and select back the previously selected DB.
Moreover the script auhtor must remember that the restore is needed,
otherwise the new commands executed by the caller, will be executed in
the context of a different DB.

So this commit fixes both the replication issue, and this hard-to-use
semantics, by removing the ability of Lua, after the script execution,
to force the caller to switch to the DB selected by the Lua script.

The new behavior of the previous sequence of commadns is to just set
X=20 in DB 0. However Lua scripts are still capable of writing / reading
from different DBs if needed.

WARNING: This is a semantical change that will break programs that are
conceived to select the client selected DB via Lua scripts.

This fixes issue #1811.
 2014-06-16 10:30:37 +02:00
antirez b0f3922383 Redis 2.8.11. 2014-06-11 10:30:05 +02:00
antirez a3b0dbccf5 Scripting: Fix for a #1118 regression simplified. 
It is more straightforward to just test for a numerical type avoiding
Lua's automatic conversion. The code is technically more correct now,
however Lua should automatically convert to number only if the original
type is a string that "looks like a number", and not from other types,
so practically speaking the fix is identical AFAIK.
 2014-06-11 10:14:26 +02:00
Matt Stancliff ba76daa42e Scripting: Fix regression from #1118 
The new check-for-number behavior of Lua arguments broke
users who use large strings of just integers.

The Lua number check would convert the string to a number, but
that breaks user data because
Lua numbers have limited precision compared to an arbitrarily
precise number wrapped in a string.

Regression fixed and new test added.

Fixes #1118 again.
 2014-06-11 10:14:26 +02:00
Matt Stancliff 8f774ee7ca Fix lack of strtold under Cygwin 
Renaming strtold to strtod then casting
the result is the standard way of dealing with
no strtold in Cygwin.
 2014-06-09 11:43:19 +02:00
Matt Stancliff 2af2414344 Fix lack of SA_ONSTACK under Cygwin 
Fixes #232
 2014-06-09 11:43:19 +02:00
Matt Stancliff b4f9761d85 Fix blocking operations from missing new lists 
Behrad Zari discovered [1] and Josiah reported [2]: if you block
and wait for a list to exist, but the list creates from
a non-push command, the blocked client never gets notified.

This commit adds notification of blocked clients into
the DB layer and away from individual commands.

Lists can be created by [LR]PUSH, SORT..STORE, RENAME, MOVE,
and RESTORE.  Previously, blocked client notifications were
only triggered by [LR]PUSH.  Your client would never get
notified if a list were created by SORT..STORE or RENAME or
a RESTORE, etc.

Blocked client notification now happens in one unified place:
  - dbAdd() triggers notification when adding a list to the DB

Two new tests are added that fail prior to this commit.

All test pass.

Fixes #1668

[1]: https://groups.google.com/forum/#!topic/redis-db/k4oWfMkN1NU
[2]: #1668
 2014-06-09 11:39:44 +02:00
Andy Grunwald 6ab4775a2a Fixed typo in word avarege in result message of --intrinsic-latency analyzer 2014-06-06 11:20:04 +02:00
Jan-Erik Rediger b98ee12f61 Small typo fixed 2014-06-06 10:45:46 +02:00
yoav 3783d90e53 Fix eval usage in tests to conform with eval semantics 2014-06-06 10:44:08 +02:00
zionwu e3112bed8d fix issue 1787 2014-06-06 10:38:04 +02:00
antirez 987a47b823 Redis 2.8.10. 2014-06-05 11:02:23 +02:00
antirez 430bec3461 Don't process min-slaves-to-write for slaves. 
Replication is totally broken when a slave has this option, since it
stops accepting updates from masters.

This fixes issue #1434.
 2014-06-05 10:50:31 +02:00
antirez 3a30be51aa Tests for min-slaves-* feature. 2014-06-05 10:50:31 +02:00
antirez 42504169c7 Fixed dbuf variable scope in luaRedisGenericCommand(). 
I'm not sure if while the visibility is the inner block, the fact we
point to 'dbuf' is a problem or not, probably the stack var isx
guaranteed to live until the function returns. However obvious code is
better anyway.
 2014-06-04 18:58:58 +02:00
antirez 26359d570b Regression test for issue #1118. 2014-06-04 18:51:31 +02:00
antirez 768994b683 Scripting: better Lua number -> string conversion in luaRedisGenericCommand(). 
The lua_to*string() family of functions use a non optimal format
specifier when converting integers to strings. This has both the problem
of the number being converted in exponential notation, which we don't
use as a Redis return value when floating point numbers are involed,
and, moreover, there is a loss of precision since the default format
specifier is not able to represent numbers that must be represented
exactly in the IEEE 754 number mantissa.

The new code handles it as a special case using a saner conversion.

This fixes issue #1118.
 2014-06-04 18:45:49 +02:00
antirez 1c0c42e668 More trailing spaces in sentinel.c removed. 2014-05-28 15:46:14 +02:00
Matt Stancliff 75266018ee Disable recursive watchdog signal handler 
If we are in the signal handler, we don't want to handle
the signal again.  In extreme cases, this can cause a stack overflow
and segfault Redis.

Fixes #1771
 2014-05-26 17:53:40 +02:00
antirez edc41baca5 Sentinel example config: explain you don't need to specify slaves. 2014-05-26 10:18:07 +02:00
antirez 1a14eba2d5 Test: fixed scripting.tcl test false positive. 2014-05-22 15:46:06 +02:00
antirez f482349707 Process events with processEventsWhileBlocked() when blocked. 
When we are blocked and a few events a processed from time to time, it
is smarter to call the event handler a few times in order to handle the
accept, read, write, close cycle of a client in a single pass, otherwise
there is too much latency added for clients to receive a reply while the
server is busy in some way (for example during the DB loading).
 2014-05-22 15:46:02 +02:00
antirez f3d3c606f1 Accept multiple clients per iteration. 
When the listening sockets readable event is fired, we have the chance
to accept multiple clients instead of accepting a single one. This makes
Redis more responsive when there is a mass-connect event (for example
after the server startup), and in workloads where a connect-disconnect
pattern is used often, so that multiple clients are waiting to be
accepted continuously.

As a side effect, this commit makes the LOADING, BUSY, and similar
errors much faster to deliver to the client, making Redis more
responsive when there is to return errors to inform the clients that the
server is blocked in an not interruptible operation.
 2014-05-22 15:45:58 +02:00
antirez 4113473997 While ANET_ERR is -1, check syscall retval for -1 itself. 2014-05-22 15:45:54 +02:00
antirez e8669ca8a7 Regression test for issue #1764. 2014-05-20 16:20:45 +02:00
michael-grunder d491a479bc Fix LUA_OBJCACHE segfault. 
When scanning the argument list inside of a redis.call() invocation
for pre-cached values, there was no check being done that the
argument we were on was in fact within the bounds of the cache size.

So if a redis.call() command was ever executed with more than 32
arguments (current cache size #define setting) redis-server could
segfault.
 2014-05-20 16:20:38 +02:00
antirez d575f7a147 Remove trailing spaces from scripting.c 2014-05-20 16:11:28 +02:00
antirez 6d2fddd2a3 Remove trailing spaces from sentinel.c. 2014-05-20 14:22:34 +02:00
antirez ad935769ae HyperLogLog regression test for issue #1762. 2014-05-19 15:45:24 +02:00
Mike Trinkala 43278af63e Correct the HyperLogLog stale cache flag to prevent unnecessary computations. 
Set the MSB as documented.
 2014-05-19 15:45:24 +02:00
antirez 3203e2dcdd Fix 2.8 backport of fastscript branch. 
No REDIS_ENCODING_EMBSTR in 2.8 internals.
 2014-05-14 18:39:42 +02:00
Akos Vandra 6e478ec890 Fixed possible buffer overflow bug if RDB file is corrupted. 
(Note: commit message modified by @antirez for clarity).
 2014-05-12 11:49:16 +02:00
Akos Vandra c0eaac9572 fixed possible buffer overflow error 2014-05-12 11:19:36 +02:00
antirez 23011dea1f Sentinel: Add "dir /tmp" directive in example sentinel.conf. 2014-05-12 10:46:56 +02:00
antirez 6ae4c1fcbc DEBUG POPULATE: call dictExpand() to avoid useless rehashing. 2014-05-09 15:05:31 +02:00
antirez 13d8b2b096 Sentinel: log when a failover will be attempted again. 
When a Sentinel performs a failover (successful or not), or when a
Sentinel votes for a different Sentinel trying to start a failover, it
sets a min delay before it will try to get elected for a failover.

While not strictly needed, because if multiple Sentinels will try
to failover the same master at the same time, only one configuration
will eventually win, this serialization is practically very useful.
Normal failovers are cleaner: one Sentinel starts to failover, the
others update their config when the Sentinel performing the failover
is able to get the selected slave to move from the role of slave to the
one of master.

However currently this timeout was implicit, so users could see
Sentinels not reacting, after a failed failover, for some time, without
giving any feedback in the logs to the poor sysadmin waiting for clues.

This commit makes Sentinels more verbose about the delay: when a master
is down and a failover attempt is not performed because the delay has
still not elaped, something like that will be logged:

    Next failover delay: I will not start a failover
    before Thu May  8 16:48:59 2014
 2014-05-08 16:49:45 +02:00
antirez 909d1883e7 Sentinel: generate +config-update-from event when a new config is received. 
This event makes clear, before the switch-master event is generated,
that a Sentinel received a configuration update from another Sentinel.
 2014-05-08 16:49:45 +02:00
antirez 80b96bc4a7 Scripting test: check that Lua can call commands rewirting argv. 
SPOP, tested in the new test, is among the commands rewritng the
client->argv argument vector (it gets rewritten as SREM) for command
replication purposes.

Because of recent optimizations to client->argv caching in the context
of the Lua internal Redis client, it is important to test for SPOP to be
callable from Lua without bad effects to the other commands.
 2014-05-07 16:17:11 +02:00
antirez 093eab56de Test: handle new osx 'leaks' error. 
Sometimes the process is still there but no longer in a state that can
be checked (after being killed). This used to happen after a call to
SHUTDOWN NOSAVE in the scripting unit, causing a false positive.
 2014-05-07 16:17:11 +02:00
antirez 32971f6251 Scripting: objects caching for Lua c->argv creation. 
Reusing small objects when possible is a major speedup under certain
conditions, since it is able to avoid the malloc/free pattern that
otherwise is performed for every argument in the client command vector.
 2014-05-07 16:17:11 +02:00
antirez 87516f1a4d Scripting: Use faster API for Lua client c->argv creation. 
Replace the three calls to Lua API lua_tostring, lua_lua_strlen,
and lua_isstring, with a single call to lua_tolstring.

~ 5% consistent speed gain measured.
 2014-05-07 16:17:11 +02:00
antirez 1b5524f697 Scripting: don't call lua_gc() after Lua script run. 
Calling lua_gc() after every script execution is too expensive, and
apparently does not make the execution smoother: the same peak latency
was measured before and after the commit.

This change accounts for scripts execution speedup in the order of 10%.
 2014-05-07 16:17:11 +02:00
antirez 8237d88f48 Scripting: cache argv in luaRedisGenericCommand(). 
~ 4% consistently measured speed improvement.
 2014-05-07 16:17:11 +02:00
antirez 6938a3d58f Fixed missing c->bufpos reset in luaRedisGenericCommand(). 
Bug introduced when adding a fast path to avoid copying the reply buffer
for small replies that fit into the client static buffer.
 2014-05-07 16:17:11 +02:00
antirez 08ea98c9e9 Scripting: replace tolower() with faster code in evalGenericCommand(). 
The function showed up consuming a non trivial amount of time in the
profiler output. After this change benchmarking gives a 6% speed
improvement that can be consistently measured.
 2014-05-07 16:17:11 +02:00
antirez 3a1a0c3240 Scripting: luaRedisGenericCommand() fast path for buffer-only replies. 
When the reply is only contained in the client static output buffer, use
a fast path avoiding the dynamic allocation of an SDS string to
concatenate the client reply objects.
 2014-05-07 16:17:11 +02:00
antirez d70281575c Define HAVE_ATOMIC for clang. 2014-05-07 16:17:11 +02:00
antirez 67e350367d Scripting: simpler reply buffer creation in luaRedisGenericCommand(). 
It if faster to just create the string with a single sdsnewlen() call.
If c->bufpos is zero, the call will simply be like sdsemtpy().
 2014-05-07 16:17:11 +02:00
Matt Stancliff b06de30dd7 Add test for deleting an expired key 
Verify proper expire-before-delete behavior.

This test passes with the expire-before-delete commit and fails
without it.
 2014-04-23 16:14:40 +02:00
Matt Stancliff e9d0d86dbf Check key expiration before deleting 
Deleting an expired key should return 0, not success.

Fixes #1648
 2014-04-23 16:14:40 +02:00
Glauber Costa 96b5dc0b98 fix null pointer access with no file pointer 
I happen to be working on a system that lacks urandom. While the code does try
to handle this case and artificially create some bytes if the file pointer is
empty, it does try to close it unconditionally, leading to a segfault.
 2014-04-23 15:35:12 +02:00
Salvatore Sanfilippo 3d2af57974 Merge pull request #1700 from nirvdrum/patch-1 
Fixed typos.
 2014-04-23 11:55:50 +02:00
antirez 35252037fd Missing return REDIS_ERR added to processMultibulkBuffer(). 
When we set a protocol error we should return with REDIS_ERR to let the
caller know it should stop processing the client.

Bug found in a code auditing related to issue #1699.
 2014-04-23 10:21:26 +02:00
Kevin Menard 662f505727 Fixed typos. 2014-04-22 10:51:19 -04:00
antirez de7ebf83e1 redis-cli help.h updated. 2014-04-22 16:15:01 +02:00
antirez 91dcc5b12f generate-command-help.rb updated with new hyperloglog group. 2014-04-22 16:15:01 +02:00
antirez 26375709c5 Redis 2.8.9. 2014-04-22 10:14:57 +02:00
antirez c5ee48a9d9 Fuzzy test for ZREMRANGEBYLEX added. 2014-04-18 16:16:20 +02:00
antirez be145453c6 ZREMRANGEBYLEX memory leak removed calling zslFreeLexRange(). 2014-04-18 16:16:20 +02:00
antirez 115782b1d7 PFCOUNT multi-key test added. 2014-04-18 16:16:20 +02:00
antirez 2cd20d3c4c Speedup hllRawSum() processing 8 bytes per iteration. 
The internal HLL raw encoding used by PFCOUNT when merging multiple keys
is aligned to 8 bits (1 byte per register) so we can exploit this to
improve performances by processing multiple bytes per iteration.

In benchmarks the new code was several times faster with HLLs with many
registers set to zero, while no slowdown was observed with populated
HLLs.
 2014-04-18 16:16:20 +02:00
antirez 3f5cd0c660 Speedup SUM(2^-reg[m]) in HyperLogLog computation. 
When the register is set to zero, we need to add 2^-0 to E, which is 1,
but it is faster to just add 'ez' at the end, which is the number of
registers set to zero, a value we need to compute anyway.
 2014-04-18 16:16:20 +02:00
antirez f0eca148ff PFCOUNT support for multi-key union. 2014-04-18 16:16:20 +02:00
antirez d25fd8be64 HyperLogLog low level merge extracted from PFMERGE. 2014-04-18 16:16:20 +02:00
antirez 4a26648a22 ZREMRANGEBYLEX implemented. 2014-04-18 16:16:20 +02:00
antirez 3fc22561e7 Always pass sorted set range objects by reference. 2014-04-18 16:16:04 +02:00
antirez ea0af9ace3 ZREMRANGE* commands refactored into a single generic function. 2014-04-18 16:14:38 +02:00
antirez 019e6f1889 Sorted set lex ranges stress tester. 2014-04-18 16:14:38 +02:00
antirez 9c9aef24a2 Basic ZRANGEBYLEX / ZLEXCOUNT tests. 2014-04-17 00:08:38 +02:00
antirez 11f89497b4 Pass by pointer and release of lex ranges. 
Given that the code was written with a 2 years pause... something
strange happened in the middle. So there was no function to free a
lex range min/max objects, and in some places the range was passed by
value.
 2014-04-17 00:08:37 +02:00
antirez d975bb587f ZLEXCOUNT implemented. 
Like ZCOUNT for lexicographical ranges.
 2014-04-16 15:26:28 +02:00
antirez 71dfb87f76 More HyperLogLog tests. 2014-04-16 15:26:28 +02:00
antirez f2e59eae44 HyperLogLog invalid representation error code set to INVALIDOBJ. 2014-04-16 15:26:28 +02:00
antirez 469b7c5168 PFDEBUG TODENSE added. 
Converts HyperLogLogs from sparse to dense. Used for testing.
 2014-04-16 15:26:28 +02:00
antirez 614fcd491e User-defined switch point between sparse-dense HLL encodings. 2014-04-16 15:26:28 +02:00
antirez 08da2b79fc PFSELFTEST improved with sparse encoding checks. 2014-04-16 15:26:27 +02:00
antirez 5a0bab2a8f PFDEBUG ENCODING added. 2014-04-16 15:26:27 +02:00
antirez 05563310f4 Set HLL_SPARSE_MAX to 3000. 
After running a few benchmarks, 3000 looks like a reasonable value to
keep HLLs with a few thousand elements small while the CPU cost is
still not huge.

This covers all the cases where the dense representation would use N
orders of magnitude more space, like in the case of many HLLs with
carinality of a few tens or hundreds.

It is not impossible that in the future this gets user configurable,
however it is easy to pick an unreasoable value just looking at savings
in the space dimension without checking what happens in the time
dimension.
 2014-04-16 15:26:27 +02:00
antirez e77a3e40f2 Error message for invalid HLL objects unified. 2014-04-16 15:26:27 +02:00
antirez c4b2a7a7f4 PFMERGE fixed to work with sparse encoding. 2014-04-16 15:26:27 +02:00
antirez ee682c4c48 Mark PFDEBUG as write command in the commands table. 
It is safer since it is able to have side effects.
 2014-04-16 15:26:27 +02:00
antirez 9ef8ef410b Correctly replicate PFDEBUG GETREG. 
Even if it is a debugging command, make sure that when it forces a
change in encoding, the command is propagated.
 2014-04-16 15:26:27 +02:00
antirez 91e05e1618 Added assertion in hllSparseAdd() when promotion to dense occurs. 
If we converted to dense, a register must be updated in the dense
representation.
 2014-04-16 15:26:27 +02:00
antirez 68a4bf70e0 hllSparseAdd(): speed optimization. 
Mostly by reordering opcodes check conditional by frequency of opcodes
in larger sparse-encoded HLLs.
 2014-04-16 15:26:27 +02:00
antirez 21e328a6d7 Detect corrupted sparse HLLs in hllSparseSum(). 2014-04-16 15:26:27 +02:00
antirez 4244aa29f9 hllSparseAdd(): faster code removing conditional. 
Bottleneck found profiling. Big run time improvement found when testing
after the change.
 2014-04-16 15:26:27 +02:00
antirez 4cd11077d2 Comment typo in hllSparseAdd(). first -> fits. 2014-04-16 15:26:27 +02:00
antirez 08183cfef5 Merge adjacent VAL opcodes in hllSparseAdd(). 
As more values are added splitting ZERO or XZERO opcodes, try to merge
adjacent VAL opcodes if they have the same value.
 2014-04-16 15:26:27 +02:00
antirez d428569d13 More robust HLL_SPARSE macros protecting 'p' with parens. 
Now the macros will work with arguments such as "ptr+1".
 2014-04-16 15:26:27 +02:00
antirez 2d1e35a58a hllSparseAdd() opcode seek stop condition fixed. 2014-04-16 15:26:27 +02:00
antirez 29c7d1974b Fixed error message generation in PFDEBUG GETREG. 
Bulk length for registers was emitted too early, so if there was a bug
the reply looked like a long array with just one element, blocking the
client as result.
 2014-04-16 15:26:27 +02:00
antirez 023f2f7a10 Fixed memmove() count in hllSparseAdd(). 2014-04-16 15:26:27 +02:00
antirez 91391760ca hllSparseAdd(): more correct dense conversion conditional. 
We want to promote if the total string size exceeds the resulting size
after the upgrade.
 2014-04-16 15:26:27 +02:00
antirez 5d87160f8a hllSparseToDense(): sanity check added. 
The function checks if all the HLL_REGISTERS were processed during the
convertion from sparse to dense encoding, returning REDIS_OK or
REDIS_ERR to signal a corruption problem.

A bug in PFDEBUG GETREG was fixed: when the object is converted to the
dense representation we need to reassign the new pointer to the header
structure pointer.
 2014-04-16 15:26:27 +02:00
antirez a18e7afbe8 PFDEBUG DECODE added. 
Provides a human readable description of the opcodes composing a
run-length encoded HLL (sparse encoding).
The command is only useful for debugging / development tasks.
 2014-04-16 15:26:27 +02:00
antirez 5c86d9e7b0 PFDEBUG added, PFGETREG removed. 
PFDEBUG will be the interface to do debugging tasks with a key
containing an HLL object.
 2014-04-16 15:26:27 +02:00
antirez 0e0ddcc4fa hllSparseToDense API changed to take ref to object. 
The new API takes directly the object doing everything needed to
turn it into a dense representation, including setting the new
representation as object->ptr.
 2014-04-16 15:26:27 +02:00
antirez 533235f7e3 hllSparseAdd() sanity check for span != 0 added. 2014-04-16 15:26:27 +02:00
antirez f12fb49bb2 Fix hllSparseAdd() new sequence replacement when next is NULL. 
sdsIncrLen() must be called anyway even if we are replacing the last
oppcode of the sparse representation.
 2014-04-16 15:26:27 +02:00
antirez a3e2adc0ae Fix seqlen computation in hllSparseAdd(). 2014-04-16 15:26:27 +02:00
antirez eaa73ef73a Abstract hllSparseAdd() / hllDenseAdd() via hllAdd(). 2014-04-16 15:26:27 +02:00
antirez 95de3eaea0 hllSparseSum(): multiply 1 * runlen for zero entries. 2014-04-16 15:26:27 +02:00
antirez c277169ed7 Macro HLL_SPARSE_XZERO_LEN fixed. 2014-04-16 15:26:27 +02:00
antirez b6ca71a40a Fix HLL sparse object creation #2. 
Two vars initialized to wrong values in createHLLObject().
 2014-04-16 15:26:27 +02:00
antirez 8b39fee4c8 Increment pointer while iterating sparse HLL object. 2014-04-16 15:26:27 +02:00
antirez c1ddead779 Fix HLL sparse object creation. 
The function didn't considered the fact that each XZERO opcode is
two bytes.
 2014-04-16 15:26:27 +02:00
antirez 0a7ba6faef Create HyperLogLog objects with sparse encoding. 2014-04-16 15:26:27 +02:00
antirez 8e889728bf HyperLogLog sparse to dense conversion function. 2014-04-16 15:26:27 +02:00
antirez aadd513160 HyperLogLog sparse representation initial implementation. 
Code never tested, but the basic layout is shaped in this commit.
Also missing:

1) Sparse -> Dense conversion function.
2) New HLL object creation using the sparse representation.
3) Implementation of PFMERGE for the sparse representation.
 2014-04-16 15:26:27 +02:00
antirez e5e5facba1 hllCount() refactored to support multiple representations. 2014-04-16 15:26:27 +02:00
antirez 88847a216f hllAdd() refactored into two functions. 
Also dense representation access macro renamed accordingly.
 2014-04-16 15:26:27 +02:00
antirez 21c97e71cd HyperLogLog refactoring to support different encodings. 
Metadata are now placed at the start of the representation as an header.
There is a proper structure to access the representation.
Still work to do in order to truly abstract the implementation from the
representation, commands still work assuming dense representation.
 2014-04-16 15:26:27 +02:00
antirez ed96e115d0 HyperLogLog sparse representation slightly modified. 
After running a few simulations with different alternative encodings,
it was found that the VAL opcode performs better using 5 bits for the
value and 2 bits for the run length, at least for cardinalities in the
range of interest.
 2014-04-16 15:26:27 +02:00
antirez 93c75f1bf4 HyperLogLog sparse representation description and macros. 2014-04-16 15:26:26 +02:00
antirez 2b1385b9ec Add casting to match printf format. 
adjustOpenFilesLimit() and clusterUpdateSlotsWithConfig() that were
assuming uint64_t is the same as unsigned long long, which is true
probably for all the systems out there that we target, but still GCC
emitted a warning since technically they are two different types.
 2014-04-16 15:26:22 +02:00
antirez 12d1d18b59 ZRANGEBYLEX and ZREVRANGEBYLEX implementation. 2014-04-16 15:26:09 +02:00
antirez a10bfded15 PFCOUNT: always unshare/decode the object. 
This will be a non-op most of the times since the object will be
unshared / decoded, however it is more technically correct to start this
way since the object may be decoded even in the read-only code path.
 2014-04-16 15:26:09 +02:00
antirez ee764b0f8d Changed HyperLogLog hash seed to a non-zero value. 
Using a seed of zero has the side effect of having the empty string
hashing to what is a very special case in the context of HyperLogLog: a
very long run of zeroes.

This did not influenced the correctness of the result with 16k registers
because of the harmonic mean, but still it is inconvenient that a so
obvious value maps to a so special hash.

The seed 0xadc83b19 is used instead, which is the first 64 bits of the
SHA1 of the empty string.

Reference: issue #1657.
 2014-04-16 15:26:09 +02:00
antirez 091f5677a0 Initial HyperLogLog tests. 2014-04-16 15:26:09 +02:00
antirez a97675923d Return "WRONGTYPE" error on PF* type mismatch. 2014-04-16 15:26:09 +02:00
antirez faa7f259cc Fix PFADD infinite loop. 
We need to guarantee that the last bit is 1, otherwise an element may
hash to just zeroes with probability 1/(2^64) and trigger an infinite
loop.

See issue #1657.
 2014-04-16 15:26:09 +02:00
antirez 8ae6887091 Make hll-gnuplot-graph.rb callable from cli. 2014-04-16 15:26:09 +02:00
antirez bdd8701c60 Remove HyperLogLog type checking duplicated code. 2014-04-16 15:26:09 +02:00
antirez b3baa51403 PFGETREG added for testing purposes. 
The new command allows to get a dump of the registers stored
into an HyperLogLog data structure for testing / debugging purposes.
 2014-04-16 15:26:09 +02:00
antirez 348d5ea246 PFCOUNT: unshare the object when cached cardinality is modified. 2014-04-16 15:26:09 +02:00
antirez ba0dc0be2c PFSELFTEST improved to test the approximation error. 2014-04-16 15:26:09 +02:00
antirez c2cd48718d hll-gnuplot-graph.rb improved with new filter. 
The function to generate graphs is also more flexible as now includes
step and max value. The step of the samples generation function is no
longer limited to min step of 1000.
 2014-04-16 15:26:09 +02:00
antirez ff551fad24 HyperLogLog: added magic / version. 
This will allow future changes like compressed representations.
Currently the magic is not checked for performance reasons but this may
change in the future, for example if we add new types encoded in strings
that may have the same size of HyperLogLogs.
 2014-04-16 15:26:09 +02:00
Raymond Myers e08100a9ba Fixed pfadd/pfcount commands emitting hll* events instead of pf* events 2014-04-16 15:26:09 +02:00
Raymond Myers 23acae6e1c Change HLL* to PF* in error messages 2014-04-16 15:26:09 +02:00
antirez e3773b0a0e Include redis.h before other stuff in hyperloglog.c. 
Otherwise fmacros.h is included later and this may break compilation on
different systems.
 2014-04-16 15:26:09 +02:00
antirez badf23f57b HyperLogLog API prefix modified from "P" to "PF". 
Using both the initials of Philippe Flajolet instead of just "P".
 2014-04-16 15:26:03 +02:00
antirez 6e71459c73 Makefile.dep updated with hyperloglog.o deps. 2014-04-16 15:25:28 +02:00
antirez 2ee2bec2d6 HyperLogLog: make API use the P prefix in honor of Philippe Flajolet. 2014-04-16 15:24:15 +02:00
antirez 1e90c0066c HLLMERGE fixed by adding a... missing loop! 2014-04-16 15:23:21 +02:00
antirez be7eef911b hll-gnuplot-graph.rb: added new filter "all". 2014-04-16 15:23:21 +02:00
antirez f2e707b679 HyperLogLog apply bias correction using a polynomial. 
Better results can be achieved by compensating for the bias of the raw
approximation just after 2.5m (when LINEARCOUNTING is no longer used) by
using a polynomial that approximates the bias at a given cardinality.

The curve used was found using this web page:

    http://www.xuru.org/rt/PR.asp

That performs polynomial regression given a set of values.
 2014-04-16 15:23:21 +02:00
antirez be7fe2b92b HLLMERGE implemented. 
Merge N HLL data structures by selecting the max value for every
M[i] register among the set of HLLs.
 2014-04-16 15:23:21 +02:00
antirez 39dd1f648d HLLCOUNT is technically a write command 
When we update the cached value, we need to propagate the command and
signal the key as modified for WATCH.
 2014-04-16 15:23:21 +02:00
antirez f07b1514bb HLLADD: propagate write when only variable name is given. 
The following form is given:

    HLLADD myhll

No element is provided in the above case so if 'myhll' var does not
exist the result is to just create an empty HLL structure, and no update
will be performed on the registers.

In this case, the DB should still be set dirty and the command
propagated.
 2014-04-16 15:23:21 +02:00
antirez 32e9996846 hll-gnuplot-graph.rb: Use |error| when filter is :max 2014-04-16 15:23:21 +02:00
antirez 9faa80ee8c Ignore txt files inside utils/hyperloglog. 
Those are generated to trace graphs using gnuplot.
 2014-04-16 15:23:21 +02:00
antirez aad5959ddc HyperLogLog: use LINEARCOUNTING up to 3m. 
The HyperLogLog original paper suggests using LINEARCOUNTING for
cardinalities < 2.5m, however for P=14 the median / max error
curves show that a value of '3' is the best pick for m = 16384.
 2014-04-16 15:23:21 +02:00
antirez d6ce659ffd hll-gnuplot-graph.rb added to plot HyperLogLog error graphs. 2014-04-16 15:23:21 +02:00
antirez c3df5965f3 HyperLogLog approximated cardinality caching. 
The more we add elements to an HyperLogLog counter, the smaller is
the probability that we actually update some register.

From this observation it is easy to see how it is possible to use
caching of a previously computed cardinality and reuse it to serve
HLLCOUNT queries as long as no register was updated in the data
structure.

This commit does exactly this by using just additional 8 bytes for the
data structure to store a 64 bit unsigned integer value cached
cardinality. When the most significant bit of the 64 bit integer is set,
it means that the value computed is no longer usable since at least a
single register was modified and we need to recompute it at the next
call of HLLCOUNT.

The value is always stored in little endian format regardless of the
actual CPU endianess.
 2014-04-16 15:23:21 +02:00
antirez 9e178afae9 String value unsharing refactored into proper function. 
All the Redis functions that need to modify the string value of a key in
a destructive way (APPEND, SETBIT, SETRANGE, ...) require to make the
object unshared (if refcount > 1) and encoded in raw format (if encoding
is not already REDIS_ENCODING_RAW).

This was cut & pasted many times in multiple places of the code. This
commit puts the small logic needed into a function called
dbUnshareStringValue().
 2014-04-16 15:22:56 +02:00
antirez 1c795db9d0 Use endian neutral hash function for HyperLogLog. 
We need to be sure that you can save a dataset in a Redis instance,
reload it in a different architecture, and continue to count in the same
HyperLogLog structure.

So 32 and 64 bit, little or bit endian, must all guarantee to output the
same hash for the same element.
 2014-04-16 15:19:40 +02:00
antirez 7f30998432 HyperLogLog internal representation modified. 
The new representation is more obvious, starting from the LSB of the
first byte and using bits going to MSB, and passing to next byte as
needed.

There was also a subtle error: first two bits were unused, everything
was carried over on the right of two bits, even if it worked because of
the code requirement of always having a byte more at the end.

During the rewrite the code was made safer trying to avoid undefined
behavior due to shifting an uint8_t for more than 8 bits.
 2014-04-16 15:19:40 +02:00
antirez 430d0ade75 Remove a few useless operations from hllCount() fast path. 2014-04-16 15:19:40 +02:00
antirez 83650a72a1 HLLCOUNT 3x faster taking fast path for default params. 2014-04-16 15:19:40 +02:00
antirez 3afd25ecc7 Use processor base types in HLL_(GET|SET)_REGISTER. 
This speedups the macros by a noticeable factor.
 2014-04-16 15:19:40 +02:00
antirez 0ffd5e4a30 HyperLogLog: use precomputed table for 2^(-M[i]). 2014-04-16 15:19:40 +02:00
antirez ca6973d3b6 hll-err.rb: speedup using pipelining. 2014-04-16 15:19:40 +02:00
antirez 2500f2a334 hll-err.rb added to test error rate of Redis HyperLogLog. 2014-04-16 15:19:40 +02:00
antirez 42592a7faa HyperLogLog algorithm fixed in two ways. 
There was an error in the computation of 2^register, and the sequence of
zeroes computed after the hashing did not included the "1".
 2014-04-16 15:19:40 +02:00
antirez 3035075b2c HLLCOUNT implemented. 2014-04-16 15:19:40 +02:00
antirez 959d0f012a HLLADD implemented. 2014-04-16 15:19:33 +02:00
antirez ac29bb2ae4 hllAdd() low level HyperLogLog "add" implemented. 2014-04-16 15:19:05 +02:00
antirez 2a91f548c4 HyperLogLog: redefine constants using "P". 2014-04-16 15:19:05 +02:00
antirez 3f159bbd44 HLL_SET_REGISTER fixed. 
There was an error in the first version of the macro.
Now the HLLSELFTEST test reports success.
 2014-04-16 15:19:05 +02:00
antirez a18e880bf1 Use REDIS_HLL_REGISTER_MAX when possible. 2014-04-16 15:19:05 +02:00
antirez 00409d8742 HLL_(SET|GET)_REGISTER types fixed. 2014-04-16 15:19:05 +02:00
antirez de3f821af6 HLLSELFTEST command implemented. 
To test the bitfield array of counters set/get macros from the Redis Tcl
suite is hard, so a specialized command that is able to test the
internals was developed.
 2014-04-16 15:18:57 +02:00
antirez 02d88fb201 HyperLogLog: initial sketch of registers access. 2014-04-16 15:17:26 +02:00
antirez ddbdd3b0df Document TTL changes in the 2.6 -> 2.8 migration doc. 2014-03-27 15:53:14 +01:00
antirez d2e59c2715 Redis 2.8.8. 2014-03-25 11:30:42 +01:00
antirez 99fc582f8c Test: do not complain when "leaks" can't run because process died. 2014-03-25 09:34:11 +01:00
antirez 3580bb485a adjustOpenFilesLimit() refactoring. 
In this commit:
* Decrement steps are semantically differentiated from the reserved FDs.
  Previously both values were 32 but the meaning was different.
* Make it clear that we save setrlimit errno.
* Don't explicitly handle wrapping of 'f', but prevent it from
  happening.
* Add comments to make the function flow more readable.

This integrates PR #1630
 2014-03-25 09:07:21 +01:00
Matt Stancliff c3510af1c0 Fix potentially incorrect errno usage 
errno may be reset by the previous call to redisLog, so capture
the original value for proper error reporting.
 2014-03-25 09:07:21 +01:00
Matt Stancliff 771f8ad0e7 Add REDIS_MIN_RESERVED_FDS define for open fds 
Also update the original REDIS_EVENTLOOP_FDSET_INCR to
include REDIS_MIN_RESERVED_FDS. REDIS_EVENTLOOP_FDSET_INCR
exists to make sure more than (maxclients+RESERVED) entries
are allocated, but we can only guarantee that if we include
the current value of REDIS_MIN_RESERVED_FDS as a minimum
for the INCR size.
 2014-03-25 09:07:21 +01:00
Matt Stancliff 3ce742d1d5 Fix infinite loop on startup if ulimit too low 
Fun fact: rlim_t is an unsigned long long on all platforms.

Continually subtracting from a rlim_t makes it get smaller
and smaller until it wraps, then you're up to 2^64-1.

This was causing an infinite loop on Redis startup if
your ulimit was extremely (almost comically) low.

The case of (f > oldlimit) would never be met in a case like:

    f = 150
    while (f > 20) f -= 128

Since f is unsigned, it can't go negative and would
take on values of:

    Iteration 1: 150 - 128 => 22
    Iteration 2:  22 - 128 => 18446744073709551510
    Iterations 3-∞: ...

To catch the wraparound, we use the previous value of f
stored in limit.rlimit_cur.  If we subtract from f and
get a larger number than the value it had previously,
we print an error and exit since we don't have enough
file descriptors to help the user at this point.

Thanks to @bs3g for the inspiration to fix this problem.
Patches existed from @bs3g at antirez#1227, but I needed to repair a few other
parts of Redis simultaneously, so I didn't get a chance to use them.
 2014-03-25 09:07:21 +01:00
Matt Stancliff b59585396b Improve error handling around setting ulimits 
The log messages about open file limits have always
been slightly opaque and confusing.  Here's an attempt to
fix their wording, detail, and meaning.  Users will have a
better understanding of how to fix very common problems
with these reworded messages.

Also, we handle a new error case when maxclients becomes less
than one, essentially rendering the server unusable.  We
now exit on startup instead of leaving the user with a server
unable to handle any connections.

This fixes antirez#356 as well.
 2014-03-25 09:07:21 +01:00
Matt Stancliff 6826af1b50 Replace magic 32 with REDIS_EVENTLOOP_FDSET_INCR 
32 was the additional number of file descriptors Redis
would reserve when managing a too-low ulimit.  The
number 32 was in too many places statically, so now
we use a macro instead that looks more appropriate.

When Redis sets up the server event loop, it uses:
    server.maxclients+REDIS_EVENTLOOP_FDSET_INCR

So, when reserving file descriptors, it makes sense to
reserve at least REDIS_EVENTLOOP_FDSET_INCR FDs instead
of only 32.  Currently, REDIS_EVENTLOOP_FDSET_INCR is
set to 128 in redis.h.

Also, I replaced the static 128 in the while f < old loop
with REDIS_EVENTLOOP_FDSET_INCR as well, which results
in no change since it was already 128.

Impact: Users now need at least maxclients+128 as
their open file limit instead of maxclients+32 to obtain
actual "maxclients" number of clients.  Redis will carve
the extra REDIS_EVENTLOOP_FDSET_INCR file descriptors it
needs out of the "maxclients" range instead of failing
to start (unless the local ulimit -n is too low to accomidate
the request).
 2014-03-25 09:07:21 +01:00
Matt Stancliff 611372fa97 Fix maxclients error handling 
Everywhere in the Redis code base, maxclients is treated
as an int with (int)maxclients or `maxclients = atoi(source)`,
so let's make maxclients an int.

This fixes a bug where someone could specify a negative maxclients
on startup and it would work (as well as set maxclients very high)
because:

    unsigned int maxclients;
    char *update = "-300";
    maxclients = atoi(update);
    if (maxclients < 1) goto fail;

But, (maxclients < 1) can only catch the case when maxclients
is exactly 0.  maxclients happily sets itself to -300, which isn't
-300, but rather 4294966996, which isn't < 1, so... everything
"worked."

maxclients config parsing checks for the case of < 1, but maxclients
CONFIG SET parsing was checking for case of < 0 (allowing
maxclients to be set to 0).  CONFIG SET parsing is now updated to
match config parsing of < 1.

It's tempting to add a MINIMUM_CLIENTS define, but... I didn't.

These changes were inspired by antirez#356, but this doesn't
fix that issue.
 2014-03-25 09:07:21 +01:00
Matt Stancliff 3bd3240660 Sentinel: remove variable causing warning 
GCC-4.9 warned about this, but clang didn't.

This commit fixes warning:
sentinel.c: In function 'sentinelReceiveHelloMessages':
sentinel.c:2156:43: warning: variable 'master' set but not used [-Wunused-but-set-variable]
     sentinelRedisInstance *ri = c->data, *master;
 2014-03-25 08:06:04 +01:00
antirez 79349affd5 Fixed undefined variable value with certain code paths. 
In sentinelFlushConfig() fd could be undefined when the following if
statement was true:

        if (rewrite_status == -1) goto werr;

This could cause random file descriptors to get closed.
 2014-03-25 08:03:11 +01:00
Matt Stancliff 80dec5e4df Sentinel: Notify user when config can't be saved 2014-03-25 08:03:07 +01:00
Jan-Erik Rediger a2ec9a9068 Small typo fixed 2014-03-25 08:02:56 +01:00
Matt Stancliff 88c6c669c3 Fix data loss when save AOF/RDB with no free space 
Previously, the (!fp) would only catch lack of free space
under OS X.  Linux waits to discover it can't write until
it actually writes contents to disk.

(fwrite() returns success even if the underlying file
has no free space to write into.  All the errors
only show up at flush/sync/close time.)

Fixes antirez/redis#1604
 2014-03-24 21:56:22 +01:00
Jan-Erik Rediger 6182dad37c Finally fix the install_server.sh script. 
Includes changes from a dozen bug reports and pull requests.
Was tested on Ubuntu, Debian and CentOS.
 2014-03-24 18:12:58 +01:00
antirez 4ebc7e3738 Sample and cache RSS in serverCron(). 
Obtaining the RSS (Resident Set Size) info is slow in Linux and OSX.
This slowed down the generation of the INFO 'memory' section.

Since the RSS does not require to be a real-time measurement, we
now sample it with server.hz frequency (10 times per second by default)
and use this value both to show the INFO rss field and to compute the
fragmentation ratio.

Practically this does not make any difference for memory profiling of
Redis but speeds up the INFO call significantly.
 2014-03-24 12:03:44 +01:00
antirez 72257f4b87 sdscatvprintf(): Try to use a static buffer. 
For small content the function now tries to use a static buffer to avoid
a malloc/free cycle that is too costly when the function is used in the
context of performance critical code path such as INFO output generation.

This change was verified to have positive effects in the execution speed
of the INFO command.
 2014-03-24 10:23:58 +01:00
antirez 7054f2a029 Cache uname() output across INFO calls. 
Uname was profiled to be a slow syscall. It produces always the same
output in the context of a single execution of Redis, so calling it at
every INFO output generation does not make too much sense.

The uname utsname structure was modified as a static variable. At the
same time a static integer was added to check if we need to call uname
the first time.
 2014-03-24 10:02:58 +01:00
antirez 7a20f09647 sdscatvprintf(): guess buflen using format length. 
sdscatvprintf() uses a loop where it tries to output the formatted
string in a buffer of the initial length, if there was not enough room,
a buffer of doubled size is tried and so forth.

The initial guess for the buffer length was very poor, an hardcoded
"16". This caused the printf to be processed multiple times without a
good reason. Given that printf functions are already not fast, the
overhead was significant.

The new heuristic is to use a buffer 4 times the length of the format
buffer, and 32 as minimal size. This appears to be a good balance for
typical uses of the function inside the Redis code base.

This change improved INFO command performances 3 times.
 2014-03-24 09:45:18 +01:00
antirez 001775f5fb Use 24 bits for the lru object field and improve resolution. 
There were 2 spare bits inside the Redis object structure that are now
used in order to enlarge 4x the range of the LRU field.

At the same time the resolution was improved from 10 to 1 second: this
still provides 194 days before the LRU counter overflows (restarting from
zero).

This is not a problem since it only causes lack of eviction precision for
objects not touched for a very long time, and the lack of precision is
only temporary.
 2014-03-21 11:19:28 +01:00
antirez c68189a19f Specify lruclock in redisServer structure via REDIS_LRU_BITS. 
The padding field was totally useless: removed.
 2014-03-21 11:17:52 +01:00
antirez ff8c81873a Set LRU parameters via REDIS_LRU_BITS define. 2014-03-21 11:17:36 +01:00
antirez e3b71a1c02 Unify stats reset for CONFIG RESETSTAT / initServer(). 
Now CONFIG RESETSTAT makes sure to reset all the fields, and in the
future it will be simpler to avoid missing new fields.
 2014-03-21 11:16:12 +01:00
antirez 0937377aa2 Sentinel: sentinelRefreshInstanceInfo() minor refactoring. 
Test sentinel.tilt condition on top and return if it is true.
This allows to remove the check for the tilt condition in the remaining
code paths of the function.
 2014-03-21 11:16:12 +01:00
antirez 686839b477 Sentinel test: 02 unit better coverage + refactoring. 2014-03-21 11:16:12 +01:00
antirez 6d0e408a27 Sentinel test: foreach_instance_id implements 'break'. 2014-03-21 11:16:12 +01:00
antirez ba2edc4191 Sentinel: instance_is_killed proc added to sentinel.tcl. 2014-03-21 11:16:12 +01:00
antirez 9c2063fb8f Sentinel: propagate down-after-ms changes to slaves and sentinels. 2014-03-21 11:16:12 +01:00
antirez ffa8f479a5 Sentinel: down-after-milliseconds is not master-specific. 
addReplySentinelRedisInstance() modified so that this field is displayed
for all the kind of instances: Sentinels, Masters, Slaves.
 2014-03-21 11:16:12 +01:00
antirez 42091a79bb Sentinel failure detection implementation improved. 
Failure detection in Sentinel is ping-pong based. It used to work by
remembering the last time a valid PONG reply was received, and checking
if the reception time was too old compared to the current current time.

PINGs were sent at a fixed interval of 1 second.

This works in a decent way, but does not scale well when we want to set
very small values of "down-after-milliseconds" (this is the node
timeout basically).

This commit reiplements the failure detection making a number of
changes. Some changes are inspired to Redis Cluster failure detection
code:

* A new last_ping_time field is added in representation of instances.
  If non zero, we have an active ping that was sent at the specified
  time. When a valid reply to ping is received, the field is zeroed
  again.
* last_ping_time is not reset when we reconnect the link or send a new
  ping, so from our point of view it represents the time we started
  waiting for the instance to reply to our pings without receiving a
  reply.
* last_ping_time is now used in order to check if the instance is
  timed out. This means that we can have a node timeout of 100
  milliseconds and yet the system will work well since the new check is
  not bound to the period used to send pings.
* Pings are now sent every second, or often if the value of
  down-after-milliseconds is less than one second. With a lower limit of
  10 HZ ping frequency.
* Link reconnection code was improved. This is used in order to try to
  reconnect the link when we are at 50% of the node timeout without a
  valid reply received yet. However the old code triggered unnecessary
  reconnections when the node timeout was very small. Now that should be
  ok.

The new code passes the tests but more testing is needed and more unit
tests stressing the failure detector, so currently this is merged only
in the unstable branch.
 2014-03-21 11:16:11 +01:00
antirez 38241c4b1e Sentinel: use CLIENT SETNAME when connecting to Redis. 
This makes debugging / monitoring of Sentinels simpler since you can
identify sentinels in CLIENT LIST output of Redis instances.
 2014-03-21 11:16:11 +01:00
Matt Stancliff 9de0755869 Fix segfault from accessing array out of bounds 
argc == 2; argv[2] == crash
 2014-03-14 22:57:10 +01:00
antirez a31a0b4336 Sentinel: be safe under crash-recovery assumptions. 
Sentinel's main safety argument is that there are no two configurations
for the same master with the same version (configuration epoch).

For this to be true Sentinels require to be authorized by a majority.
Additionally Sentinels require to do two important things:

* Never vote again for the same epoch.
* Never exchange an old vote for a fresh one.

The first prerequisite, in a crash-recovery system model, requires to
persist the master->leader_epoch on durable storage before to reply to
messages. This was not the case.

We also make sure to persist the current epoch in order to never reply
to stale votes requests from other Sentinels, after a recovery.

The configuration is persisted by making use of fsync(), this is
considered in the context of this code a good enough guarantee that
after a restart our durable state is restored, however this may not
always be the case depending on the kind of hardware and operating
system used.
 2014-03-14 22:57:06 +01:00
antirez 6b0e36ffc6 Sentinel: fake PUBLISH command to receive HELLO messages. 
Now the way HELLO messages are received is unified.
Now it is no longer needed for Sentinels to converge to the higher
configuration for a master to be able to chat via some Redis instance,
the are able to directly exchanges configurations.

Note that this commit does not include the (trivial) change needed to
send HELLO messages to Sentinel instances as well, since for an error I
committed the change in the previous commit that refactored hello
messages processing into a separated function.
 2014-03-14 11:04:54 +01:00
antirez bd48ff69b0 Sentinel: HELLO processing refactored into sentinelProcessHelloMessage(). 2014-03-14 10:56:56 +01:00
antirez 3703112671 Linenoise updated, multiline mode enabled in redis-cli. 2014-03-13 15:12:04 +01:00
zhanghailei ccd2c18c1c According to context,the size should be 16 rather than 64 2014-03-11 10:12:19 +01:00
zhanghailei daa5d2a6c9 FIXED a typo more thank should be more than 2014-03-11 10:12:13 +01:00
zhanghailei ca0720b694 refer to updateLRUClock's comment REDIS_LRU_CLOCK_MAX is 22 bits,but #define REDIS_LRU_CLOCK_MAX ((1<<21)-1) only 21 bits 2014-03-11 10:12:00 +01:00
Matt Stancliff ffe742f92a Reset op_sec_last_sample_ops when reset requested 
This value needs to be set to zero (in addition to
stat_numcommands) or else people may see
a negative operations per second count after they
run CONFIG RESETSTAT.

Fixes antirez/redis#1577
 2014-03-11 10:10:54 +01:00
antirez 6fb9e2d717 Typo in sentinel.conf, exists -> exits. 2014-03-11 10:10:30 +01:00
antirez eb9e1526e6 DEBUG ERROR implemented. 
The new "error" subcommand of the DEBUG command can reply with an user
selected error, specified as its sole argument:

    DEBUG ERROR "LOADING please wait..."

The error is generated just prefixing the command argument with a "-"
character, and replacing newlines with spaces (since error replies can't
include newlines).

The goal of the command is to help in Client libraries unit tests by
making simple to simulate a command call triggering a given error.
 2014-03-10 23:04:37 +01:00
Matt Stancliff a6970570e3 Fix return value check for anetTcpAccept 
anetTcpAccept returns ANET_ERR, not AE_ERR.

This isn't a physical error since both ANET_ERR
and AE_ERR are -1, but better to be consistent.
 2014-03-10 15:47:20 +01:00
antirez fe0ab7d234 Fixed memory leak in SORT LIMIT option argument parsing on error. 2014-03-10 15:45:29 +01:00
antirez 464fef9bf8 Redis 2.8.7. 2014-03-05 14:42:50 +01:00
antirez aab16ead92 Cast saveparams[].seconds to long for %ld format specifier. 2014-03-05 11:26:50 +01:00
antirez 86afc0706d Merge branch '2.8' of github.com:/antirez/redis into 2.8 2014-03-05 10:22:56 +01:00
antirez 55b8f6ec1c Document why we update peak memory in INFO. 2014-03-05 10:16:20 +01:00
Matt Stancliff 647a261465 Force INFO used_memory_peak to match peak memory 
used_memory_peak only updates in serverCron every server.hz,
but Redis can use more memory and a user can request memory
INFO before used_memory_peak gets updated in the next
cron run.

This patch updates used_memory_peak to the current
memory usage if the current memory usage is higher
than the recorded used_memory_peak value.

(And it only calls zmalloc_used_memory() once instead of
twice as it was doing before.)
 2014-03-05 10:16:16 +01:00
antirez 9c66cd91be Sentinel test: Makefile target added. 2014-03-05 10:16:12 +01:00
michael-grunder 6991792943 Improved bigkeys with progress, pipelining and summary 
This commit reworks the redis-cli --bigkeys command to provide more
information about our progress as well as output summary information
when we're done.

 - We now show an approximate percentage completion as we go
 - Hiredis pipelining is used for TYPE and SIZE retreival
 - A summary of keyspace distribution and overall breakout at the end
 2014-03-05 10:16:06 +01:00
antirez 7d65b7199a BITPOS fuzzy testing. 2014-03-05 10:16:02 +01:00
antirez c19cfde65d Basic BITPOS tests. 2014-03-05 10:15:55 +01:00
antirez 55f4b20f31 Sentinel test: set less time sensitive defaults. 
This commit sets the failover timeout to 30 seconds instead of the 180
seconds default, and allows to reconfigure multiple slaves at the same
time.

This makes tests less sensible to timing, with the result that there are
less false positives due to normal behaviors that require time to
succeed or to be retried.

However the long term solution is probably some way in order to detect
when a test failed because of timing issues (for example split brain
during leader election) and retry it.
 2014-03-05 10:15:32 +01:00
antirez 1606978a0b Sentinel: more aggressive failover start desynchronization. 
Sentinel needs to avoid split brain conditions due to multiple sentinels
trying to get voted at the exact same time.

So far some desynchronization was provided by fluctuating server.hz,
that is the frequency of the timer function call. However the
desynchonization provided in this way was not enough when using many
Sentinel instances, especially when a large quorum value is used in
order to force a greater degree of agreement (more than N/2+1).

It was verified that it was likely to trigger a split brain
condition, forcing the system to try again after a timeout.
Usually the system will succeed after a few retries, but this is not
optimal.

This commit desynchronizes instances in a more effective way to make it
likely that the first attempt will be successful.
 2014-03-05 10:15:32 +01:00
antirez 6200191943 CONFIG REWRITE should be logged at WARNING level. 2014-03-05 10:15:32 +01:00
antirez 2af2173a60 Sentinel test: debugging console improved. 2014-03-05 10:15:32 +01:00
antirez 1dc1e31c2b Sentinel test: initial debugging console. 2014-03-05 10:15:32 +01:00
antirez b132121a71 Sentinel test: be more patient in create_redis_master_slave_cluster. 2014-03-05 10:15:32 +01:00
antirez d1d706c923 Sentiel test: add test start time in output. 2014-03-05 10:15:32 +01:00
antirez c99cd2fd40 Sentinel test: use 1000 as retry in initial 00 unit test. 2014-03-05 10:15:32 +01:00
antirez f33d4a6c24 Sentinel test: initial tests in 03 unit. 2014-03-05 10:15:32 +01:00
antirez c2d99d49b5 Sentinel test: foreach_instance_id now supports 'continue'. 2014-03-05 10:15:32 +01:00
antirez e40f8378bd Sentienl test: fixed typo in unit 03 top comment. 2014-03-05 10:15:32 +01:00
antirez a0335e18ba Sentinel test: Makefile target added. 2014-02-28 16:00:17 +01:00
antirez 950cb76e7b BITPOS fuzzy testing. 2014-02-27 15:58:34 +01:00
antirez 42e3630d77 Basic BITPOS tests. 2014-02-27 15:58:30 +01:00
antirez 1892b56224 warnigns -> warnings in redisBitpos(). 2014-02-27 15:58:27 +01:00
antirez 2c8036f7b2 More consistent BITPOS behavior with bit=0 and ranges. 
With the new behavior it is possible to specify just the start in the
range (the end will be assumed to be the first byte), or it is possible
to specify both start and end.

This is useful to change the behavior of the command when looking for
zeros inside a string.

1) If the user specifies both start and end, and no 0 is found inside
   the range, the command returns -1.

2) If instead no range is specified, or just the start is given, even
   if in the actual string no 0 bit is found, the command returns the
   first bit on the right after the end of the string.

So for example if the string stored at key foo is "\xff\xff":

    BITPOS foo (returns 16)
    BITPOS foo 0 -1 (returns -1)
    BITPOS foo 0 (returns 16)

The idea is that when no end is given the user is just looking for the
first bit that is zero and can be set to 1 with SETBIT, as it is
"available". Instead when a specific range is given, we just look for a
zero within the boundaries of the range.
 2014-02-27 15:58:21 +01:00
antirez 3294f74fef Initial implementation of BITPOS. 
It appears to work but more stress testing, and both unit tests and
fuzzy testing, is needed in order to ensure the implementation is sane.
 2014-02-27 15:58:14 +01:00
antirez 82d2e295b8 Added two more BITCOUNT tests stressing misaligned access. 2014-02-27 15:57:56 +01:00
antirez a3eb3f9c3b BITCOUNT fuzzy test with random start/end added. 
It was verified in practice that this test is able to stress much more
the implementation by introducing errors that were only trivially to
detect with different offsets but impossible to detect starting always
at zero and counting bits the full length of the string.
 2014-02-27 15:57:53 +01:00
antirez 30a92b6c76 Fix misaligned word access in redisPopcount(). 2014-02-27 15:57:49 +01:00
antirez c1cc28f230 warnigns -> warnings in redisBitpos(). 2014-02-27 15:55:56 +01:00
antirez d79f9ebdb5 More consistent BITPOS behavior with bit=0 and ranges. 
With the new behavior it is possible to specify just the start in the
range (the end will be assumed to be the first byte), or it is possible
to specify both start and end.

This is useful to change the behavior of the command when looking for
zeros inside a string.

1) If the user specifies both start and end, and no 0 is found inside
   the range, the command returns -1.

2) If instead no range is specified, or just the start is given, even
   if in the actual string no 0 bit is found, the command returns the
   first bit on the right after the end of the string.

So for example if the string stored at key foo is "\xff\xff":

    BITPOS foo (returns 16)
    BITPOS foo 0 -1 (returns -1)
    BITPOS foo 0 (returns 16)

The idea is that when no end is given the user is just looking for the
first bit that is zero and can be set to 1 with SETBIT, as it is
"available". Instead when a specific range is given, we just look for a
zero within the boundaries of the range.
 2014-02-27 15:55:47 +01:00
antirez 25e2791ec3 Initial implementation of BITPOS. 
It appears to work but more stress testing, and both unit tests and
fuzzy testing, is needed in order to ensure the implementation is sane.
 2014-02-27 15:54:31 +01:00
antirez c955b47d62 Added two more BITCOUNT tests stressing misaligned access. 2014-02-27 15:53:40 +01:00
antirez bfe4c560e6 BITCOUNT fuzzy test with random start/end added. 
It was verified in practice that this test is able to stress much more
the implementation by introducing errors that were only trivially to
detect with different offsets but impossible to detect starting always
at zero and counting bits the full length of the string.
 2014-02-27 15:53:40 +01:00
antirez 580ed0768f Fix misaligned word access in redisPopcount(). 2014-02-27 15:53:40 +01:00
antirez d90e3e3a70 Sentinel test: add stub for unit 04. 2014-02-25 15:55:37 +01:00
antirez 89d95ec2e7 Sentinel test: added TODO items in 02 unit. 2014-02-25 15:55:37 +01:00
antirez 65194d11c3 redis-cli: also remove useless uint8_t. 2014-02-25 15:09:15 +01:00
antirez 16c2189d4e redis-cli: don't use uint64_t where actually not needed. 
The computation is just something to take the CPU busy, no need to use a
specific type. Since stdint.h was not included this prevented
compilation on certain systems.
 2014-02-25 15:09:15 +01:00
antirez 1b2bcd4210 redis-cli: check argument existence for --pattern. 2014-02-25 15:09:15 +01:00
antirez 6ec3dc2c42 redis-cli: --intrinsic-latency run mode added. 2014-02-25 15:09:15 +01:00
antirez 02ae6cb01a redis-cli: added comments to split program in parts. 2014-02-25 15:09:15 +01:00
antirez fe05112029 Sentinel test: check role at end of unit 01. 2014-02-25 15:08:41 +01:00
michael-grunder cf76fdd04f Update --bigkeys to use SCAN 
This commit changes the findBigKeys() function in redis-cli.c to use the new
SCAN command for iterating the keyspace, rather than RANDOMKEY.  Because we
can know when we're done using SCAN, it will exit after exhausting the keyspace.
 2014-02-25 15:08:38 +01:00
antirez 9e34e296cf Sentinel test: kill masters instead of using DEBUG SLEEP in all tests. 2014-02-25 15:08:34 +01:00
antirez dfacfafc2a Sentinel test: restart instances left killed by previous unit. 
An unit can abort in the middle for an error. The next unit should not
assume that the instances are in a clean state, and must restart what
was left killed.
 2014-02-25 10:30:35 +01:00
antirez 2fab983c69 Sentinel test: jump to next unit on test failure. 
Sentinel tests are designed to be dependent on the previous tests in the
same unit, so usually we can't continue with the next test in the same
unit if a previous test failed.
 2014-02-25 10:30:35 +01:00
antirez a6d560db64 Sentinel test: test majority crashing Sentinels. 
The test was previously performed by removing the master from the
Sentinel monitored masters. The test with the Sentinels crashed is
more similar to real-world partitions / failures.
 2014-02-25 10:30:35 +01:00
antirez 3f16fee39a Sentinel test: restart_instance should refresh pid attrib. 
Also kill_instance was modified to warn when a test will try to kill the
same instance multiple times for error.
 2014-02-25 10:30:35 +01:00
antirez e8e2ac072c Sentinel test: more stuff mored from 00-base to init. 
The area a number of mandatory tests to craete a stable setup for
testing that is not too sensitive to timing issues. All those tests
moved to includes/init-tests, and marked as (init).
 2014-02-25 10:30:35 +01:00
antirez 85fa77e02e Sentinel: log quorum with +monitor event. 2014-02-25 10:30:35 +01:00
antirez 2d48d7c2fc Sentinel test: removed useless code to set SDOWN timeout. 
The new common initialization code used to start a new unit already set
the timeout to 2000 milliseconds.
 2014-02-25 10:30:35 +01:00
antirez 6e6106797e Sentinel: generate +monitor events at startup. 2014-02-25 10:30:35 +01:00
antirez 96162c0c58 Sentinel: log +monitor and +set events. 
Now that we have a runtime configuration system, it is very important to
be able to log how the Sentinel configuration changes over time because
of API calls.
 2014-02-25 10:30:35 +01:00
antirez 39eacde136 Sentinel: added missing exit(1) after checking for config file. 2014-02-25 10:30:35 +01:00
antirez 28aa15a6ea Sentinel test: tmp dir and gitignore added. 2014-02-25 10:30:35 +01:00
antirez 3917224f7c Sentinel test: minor fixes to --pause-on-error. 2014-02-25 10:30:35 +01:00
antirez 001ab29915 Sentinel test: --pause-on-error option added. 
Pause the test with running instances available for state inspection on
error.
 2014-02-25 10:30:35 +01:00
antirez d31a394148 Sentinel test: added empty units to fill later. 2014-02-25 10:30:35 +01:00
antirez d83ab8f9d4 Sentinel: IDONTKNOW error removed. 
This error was conceived for the older version of Sentinel that worked
via master redirection and that was not able to get configuration
updates from other Sentinels via the Pub/Sub channel of masters or
slaves.

This reply does not make sense today, every Sentinel should reply with
the best information it has currently. The error will make even more
sense in the future since the plan is to allow Sentinels to update the
configuration of other Sentinels via gossip with a direct chat without
the prerequisite that they have at least a monitored instance in common.
 2014-02-25 10:30:35 +01:00
antirez 1dfab3d107 Sentinel test: framework improved and conf-update unit added. 
It is now possible to kill and restart sentinel or redis instances for
more real-world testing.

The 01 unit tests the capability of Sentinel to update the configuration
of Sentinels rejoining the cluster, however the test is pretty trivial
and more tests should be added.
 2014-02-25 10:30:35 +01:00
antirez aee165e084 Sentinel test: move init tests as includes. 
Most units will start with these two basic tests to create an
environment where the real tests are ran.
 2014-02-25 10:30:35 +01:00
antirez 73b290cc5b Sentinel test: ability to run just a subset of test files. 2014-02-25 10:30:35 +01:00
Matt Stancliff 3ced78dc6c Auto-enter slaveMode when SYNC from redis-cli 
If someone asks for SYNC or PSYNC from redis-cli,
automatically enter slaveMode (as if they ran
redis-cli --slave) and continue printing the replication
stream until either they Ctrl-C or the master gets disconnected.
 2014-02-25 10:30:35 +01:00
antirez 6441a41f94 Sentinel: report instances role switch events. 
This is useful mostly for debugging of issues.
 2014-02-20 12:28:03 +01:00
antirez 05dde71919 Sentinel test: some reliability fixes to 00-base tests. 2014-02-20 12:28:03 +01:00
antirez c499d82f28 Sentinel test: check that role matches at end of 00-base. 2014-02-20 12:28:03 +01:00
antirez c01aff1234 Sentinel test: ODOWN and agreement. 2014-02-20 12:28:03 +01:00
antirez 56cb037a86 Sentinel test: check reconfig of slaves and old master. 2014-02-20 12:28:03 +01:00
antirez d4edf63bc4 Sentinel test: basic failover tested. Framework improvements. 2014-02-20 12:28:02 +01:00
antirez 722d4f0c41 Sentinel test: basic tests for MONITOR and auto-discovery. 2014-02-20 12:28:02 +01:00
antirez b26660e130 Sentinel test: info fields, master-slave setup, fixes. 2014-02-20 12:28:02 +01:00
antirez ac5fd5a519 Prefix test file names with numbers to force exec order. 2014-02-20 12:28:02 +01:00
antirez 890559ea1c Sentinel test: provide basic commands to access instances. 2014-02-20 12:28:02 +01:00
antirez 905c55d5ae Sentinel: SENTINEL_SLAVE_RECONF_RETRY_PERIOD -> RECONF_TIMEOUT 
Rename define to match the new meaning.
 2014-02-18 10:30:29 +01:00
antirez 1b345ec3f0 Sentinel: fix slave promotion timeout. 
If we can't reconfigure a slave in time during failover, go forward as
anyway the slave will be fixed by Sentinels in the future, once they
detect it is misconfigured.

Otherwise a failover in progress may never terminate if for some reason
the slave is uncapable to sync with the master while at the same time
it is not disconnected.
 2014-02-18 10:30:28 +01:00
antirez 351e790032 Sentinel: initial testing framework. 
Nothing tested at all so far... Just the infrastructure spawning N
Sentinels and N Redis instances that the test will use again and again.
 2014-02-17 17:38:58 +01:00
antirez f71c7de88e Test: colorstr moved to util.tcl. 2014-02-17 17:38:54 +01:00
antirez 0886c98fef Test: code to test server availability refactored. 
Some inline test moved into server_is_up procedure.
Also find_available_port was moved into util since it is going
to be used for the Sentinel test as well.
 2014-02-17 12:29:54 +01:00
antirez 4237f14a8a Get absoulte config file path before processig 'dir'. 
The code tried to obtain the configuration file absolute path after
processing the configuration file. However if config file was a relative
path and a "dir" statement was processed reading the config, the absolute
path obtained was wrong.

With this fix the absolute path is obtained before processing the
configuration while the server is still in the original directory where
it was executed.
 2014-02-17 12:14:19 +01:00
antirez 5efee4f0bb Sentinel: better specify startup errors due to config file. 
Now it logs the file name if it is not accessible. Also there is a
different error for the missing config file case, and for the non
writable file case.
 2014-02-17 12:10:12 +01:00
antirez 1d4d9e7b14 Redis 2.8.6. 2014-02-13 15:43:40 +01:00
antirez 85492dcfef Update cached time in rdbLoad() callback. 
server.unixtime and server.mstime are cached less precise timestamps
that we use every time we don't need an accurate time representation and
a syscall would be too slow for the number of calls we require.

Such an example is the initialization and update process of the last
interaction time with the client, that is used for timeouts.

However rdbLoad() can take some time to load the DB, but at the same
time it did not updated the time during DB loading. This resulted in the
bug described in issue #1535, where in the replication process the slave
loads the DB, creates the redisClient representation of its master, but
the timestamp is so old that the master, under certain conditions, is
sensed as already "timed out".

Thanks to @yoav-steinberg and Redis Labs Inc for the bug report and
analysis.
 2014-02-13 15:13:35 +01:00
antirez eb10bb31ac Log when CONFIG REWRITE goes bad. 2014-02-13 14:33:50 +01:00
antirez ebdb37cea7 Test: regression for issue #1549. 
It was verified that reverting the commit that fixes the bug, the test
no longer passes.
 2014-02-13 12:27:08 +01:00
antirez 9b73a274ac Fix script cache bug in the scripting engine. 
This commit fixes a serious Lua scripting replication issue, described
by Github issue #1549. The root cause of the problem is that scripts
were put inside the script cache, assuming that slaves and AOF already
contained it, even if the scripts sometimes produced no changes in the
data set, and were not actaully propagated to AOF/slaves.

Example:

    eval "if tonumber(KEYS[1]) > 0 then redis.call('incr', 'x') end" 1 0

Then:

    evalsha <sha1 step 1 script> 1 0

At this step sha1 of the script is added to the replication script cache
(the script is marked as known to the slaves) and EVALSHA command is
transformed to EVAL. However it is not dirty (there is no changes to db),
so it is not propagated to the slaves. Then the script is called again:

    evalsha <sha1 step 1 script> 1 1

At this step master checks that the script already exists in the
replication script cache and doesn't transform it to EVAL command. It is
dirty and propagated to the slaves, but they fail to evaluate the script
as they don't have it in the script cache.

The fix is trivial and just uses the new API to force the propagation of
the executed command regardless of the dirty state of the data set.

Thank you to @minus-infinity on Github for finding the issue,
understanding the root cause, and fixing it.
 2014-02-13 12:16:34 +01:00
antirez 96973a7c33 AOF write error: retry with a frequency of 1 hz. 2014-02-12 16:57:17 +01:00
antirez fadbbdd3f4 AOF: don't abort on write errors unless fsync is 'always'. 
A system similar to the RDB write error handling is used, in which when
we can't write to the AOF file, writes are no longer accepted until we
are able to write again.

For fsync == always we still abort on errors since there is currently no
easy way to avoid replying with success to the user otherwise, and this
would violate the contract with the user of only acknowledging data
already secured on disk.
 2014-02-12 16:57:13 +01:00
antirez 688d32e16b Don't count time to feed MONITORs in SLOWLOG. 2014-02-07 18:29:26 +01:00
antirez 3e4968339b Sentinel: allow SHUTDOWN command in Sentinel mode. 2014-02-07 11:22:30 +01:00
antirez 301a0cfc69 Check for EAGAIN in sendBulkToSlave(). 
Sometime an osx master with a Linux server over a slow link caused
a strange error where osx called the writable function for
the socket but actually apparently there was no room in the socket
buffer to accept the write: write(2) call returned an EAGAIN error,
that was not checked, so we considered write(2) == 0 always as a connection
reset, which was unfortunate since the bulk transfer has to start again.

Also more errors are logged with the WARNING level in the same code path
now.
 2014-02-05 16:41:04 +01:00
antirez 4e809a9a19 Redis 2.8.5. 2014-02-04 11:17:21 +01:00
antirez ddcf160309 Move mstime_t define outside sentinel.c. 
The define is now used in other parts of Redis 2.8 tree instead of long
long.

A nice side effect is that now 2.8 and unstable sentinel.c files are
identical as it should be.
 2014-02-03 16:34:46 +01:00
antirez c5bc592650 Scripting: expire keys in scripts only at first access. 
Keys expiring in the middle of the execution of Lua scripts are to
create inconsistencies in masters and / or AOF files. See the following
example:

    if redis.call("exists",KEYS[1]) == 1
    then
        redis.call("incr","mycounter")
    end

    if redis.call("exists",KEYS[1]) == 1
    then
        return redis.call("incr","mycounter")
    end

The script executes two times the same *if key exists then incrementcounter*
logic. However the two executions will work differently in the master and
the slaves, provided some unlucky timing happens.

In the master the first time the key may still exist, while the second time
the key may no longer exist. This will result in the key incremented just one
time. However as a side effect the master will generate a synthetic
`DEL` command in the replication channel in order to force the slaves to
expire the key (given that key expiration is master-driven).

When the same script will run in the slave, the key will no longer be
there, so the script will not increment the key.

The key idea used to implement the expire-at-first-lookup semantics was
provided by Marc Gravell.
 2014-02-03 16:29:25 +01:00
antirez 5201ca0ca1 Allow CONFIG and SHUTDOWN while in stale-slave state. 2014-02-03 15:51:07 +01:00
antirez 3da5cbe5bb Scripting: use mstime() and mstime_t for lua_time_start. 
server.lua_time_start is expressed in milliseconds. Use mstime_t instead
of long long, and populate it with mstime() instead of ustime()/1000.

Functionally identical but more natural.
 2014-02-03 15:46:47 +01:00
PatrickJS 0be31e2d22 update copyright year 2014-02-03 11:19:25 +01:00
antirez 66304fb122 Test: fixed osx msg passing issue in testing framework. 
The Redis test uses a server-clients model in order to parallelize the
execution of different tests. However in recent versions of osx not
setting the channel to a binary encoding caused issues even if AFAIK no
binary data is really sent via this channel. However now the channels
are deliberately set to a binary encoding and this solves the issue.

The exact issue was the test not terminating and giving the impression
of running forever, since test clients or servers were unable to
exchange the messages to continue.
 2014-01-31 16:25:13 +01:00
antirez 1406112ec4 Redis.conf comment about tcp-backlog option improved. 2014-01-31 15:03:25 +01:00
antirez 917b851491 Option "backlog" renamed "tcp-backlog". 
This is especially important since we already have a concept of backlog
(the replication backlog).
 2014-01-31 15:03:22 +01:00
Nenad Merdanovic 8dda9dbef0 Add support for listen(2) backlog definition 
In high RPS environments, the default listen backlog is not sufficient, so
giving users the power to configure it is the right approach, especially
since it requires only minor modifications to the code.
 2014-01-31 15:03:19 +01:00
antirez f0652c37a5 Sentinel: check arity for SENTINEL MASTER command. 
This fixes issue #1530.
 2014-01-31 10:14:07 +01:00
antirez a2c9d38a9f SENTINEL SET master quorum implemented. 2014-01-28 11:23:51 +01:00
antirez 89a731b3aa Fixed inverted if condition in MISCONF error code path. 2014-01-28 10:10:56 +01:00
antirez ecfefde760 Don't log MONITOR clients as disconnecting slaves. 2014-01-25 11:54:02 +01:00
antirez 43503ae5d6 redis-cli --help output improved with --scan and periods. 2014-01-22 12:08:08 +01:00
antirez b9c84d4aef redis-cli: support for --scan option. 2014-01-22 12:08:04 +01:00
antirez 248e916550 Use fflush() before fsync() in rio.c. 
Incremental flushing in rio.c is only used to avoid huge kernel buffers
synched to slow disks creating big latency spikes, so this fix has no
durability implications, however it is certainly more correct to make
sure that the FILE buffers are flushed to the kernel before calling
fsync on the file descriptor.

Thanks to Li Shao Kai for reporting this issue in the Redis mailing
list.
 2014-01-22 09:56:35 +01:00
antirez d401022d14 Fix typo in aofRewriteBufferAppend() comment. 2014-01-14 15:38:26 +01:00
antirez 4ea91aadfb Set REDIS_AOF_REWRITE_MIN_SIZE to 64mb. 
64mb is the default value in redis.conf. For some reason instead the
hard-coded default was 1mb that is too small.
 2014-01-14 11:28:18 +01:00
antirez 71f7eb2590 Redis 2.8.4. 2014-01-13 17:09:58 +01:00
antirez df00b6ac05 SENTINEL SET: error on bad option name + flush config on error. 2014-01-13 16:39:56 +01:00
antirez b8db8a0c48 SENTINEL SET implemented. 
The new command allows to change master-specific configurations
at runtime. All the settable parameters can be retrivied via the
SENTINEL MASTER command, so there is no equivalent "GET" command.
 2014-01-13 16:39:51 +01:00
antirez 8ed0b49525 Sentinel: fix wrong arity error message. 2014-01-13 16:39:47 +01:00
antirez 560e548dc6 Sentinel: SENTINEL REMOVE command added. 
The command totally removes a monitored master.
 2014-01-13 16:39:44 +01:00
antirez 0ca750d94a Sentinel: releaseSentinelRedisInstance() top comment fixed. 
The claim about unlinking the instance from the connected hash tables
was the opposite of the reality. Also the current actual behavior is
safer in most cases, so it is better to manually unlink when needed.
 2014-01-13 16:39:40 +01:00
antirez 1f9580c125 Sentinel: flush config on disk when new master is added. 2014-01-13 16:39:36 +01:00
antirez bb207007ca anetResolveIP() prototype added to anet.h. 2014-01-13 16:39:31 +01:00
antirez b7dc3204a6 Sentinel: SENTINEL MONITOR command implemented. 
It allows to add new masters to monitor at runtime.
 2014-01-13 16:39:27 +01:00
antirez 2e0ba7bb5b anetResolveIP() added to anet.c. 
The new function is used when we want to normalize an IP address without
performing a DNS lookup if the string to resolve is not a valid IP.

This is useful every time only IPs are valid inputs or when we want to
skip DNS resolution that is slow during runtime operations if we are
required to block.
 2014-01-13 16:39:24 +01:00
antirez 52cf0975c7 Sentinel: added SENTINEL MASTER <name> command. 
With SENTINEL MASTERS it was already possible to list all the configured
masters, but not a specific one.
 2014-01-13 16:39:18 +01:00
antirez 6bcc370c9d Add all the configurable fields to addReplySentinelRedisInstance(). 
Note: the auth password with the master is voluntarily not exposed.
 2014-01-13 16:39:15 +01:00
antirez ea2bffa030 Trip comment to 80 cols in SentinelCommand(). 2014-01-13 16:39:11 +01:00
antirez bca65866ff Test: regression for issues #1483. 2014-01-09 11:19:11 +01:00
antirez e4c51759f5 Fix RESTORE ttl handling in 32 bit archs. 
long was used instead of long long in order to handle a 64 bit
resolution millisecond timestamp.

This fixes issue #1483.
 2014-01-09 11:12:53 +01:00
antirez 51f71e2d8e Fix keyspace events flags-to-string conversion. 
Fixes issue #1491 on Github.
 2014-01-08 17:18:00 +01:00
antirez 05d219a6fa Test: stress events flags to/from string conversion. 2014-01-08 17:16:09 +01:00
antirez 2a1a31ca9d Don't send REPLCONF ACK to old masters. 
Masters not understanding REPLCONF ACK will reply with errors to our
requests causing a number of possible issues.

This commit detects a global replication offest set to -1 at the end of
the replication, and marks the client representing the master with the
REDIS_PRE_PSYNC flag.

Note that this flag was called REDIS_PRE_PSYNC_SLAVE but now it is just
REDIS_PRE_PSYNC as it is used for both slaves and masters starting with
this commit.

This commit fixes issue #1488.
 2014-01-08 14:27:49 +01:00
antirez 418d3d358a Clarify a comment in slaveTryPartialResynchronization(). 2014-01-08 14:11:02 +01:00
antirez 0a1a236e3e Log disconnection with slave only when ip:port is available. 2013-12-25 18:41:10 +01:00
antirez 27d06111db anetPeerToString / SockName: port can be NULL on errors too. 2013-12-25 18:39:49 +01:00
antirez 5b7c16137d anetTcpGenericConnect() bug introduced in 9d19977 fixed. 
Durign a refactoring I mispelled _port for port.
This is one of the reasons I never used _varname myself.
 2013-12-25 18:38:33 +01:00
antirez d07d4a876c Remove useless goto from anetTcpGenericConnect(). 2013-12-25 18:24:04 +01:00
antirez 9d1997706c anetTcpGenericConnect() code improved + 1 bug fix. 
Now the socket is closed if anetNonBlock() fails, and in general the
code structure makes it harder to introduce this kind of bugs in the
future.

Reference: pull request #1059.
 2013-12-25 18:16:46 +01:00
antirez 4ad219adc8 Fix CONFIG REWRITE handling of unknown options. 
There were two problems with the implementation.

1) "save" was not correctly processed when no save point was configured,
   as reported in issue #1416.
2) The way the code checked if an option existed in the "processed"
   dictionary was wrong, as we add the element with as a key associated
   with a NULL value, so dictFetchValue() can't be used to check for
   existance, but dictFind() must be used, that returns NULL only if the
   entry does not exist at all.
 2013-12-23 12:50:52 +01:00
antirez c6db326d1d Configuring port to 0 disables IP socket as specified. 
This was no longer the case with 2.8 becuase of a bug introduced with
the IPv6 support. Now it is fixed.

This fixes issue #1287 and #1477.
 2013-12-23 11:34:15 +01:00
antirez 4456ee1173 Make new masters inherit replication offsets. 
Currently replication offsets could be used into a limited way in order
to understand, out of a set of slaves, what is the one with the most
updated data. For example this comparison is possible of N slaves
were replicating all with the same master.

However the replication offset was not transferred from master to slaves
(that are later promoted as masters) in any way, so for instance if
there were three instances A, B, C, with A master and B and C
replication from A, the following could happen:

C disconnects from A.
B is turned into master.
A is switched to master of B.
B receives some write.

In this context there was no way to compare the offset of A and C,
because B would use its own local master replication offset as
replication offset to initialize the replication with A.

With this commit what happens is that when B is turned into master it
inherits the replication offset from A, making A and C comparable.
In the above case assuming no inconsistencies are created during the
disconnection and failover process, A will show to have a replication
offset greater than C.

Note that this does not mean offsets are always comparable to understand
what is, in a set of instances, since in more complex examples the
replica with the higher replication offset could be partitioned away
when picking the instance to elect as new master. However this in
general improves the ability of a system to try to pick a good replica
to promote to master.
 2013-12-22 11:54:10 +01:00
antirez 5fa937d956 Slave disconnection is an event worth logging. 2013-12-22 10:16:02 +01:00
antirez 9ba9d78ada Log when a slave lose the connection with its master. 2013-12-21 00:24:29 +01:00
antirez d33f3689d7 Clarify include directive behavior in example redis.conf. 2013-12-19 16:03:52 +01:00
antirez fb8a480f54 CONFIG REWRITE: no special handling or include and rename-command. 
CONFIG REWRITE is now wiser and does not touch what it does not
understand inside redis.conf.
 2013-12-19 16:03:49 +01:00
Yubao Liu d62bbfda16 CONFIG REWRITE: don't throw some options on config rewrite 
Those options will be thrown without this patch:
  include, rename-command, min-slaves-to-write, min-slaves-max-lag,
appendfilename.
 2013-12-19 16:03:37 +01:00
antirez 090bcc946f CONFIG REWRITE: old development comments removed. 2013-12-19 16:02:50 +01:00
antirez ddd529bc3b CONFIG REWRITE: don't wipe unknown options. 
With this commit options not explicitly rewritten by CONFIG REWRITE are
not touched at all. These include new options that may not have support
for REWRITE, and other special cases like rename-command and include.
 2013-12-19 16:02:46 +01:00
antirez 673c42bb4e Example redis.conf formatted to better show appendfilename option. 2013-12-19 10:19:13 +01:00
antirez 33f6f35f34 Makefile.dep updated. 2013-12-13 13:14:47 +01:00
antirez 8eb1cb3b52 SDIFF iterator misuse bug regression test added. 
See commit c00453d for more info about the bug.
 2013-12-13 11:37:35 +01:00
antirez 993e0ede76 SDIFF iterator misuse fixed in diff algorithm #1. 
The bug could be easily triggered by:

    SADD foo a b c 1 2 3 4 5 6
    SDIFF foo foo

When the key was the same in two sets, an unsafe iterator was used to
check existence of elements in the same set we were iterating.
Usually this would just result into a wrong output, however with the
dict.c API misuse protection we have in place, the result was actually
an assertion failed that was triggered by the CI test, while creating
random datasets for the "MASTER and SLAVE consistency" test.
 2013-12-13 11:29:59 +01:00
antirez 2507e366b2 Sentinel: dead code removed. 2013-12-13 11:01:20 +01:00
antirez f9e9448a5b Makefile: remove odd syntax not compatible with some make versions. 
See issue #1448.
 2013-12-12 15:19:29 +01:00
Yubao Liu 45c60a903a fix typo in redis.conf and sentinel.conf 2013-12-12 11:32:05 +01:00
codeeply 5e75e681fe comment mistake fixed 2013-12-12 11:25:48 +01:00
antirez 8995634014 Redis 2.8.3. 2013-12-11 15:31:57 +01:00
antirez 9a8ae5a553 Replication: publish the slave_repl_offset when disconnected from master. 
When a slave was disconnected from its master the replication offset was
reported as -1. Now it is reported as the replication offset of the
previous master, so that failover can be performed using this value in
order to try to select a slave with more processed data from a set of
slaves of the old master.
 2013-12-11 15:24:50 +01:00
Yossi Gottlieb 0ff078d8d0 Return proper error on requests with an unbalanced number of quotes. 2013-12-11 13:21:55 +01:00
Yossi Gottlieb 25ba2e9607 Fix wrong repldboff type which causes dropped replication in rare cases. 2013-12-11 11:37:58 +01:00
antirez 563d6b3f98 Slaves heartbeats during sync improved. 
The previous fix for false positive timeout detected by master was not
complete. There is another blocking stage while loading data for the
first synchronization with the master, that is, flushing away the
current data from the DB memory.

This commit uses the newly introduced dict.c callback in order to make
some incremental work (to send "\n" heartbeats to the master) while
flushing the old data from memory.

It is hard to write a regression test for this issue unfortunately. More
support for debugging in the Redis core would be needed in terms of
functionalities to simulate a slow DB loading / deletion.
 2013-12-10 18:42:22 +01:00
antirez b6610a569d dict.c: added optional callback to dictEmpty(). 
Redis hash table implementation has many non-blocking features like
incremental rehashing, however while deleting a large hash table there
was no way to have a callback called to do some incremental work.

This commit adds this support, as an optiona callback argument to
dictEmpty() that is currently called at a fixed interval (one time every
65k deletions).
 2013-12-10 18:18:36 +01:00
antirez 26cf5c8ac6 Log empty DB + Loading data into two separated messages. 2013-12-10 17:51:16 +01:00
antirez 303cc97ff9 Don't send more than 1 newline/sec while loading RDB. 2013-12-10 17:50:26 +01:00
antirez 75bf5a4a4a Slaves heartbeat while loading RDB files. 
Starting with Redis 2.8 masters are able to detect timed out slaves,
while before 2.8 only slaves were able to detect a timed out master.

Now that timeout detection is bi-directional the following problem
happens as described "in the field" by issue #1449:

1) Master and slave setup with big dataset.
2) Slave performs the first synchronization, or a full sync
   after a failed partial resync.
3) Master sends the RDB payload to the slave.
4) Slave loads this payload.
5) Master detects the slave as timed out since does not receive back the
   REPLCONF ACK acknowledges.

Here the problem is that the master has no way to know how much the
slave will take to load the RDB file in memory. The obvious solution is
to use a greater replication timeout setting, but this is a shame since
for the 0.1% of operation time we are forced to use a timeout that is
not what is suited for 99.9% of operation time.

This commit tries to fix this problem with a solution that is a bit of
an hack, but that modifies little of the replication internals, in order
to be back ported to 2.8 safely.

During the RDB loading time, we send the master newlines to avoid
being sensed as timed out. This is the same that the master already does
while saving the RDB file to still signal its presence to the slave.

The single newline is used because:

1) It can't desync the protocol, as it is only transmitted all or
nothing.
2) It can be safely sent while we don't have a client structure for the
master or in similar situations just with write(2).
 2013-12-10 15:40:36 +01:00
antirez 8d0083ba25 Handle inline requested terminated with just \n. 2013-12-10 15:40:33 +01:00
antirez fba0b23e72 Sentinel: fix reported role info sampling. 
The way the role change was recoded was not sane and too much
convoluted, causing the role information to be not always updated.

This commit fixes issue #1445.
 2013-12-06 12:49:27 +01:00
antirez dceaca1f69 Sentinel: fix reported role fields when master is reset. 
When there is a master address switch, the reported role must be set to
master so that we have a chance to re-sample the INFO output to check if
the new address is reporting the right role.

Otherwise if the role was wrong, it will be sensed as wrong even after
the address switch, and for enough time according to the role change
time, for Sentinel consider the master SDOWN.

This fixes isue #1446, that describes the effects of this bug in
practice.
 2013-12-06 12:49:23 +01:00
antirez bf307cfb4d Fixed typo in redis.conf. 2013-12-06 10:48:40 +01:00
Anurag Ramdasan 4f9d30b33b Grammar fix. 2013-12-05 18:54:01 +01:00
Anurag Ramdasan b67f39da09 fixed typo 2013-12-05 17:18:37 +01:00
Anurag Ramdasan 4c53178c6c Fixed grammar: 'usually' to 'usual' 2013-12-05 16:42:28 +01:00
antirez 7f6743a581 Fixed grammar: before H the article is a, not an. 2013-12-05 16:37:21 +01:00
antirez edae78999c Fixed typos in redis.conf file. 2013-12-05 16:30:15 +01:00
antirez 3d7263aa41 Removed old comments and dead code from freeClient(). 2013-12-03 13:54:15 +01:00
antirez 333453646c Grammar fix in freeClient(). 2013-12-03 13:40:51 +01:00
antirez 4d650a3baa Redis 2.8.2. 2013-12-02 16:07:46 +01:00
antirez dd0ac4ac72 Sentinel: don't write HZ when flushing config. 
See issue #1419.
 2013-12-02 15:56:42 +01:00
antirez 75347ada7f Sentinel: better time desynchronization. 
Sentinels are now desynchronized in a better way changing the time
handler frequency between 10 and 20 HZ. This way on average a
desynchronization of 25 milliesconds is produced that should be larger
enough compared to network latency, avoiding most split-brain condition
during the vote.

Now that the clocks are desynchronized, to have larger random delays when
performing operations can be easily achieved in the following way.
Take as example the function that starts the failover, that is
called with a frequency between 10 and 20 HZ and will start the
failover every time there are the conditions. By just adding as an
additional condition something like rand()%4 == 0, we can amplify the
desynchronization between Sentinel instances easily.

See issue #1419.
 2013-12-02 15:56:39 +01:00
antirez 83333b08d0 Stop writes on MISCONF only if instance is a master. 
From the point of view of the slave not accepting writes from the master
can only create a bigger consistency issue.
 2013-11-28 16:25:49 +01:00
antirez 50d140e90b SLAVEOF command refactored into a proper API. 
We now have replicationSetMaster() and replicationUnsetMaster() that can
be called in other contexts (for instance Redis Cluster).
 2013-11-28 16:19:16 +01:00
antirez 2eb8a46061 Reply to PING with error when there is a MISCONF state. 2013-11-28 16:16:58 +01:00
antirez a46f841df3 Sentinel: log vote received from other Sentinels. 2013-11-28 15:23:51 +01:00
huangz1990 e5c577e679 fix a bug in sentinel.c about pub/sub link 2013-11-26 15:11:58 +01:00
antirez dd239c37db Redis 2.8.1. 2013-11-25 11:21:49 +01:00
Salvatore Sanfilippo c61cc761d5 Merge pull request #1370 from yaauie/deprecate-redis-copy-script-2.8 
Deprecate redis copy script 2.8
 2013-11-25 02:10:32 -08:00
antirez d13635b2a9 Sentinel: fixes inverted strcmp() test preventing config updates. 
The result of this one-char bug was pretty serious, if the new master
had the same port of the previous master, but just a different IP
address, non-leader Sentinels would not be able to recognize the
configuration change.

This commit fixes issue #1394.

Many thanks to @shanemadden that reported the bug and helped
investigating it.
 2013-11-25 10:57:20 +01:00
antirez d240202261 Sentinel: fix type specifier for Hello msg generation. 
This fixes issue #1395.
 2013-11-25 10:25:10 +01:00
antirez e83741746c Fix false positive in memory efficiency test. 
Fixes issue #1298.
 2013-11-25 10:21:18 +01:00
antirez 4e6d34bd2c Redis 2.8.0. 2013-11-22 10:25:29 +01:00
antirez f75fccafe7 Sentinel: example sentinel.conf updated. 2013-11-21 17:07:10 +01:00
antirez 312ca4dacc Sentinel: different comments updated to new implementation. 2013-11-21 16:23:11 +01:00
antirez c84275ee60 Sentinel: cleanup around SENTINEL_INFO_VALIDITY_TIME. 2013-11-21 16:06:07 +01:00
antirez 0b2639123b Sentinel: removed mem leak and useless code. 2013-11-21 15:44:10 +01:00
antirez 750b007d7c Sentinel: manual failover works again. 2013-11-21 15:24:27 +01:00
antirez 4bd1dd1c53 Sentinel: test for writable config file. 
This commit introduces a funciton called when Sentinel is ready for
normal operations to avoid putting Sentinel specific stuff in redis.c.
 2013-11-21 15:24:22 +01:00
antirez 46e2f3468c Sentinel: check for disconnected links in sentinelSendHello(). 
Does not fix any bug as the test is performed by the caller, but better
to have the check.
 2013-11-21 15:24:15 +01:00
antirez 87098a39f0 Sentinel: Hello message sending code refactored. 2013-11-21 15:24:11 +01:00
antirez 9ecd819ecb Sentinel: select slave with best (greater) replication offset. 2013-11-21 15:24:08 +01:00
antirez eeb8cb3305 Sentinel: take the replication offset in slaves state. 2013-11-21 15:24:03 +01:00
antirez c517f72343 CONFIG REWRITE: don't add the signature if it already exists. 
At the end of the file, CONFIG REWRITE adds a comment line that:

    # Generated by CONFIG REWRITE

Followed by the additional config options required. However this was
added again and again at every rewrite in praticular conditions (when a
given set of options change in a given time during the time).

Now if it was alrady encountered, it is not added a second time.

This is especially important for Sentinel that rewrites the config at
every state change.
 2013-11-21 15:23:47 +01:00
antirez 812f76a850 Sentinel: distinguish between is-master-down-by-addr requests. 
Some are just to know if the master is down, and in this case the runid
in the request is set to "*", others are actually in order to seek for a
vote and get elected. In the latter case the runid is set to the runid
of the instance seeking for the vote.
 2013-11-21 15:22:55 +01:00
antirez fc93198ff9 Sentinel: various fixes to leader election implementation. 2013-11-21 15:22:52 +01:00
antirez 44b3684633 Sentinel: failover script execution fixed. 2013-11-21 15:22:48 +01:00
antirez 6d0400f569 Sentinel: no longer used defines removed. 2013-11-21 15:22:44 +01:00
antirez 1bbce5bf01 Sentinel: when writing config on disk, remember sentinels runid. 2013-11-21 15:22:27 +01:00
antirez 679dc8b09d Sentinel: arity of known-sentinel/slave is 4 not 3. 2013-11-21 15:22:21 +01:00
antirez 3e1fc6278d Sentinel: rewriteConfigSentinelOption() sub-iterators var typo fixed. 2013-11-21 15:22:18 +01:00
antirez 9a9c0cfaa6 Sentinel: call sentinelFlushConfig() to persist state when needed. 
Also the sentinel configuration rewriting was modified in order to
account for failover in progress, where we need to provide the promoted
slave address as master address, and the old master address as one of
the slaves address.
 2013-11-21 15:22:14 +01:00
antirez 93d924ff1c Sentinel: sentinelFlushConfig() to CONFIG REWRITE + fsync. 2013-11-21 15:22:11 +01:00
antirez a52909c5f2 Sentinel: CONFIG REWRITE support for Sentinel config. 2013-11-21 15:22:07 +01:00
antirez 8c3e197040 Sentinel: can-failover option removed, many comments fixed. 2013-11-21 15:22:02 +01:00
antirez 04b1fb0b1a Fix typo 'configuraiton' in rewriteConfigRewriteLine() comment. 2013-11-21 15:21:58 +01:00
antirez 0b9853ecdc Sentinel: added config options useful to take state on config rewrite. 
We'll use CONFIG REWRITE (internally) in order to store the new
configuration of a Sentinel after the internal state changes. In order
to do so, we need configuration options (that usually the user will not
touch at all) about config epoch of the master, Sentinels and Slaves
known for this master, and so forth.
 2013-11-21 15:21:55 +01:00
antirez 737062745d Sentinel: failover abort function simplified. 2013-11-21 15:21:50 +01:00
antirez 66b03c1a40 Sentinel: slaves reconfig delay modified. 
The time Sentinel waits since the slave is detected to be configured to
the wrong master, before reconfiguring it, is now the failover_timeout
time as this makes more sense in order to give the Sentinel performing
the failover enoung time to reconfigure the slaves slowly (if required
by the configuration).

Also we now PUBLISH more frequently the new configuraiton as this allows
to switch the reapprearing master back to slave faster.
 2013-11-21 15:21:46 +01:00
antirez 8ba31c218b Sentinel: failover restart time is now multiple of failover timeout. 
Also defaulf failover timeout changed to 3 minutes as the failover is a
fairly fast procedure most of the times, unless there are a very big
number of slaves and the user picked to configure them sequentially (in
that case the user should change the failover timeout accordingly).
 2013-11-21 15:21:41 +01:00
antirez ccaba966bc Sentinel: state machine and timeouts simplified. 2013-11-21 15:21:37 +01:00
antirez 3c4497e83c Sentinel: election timeout define. 2013-11-21 15:21:34 +01:00
antirez e15ba6a697 Sentinel: fix address of master in Hello messages. 
Once we switched configuration during a failover, we should advertise
the new address.

This was a serious race condition as the Sentinel performing the
failover for a moment advertised the old address with the new
configuration epoch: once trasmitted to the other Sentinels the broken
configuration would remain there forever, until the next failover
(because a greater configuration epoch is required to overwrite an older
one).
 2013-11-21 15:21:30 +01:00
antirez 1a6abe7d79 Sentinel: master address selection in get-master-address refactored. 2013-11-21 15:21:26 +01:00
antirez 0eeb0a0782 Sentinel: fix conditional to only affect slaves with wrong master. 2013-11-21 15:21:21 +01:00
antirez 64c8de8657 Sentinel: simplify and refactor slave reconfig code. 2013-11-21 15:20:56 +01:00
antirez 782f9cacaf Sentinel: reconfigure slaves to right master. 2013-11-21 15:20:44 +01:00
antirez 7dbc0a63f5 Sentinel: remember last time slave changed master. 2013-11-21 15:20:40 +01:00
antirez 612dbb2a91 Sentinel: redirect-to-master is not ok with new algorithm. 
Now Sentinel believe the current configuration is always the winner and
should be applied by Sentinels instead of trying to adapt our view of
the cluster based on what we observe.

So the only way to modify what a Sentinel believe to be the truth is to
win an election and advertise the new configuration via Pub / Sub with a
greater configuration epoch.
 2013-11-21 15:20:36 +01:00
antirez 4ccf807abc Sentinel: safer slave reconfig, master reported role should match. 2013-11-21 15:20:31 +01:00
antirez e98d82c639 Sentinel: role reporting fixed and added in SENTINEL output. 2013-11-21 15:20:25 +01:00
antirez be19e5450c Sentinel: being a master and reporting as slave is considered SDOWN. 2013-11-21 15:20:20 +01:00
antirez 8c551d65a1 Sentinel: make sure role_reported is always updated. 2013-11-21 15:20:15 +01:00
antirez 9577fed8a3 Sentinel: track role change time. Wait before reconfigurations. 2013-11-21 15:20:11 +01:00
antirez fc10fb17da Sentinel: fix no-down check in master->slave conversion code. 2013-11-21 15:20:07 +01:00
antirez b02ef3d59a Sentinel: readd slaves back after a master reset. 2013-11-21 15:19:56 +01:00
antirez 1a1dc3de38 Sentinel: sentinelResetMaster() new flag to avoid removing set of sentinels. 
This commit also removes some dead code and cleanup generic flags.
 2013-11-21 15:19:52 +01:00
antirez 9f0e52a13d Sentinel: receive Pub/Sub messages from slaves. 2013-11-21 15:19:49 +01:00
antirez f7604b4c07 Sentinel: change event name when converting master to slave. 2013-11-21 15:19:45 +01:00
antirez 1569326277 Sentinel: added config-epoch to SENTINEL masters output. 2013-11-21 15:19:41 +01:00
antirez 9f20780de6 Sentinel: new failover algo, desync slaves and update config epoch. 2013-11-21 15:19:34 +01:00
antirez 2488257ad8 Sentinel: when starting failover seek for votes ASAP. 2013-11-21 15:19:30 +01:00
antirez 6593f33222 Sentinel: +new-epoch events. 2013-11-21 15:19:26 +01:00
antirez 48acc675dd Sentinel: wait some time between failover attempts. 2013-11-21 15:19:22 +01:00
antirez 447c2787a0 Sentinel: allow to vote for myself. 2013-11-21 15:19:16 +01:00
antirez eba4775b5d Sentinel: fix PUBLISH to masters and slaves. 2013-11-21 15:19:11 +01:00
antirez b95c6ed7b7 Sentinel: epoch introduced in leader vote. 2013-11-21 15:18:52 +01:00
antirez 663d79c0d5 Sentinel: leadership handling changes WIP. 
Changes to leadership handling.

Now the leader gets selected by every Sentinel, for a specified epoch,
when the SENTINEL is-master-down-by-addr is sent.

This command now includes the runid and the currentEpoch of the instance
seeking for a vote. The Sentinel only votes a single time in a given
epoch.

Still a work in progress, does not even compile at this stage.
 2013-11-21 15:18:45 +01:00
antirez b72985d7f7 Sentinel: handle Hello messages received via slaves correctly. 
Even when messages are received via the slave, we should perform
operations (like adding a new Sentinel) in the context of the master.
 2013-11-21 15:18:41 +01:00
antirez fe7f96f18c Sentinel: remove code not useful in the new design. 2013-11-21 15:18:36 +01:00
antirez be2ef1b59f Sentinel: epoch introduced. 
Sentinel state now includes the idea of current epoch and config epoch.
In the Hello message, that is now published both on masters and slaves,
a Sentinel no longer just advertises itself but also broadcasts its
current view of the configuration: the master name / ip / port and its
current epoch.

Sentinels receiving such information switch to the new master if the
configuration epoch received is newer and the ip / port of the master
are indeed different compared to the previos ones.
 2013-11-21 15:18:31 +01:00
antirez 98682cd178 Log to what master a slave is going to connect to. 2013-11-11 09:25:41 +01:00
antirez 68b911478f Fixed typo in release candidate notes. 2013-11-10 10:25:09 +01:00
Ryan Biesemeyer 7d30b3121d Deprecate utils/redis-copy.rb in favor of redis-copy gem 2013-11-08 06:33:04 +00:00
antirez 875bc51909 Fix broken rdbWriteRaw() return value check in rdb.c. 
Thanks to @PhoneLi for reporting.
 2013-11-07 23:54:46 +01:00
antirez c874e39c45 Sentinel: sentinelSendSlaveOf() was missing a var and the prototype. 2013-11-06 11:29:57 +01:00
antirez 1767998751 Sentinel: increment pending_commands counter in two more places. 
AUTH and SCRIPT KILL were sent without incrementing the pending commands
counter. Clearly this needs some kind of wrapper doing it for the caller
in order to be less bug prone.
 2013-11-06 11:29:53 +01:00
antirez 97810c45e8 Sentinel: always send CONFIG REWRITE when changing instance role. 
This change makes Sentinel less fragile about a number of failure modes.

This commit also fixes a different bug as a side effect, SLAVEOF command
was sent multiple times without incrementing the pending commands count.
 2013-11-06 11:29:49 +01:00
antirez f7f97bf730 Redis 2.7.106 (2.8 Release Candidate 6). 2013-11-06 09:24:26 +01:00
antirez 3a66e0c157 Fixed typo in parseScanCursorOrReply(): ad -> at. 
Thanks to @badboy for reporting it.
 2013-11-05 17:25:34 +01:00
antirez 060d56e7eb SCAN code refactored to parse cursor first. 
The previous implementation of SCAN parsed the cursor in the generic
function implementing SCAN, SSCAN, HSCAN and ZSCAN.

The actual higher-level command implementation only checked for empty
keys and return ASAP in that case. The result was that inverting the
arguments of, for instance, SSCAN for example and write:

    SSCAN 0 key

Instead of

    SSCAN key 0

Resulted into no error, since 0 is a non-existing key name very likely.
Just the iterator returned no elements at all.

In order to fix this issue the code was refactored to extract the
function to parse the cursor and return the error. Every higher level
command implementation now parses the cursor and later checks if the key
exist or not.
 2013-11-05 17:25:29 +01:00
antirez 162acd8ac2 SCAN: when iterating ziplists or intsets always return cursor of 0. 
The previous implementation assumed that the first call always happens
with cursor set to 0, this may not be the case, and we want to return 0
anyway otherwise the (broken) client code will loop forever.
 2013-11-05 17:24:33 +01:00
antirez 406be0ea28 Use strtoul() instead of sscanf() in SCAN implementation. 2013-11-05 17:24:30 +01:00
antirez 4b615eea8f Added tests for [SHZ]SCAN with MATCH. 2013-11-05 17:24:26 +01:00
antirez 54a5a7dff8 HSCAN/ZSCAN: skip value when matching. 
This fixes issue #1360 and #1362.
 2013-11-05 17:24:22 +01:00
antirez bebbc7f9f7 Pass int64_t to intsetGet() instead of long long. 2013-11-05 17:24:17 +01:00
antirez 736e343509 removed not used vars in dictScan(). 2013-11-05 17:24:12 +01:00
antirez 59b30b327d SCAN: no sdsEncodedObject() API in Redis 2.8. 2013-10-31 18:22:29 +01:00
antirez 3d3e350b12 SSCAN with integer encoded object test improved. 2013-10-31 10:38:56 +01:00
antirez 3cc7d486bc Inverted variable boolean value and name after scanGenericCommand() refactoring. 2013-10-31 10:38:52 +01:00
antirez 06455014aa scanGenericCommand() refactoring and handling of integer encoded elements. 
This commit fixes issue #1354.
 2013-10-31 10:38:48 +01:00
antirez 141f30f421 Regression test added for [SHZ]SCAN issue #1354. 2013-10-31 10:38:45 +01:00
antirez c53cab5dde Test: added a SCAN test trying to trigger HT resize. 2013-10-30 16:51:48 +01:00
antirez ab6f4195dd Test: added ZSCAN test. 2013-10-30 16:27:43 +01:00
antirez 13e879c9cf Test: added HSCAN test. 2013-10-30 16:27:39 +01:00
antirez 9ea69a58ac Test: added SSCAN test. 2013-10-30 16:27:36 +01:00
antirez bd1962acc4 SCAN test keys sorting turned into more idiomatic Tcl. 2013-10-30 16:27:31 +01:00
antirez 02617b6e92 SCAN: tests moved to unit/scan.tcl. 2013-10-30 16:27:28 +01:00
antirez bb2df795e0 Aesthetic fix (missing space) into HSCAN and ZSCAN implementations. 
Thanks to @badboy for reporting.
 2013-10-29 16:21:21 +01:00
antirez 35816a2e4d ZSCAN implemented. 2013-10-29 16:21:18 +01:00
antirez 1cc7e646af HSCAN implemented. 2013-10-29 16:21:13 +01:00
antirez 24ecabc995 SSCAN implemented. 2013-10-29 16:21:01 +01:00
antirez 17785f40b5 dictScan(): empty hash table requires special handling. 2013-10-29 16:20:13 +01:00
antirez 7f063b1c8c SCAN is a random command and does not require output sorting. 
Sorting the output helps when we want to turn a non-deterministic into a
deterministic command, in that case this is not possible.
 2013-10-29 16:20:10 +01:00
antirez 9e62d9e79c SCAN: refactored into scanGenericCommand. 
The new implementation is capable of iterating the keyspace but also
sets, hashes, and sorted sets, and can be used to implement SSCAN, ZSCAN
and HSCAN.
 2013-10-29 16:20:06 +01:00
antirez 1a06bbe430 Fixed typos in dictScan() comment. 2013-10-29 16:20:02 +01:00
antirez 0880bec4e2 dictScan() algorithm documented. 2013-10-29 16:19:57 +01:00
antirez 1d659784f4 SCAN: stay inside 80 cols. 2013-10-29 16:19:54 +01:00
antirez 854b8be616 Revert "Fixed typo in SCAN comment. iff -> if." 
Probably here Pieter means "if and only if".

This reverts commit 43fdf3b404.
 2013-10-29 16:19:49 +01:00
antirez 5fa5153bf5 SCAN: simplify keys list cleanup using listSetFreeMethod(). 2013-10-29 16:19:45 +01:00
antirez 108c6e6734 SCAN: Fix test after option renamed from PATTERN to MATCH. 2013-10-29 16:19:41 +01:00
antirez 21da9b06a4 SCAN: improve variable names for readability. 2013-10-29 16:19:37 +01:00
antirez 907e06cd61 SCAN: remove additional newlines to conform to Redis code base. 2013-10-29 16:19:33 +01:00
antirez b44f1589a4 SCAN: remove useless assertion, already enforced by command table. 2013-10-29 16:19:28 +01:00
antirez d1c1f62ed5 SCAN: use define REDIS_LONGSTR_SIZE instead of fixed len. 2013-10-29 16:19:25 +01:00
antirez 8cdaf604ce Fixed typo in SCAN comment. iff -> if. 2013-10-29 16:19:21 +01:00
antirez 33409b4c6c SCAN option name changed: pattern -> match. 2013-10-29 16:19:17 +01:00
Pieter Noordhuis fd22106a98 Fix error in scan algorithm 
The irrelevant bits shouldn't be masked to 1. This can result in slots being
skipped when the hash table is resized between calls to the iterator.
 2013-10-29 16:19:12 +01:00
Pieter Noordhuis 8c4ff679fe SCAN requires at least 1 argument 2013-10-29 16:19:08 +01:00
Pieter Noordhuis 0dd95e23ab Add SCAN command 2013-10-29 16:18:24 +01:00
antirez 9546f7846b redis-benchmark: update help for new __rand_int__ form. 2013-10-28 18:14:11 +01:00
antirez 508d3c1bb7 Redis 2.7.105 (2.8 Release Candidate 5). 2013-10-09 13:08:42 +02:00
antirez df0c96002d Replication: install the write handler when reusing a cached master. 
Sometimes when we resurrect a cached master after a successful partial
resynchronization attempt, there is pending data in the output buffers
of the client structure representing the master (likely REPLCONF ACK
commands).

If we don't reinstall the write handler, it will never be installed
again by addReply*() family functions as they'll assume that if there is
already data pending, the write handler is already installed.

This bug caused some slaves after a successful partial sync to never
send REPLCONF ACK, and continuously being detected as timing out by the
master, with a disconnection / reconnection loop.
 2013-10-04 16:14:57 +02:00
antirez d7fa6d9aba Replication: fix master timeout. 
Since we started sending REPLCONF ACK from slaves to masters, the
lastinteraction field of the client structure is always refreshed as
soon as there is room in the socket output buffer, so masters in timeout
are detected with too much delay (the socket buffer takes a lot of time
to be filled by small REPLCONF ACK <number> entries).

This commit only counts data received as interactions with a master,
solving the issue.
 2013-10-04 13:01:49 +02:00
antirez c8c1006cf4 PSYNC: safer handling of PSYNC requests. 
There was a bug that over-esteemed the amount of backlog available,
however this could only happen when a slave was asking for an offset
that was in the "future" compared to the master replication backlog.

Now this case is handled well and logged as an incident in the master
log file.
 2013-10-04 12:27:40 +02:00
antirez 81f614ef6f Add REWRITE to CONFIG subcommands help message. 2013-10-04 12:27:36 +02:00
Michel Martens 20c506f432 Document the redis-cli --csv option. 2013-09-26 10:12:51 +02:00
antirez af967ff9cb Allow AUTH / PING when disconnected from slave and serve-stale-data is no. 2013-09-17 09:46:40 +02:00
antirez 2d55f1ca7f redis-cli: fix big keys search when the key no longer exist. 
The code freed a reply object that was never created, resulting in a
segfault every time randomkey returned a key that was deleted before we
queried it for size.
 2013-09-04 10:37:55 +02:00
Maxim Zakharov 9a0f27a5d0 mistype fixed 2013-09-03 15:15:42 +02:00
Maxim Zakharov c713b1ebdd A mistype fixed 2013-09-03 15:15:34 +02:00
antirez 56ce068664 Redis 2.7.104 (2.8 Release Candidate 4). 2013-08-30 09:52:18 +02:00
antirez 23acf93f97 Test: Lua stack leak regression test added. 2013-08-30 08:59:16 +02:00
antirez 0ea9a20d47 Test: added a memory efficiency test. 2013-08-30 08:48:07 +02:00
antirez 752b1fca90 Fixed critical memory leak from EVAL. 
Multiple missing calls to lua_pop prevented the error handler function
pushed on the stack for lua_pcall() to be popped before returning,
causing a memory leak in almost all the code paths of EVAL (both
successful calls and calls returning errors).

This caused two issues: Lua leaking memory (and this was very visible
from INFO memory output, as the 'used_memory_lua' field reported an
always increasing amount of memory used), and as a result slower and
slower GC cycles resulting in all the CPU being used.

Thanks to Tanguy Le Barzic for noticing something was wrong with his 2.8
slave, and for creating a testing EC2 environment where I was able to
investigate the issue.
 2013-08-29 11:49:28 +02:00
Salvatore Sanfilippo 3e62d7c308 Merge pull request #1244 from yihuang/2.8 
fix lua_cmsgpack pack map as array
 2013-08-27 06:15:32 -07:00
antirez a57294b339 Fix an hypothetical issue in processMultibulkBuffer(). 2013-08-27 13:00:14 +02:00
antirez facc5c0661 tryObjectEncoding(): optimize sds strings if possible. 
When no encoding is possible, at least try to reallocate the sds string
with one that does not waste memory (with free space at the end of the
buffer) when the string is large enough.
 2013-08-27 12:36:09 +02:00
antirez 79a1d335c8 tryObjectEncoding(): don't call stringl2() for too big strings. 
We are sure that a string that is longer than 21 chars cannot be
represented by a 64 bit signed integer, as -(2^64) is 21 chars:

strlen(-18446744073709551616) => 21
 2013-08-27 12:30:42 +02:00
antirez 0f32c37fc9 Fix DEBUG SDSLEN after 2.8 back port. 2013-08-27 12:26:02 +02:00
antirez df21771c69 Don't over-allocate the sds string for large bulk requests. 
The call to sdsMakeRoomFor() did not accounted for the amount of data
already present in the query buffer, resulting into over-allocation.
 2013-08-27 12:22:12 +02:00
antirez f05b1f6889 DEBUG SDSLEN added. 
This command is only useful for low-level debugging of memory issues due
to sds wasting memory as empty buffer at the end of the string.
 2013-08-27 12:22:07 +02:00
antirez 9c3e6c3aa0 Update server.lastbgsave_status when fork() fails. 2013-08-27 10:16:25 +02:00
antirez 9df63fcabf Only run the fast active expire cycle if master & enabled. 2013-08-27 09:31:43 +02:00
antirez f5903846bc Opening TCP listening ports refactored into a function. 2013-08-22 14:06:54 +02:00
antirez bd6327855b Print error message when can't bind * on any address. 2013-08-22 13:03:07 +02:00
antirez 2c94d80f58 Fix for issue #1214 simplified. 2013-08-21 11:41:35 +02:00
Allan 778f753deb fixed initServer fail while having no IPv6 nor IPv4 2013-08-21 11:41:30 +02:00
Allan 6c2b34a465 fixed initServer failed if no IPV4 or no IPV6 2013-08-21 11:41:25 +02:00
Allan 0a4656d63f fixed bug issue of #1213 2013-08-21 11:41:21 +02:00
antirez 7c861340b1 Use printf %zu specifier to print private_dirty. 2013-08-20 12:09:36 +02:00
antirez 2c018b14fb dictFingerprint(): cast pointers to integer of same size. 2013-08-20 12:09:31 +02:00
antirez 6a88a84ec7 Redis 2.7.103 (2.8 Release Candidate 3). 2013-08-19 18:04:32 +02:00
antirez 5621272575 Revert "Fixed type in dict.c comment: 265 -> 256." 
This reverts commit 009515f914.
 2013-08-19 17:25:57 +02:00
antirez 009515f914 Fixed type in dict.c comment: 265 -> 256. 2013-08-19 15:11:07 +02:00
antirez 837817b2b7 assert.h replaced with redisassert.h when appropriate. 
Also a warning was suppressed by including unistd.h in redisassert.h
(needed for _exit()).
 2013-08-19 15:02:16 +02:00
antirez bd353d9a3c Added redisassert.h as drop in replacement for assert.h. 
By using redisassert.h version of assert() you get stack traces in the
log instead of a process disappearing on assertions.
 2013-08-19 15:02:13 +02:00
antirez c7da5fc6c1 dictFingerprint() fingerprinting made more robust. 
The previous hashing used the trivial algorithm of xoring the integers
together. This is not optimal as it is very likely that different
hash table setups will hash the same, for instance an hash table at the
start of the rehashing process, and at the end, will have the same
fingerprint.

Now we hash N integers in a smarter way, by summing every integer to the
previous hash, and taking the integer hashing again (see the code for
further details). This way it is a lot less likely that we get a
collision. Moreover this way of hashing explicitly protects from the
same set of integers in a different order to hash to the same number.

This commit is related to issue #1240.
 2013-08-19 15:02:10 +02:00
antirez 8368550733 Fix comments for correctness in zunionInterGenericCommand(). 
Related to issue #1240.
 2013-08-19 15:02:06 +02:00
antirez 3ad87c652b Properly init/release iterators in zunionInterGenericCommand(). 
This commit does mainly two things:

1) It fixes zunionInterGenericCommand() by removing mass-initialization
of all the iterators used, so that we don't violate the unsafe iterator
API of dictionaries. This fixes issue #1240.

2) Since the zui* APIs required the allocator to be initialized in the
zsetopsrc structure in order to use non-iterator related APIs, this
commit fixes this strict requirement by accessing objects directly via
the op->subject->ptr pointer we have to the object.
 2013-08-19 15:02:02 +02:00
antirez 05379f49ee dict.c iterator API misuse protection. 
dict.c allows the user to create unsafe iterators, that are iterators
that will not touch the dictionary data structure in any way, preventing
copy on write, but at the same time are limited in their usage.

The limitation is that when itearting with an unsafe iterator, no call
to other dictionary functions must be done inside the iteration loop,
otherwise the dictionary may be incrementally rehashed resulting into
missing elements in the set of the elements returned by the iterator.

However after introducing this kind of iterators a number of bugs were
found due to misuses of the API, and we are still finding
bugs about this issue. The bugs are not trivial to track because the
effect is just missing elements during the iteartion.

This commit introduces auto-detection of the API misuse. The idea is
that an unsafe iterator has a contract: from initialization to the
release of the iterator the dictionary should not change.

So we take a fingerprint of the dictionary state, xoring a few important
dict properties when the unsafe iteartor is initialized. We later check
when the iterator is released if the fingerprint is still the same. If it
is not, we found a misuse of the iterator, as not allowed API calls
changed the internal state of the dictionary.

This code was checked against a real bug, issue #1240.

This is what Redis prints (aborting) when a misuse is detected:

Assertion failed: (iter->fingerprint == dictFingerprint(iter->d)),
function dictReleaseIterator, file dict.c, line 587.
 2013-08-19 15:01:58 +02:00
yihuang 706056ca16 fix lua_cmsgpack pack map as array 2013-08-13 17:48:40 +08:00
antirez 14a1cba3e3 Use precomptued objects for bulk and mbulk prefixes. 2013-08-12 12:43:26 +02:00
antirez 004f00bfa4 replicationFeedSlaves() func name typo: feedReplicationBacklogWithObject -> feedReplicationBacklog. 2013-08-12 12:38:52 +02:00
antirez 45d4e06e93 replicationFeedSlave() reworked for correctness and speed. 
The previous code using a static buffer as an optimization was lame:

1) Premature optimization, actually it was *slower* than naive code
   because resulted into the creation / destruction of the object
   encapsulating the output buffer.
2) The code was very hard to test, since it was needed to have specific
   tests for command lines exceeding the size of the static buffer.
3) As a result of "2" the code was bugged as the current tests were not
   able to stress specific corner cases.

It was replaced with easy to understand code that is safer and faster.
 2013-08-12 12:31:20 +02:00
antirez 3eab283b65 Fix a PSYNC bug caused by a variable name typo. 2013-08-12 11:51:03 +02:00
antirez 0e72cc601f Fix sdsempty() prototype in sds.h. 2013-08-12 11:38:28 +02:00
antirez 689829f2e1 redis-benchmark: changes to random arguments substitution. 
Before this commit redis-benchmark supported random argumetns in the
form of :rand:000000000000. In every string of that form, the zeros were
replaced with a random number of 12 digits at every command invocation.

However this was far from perfect as did not allowed to generate simply
random numbers as arguments, there was always the :rand: prefix.

Now instead every argument in the form __rand_int__ is replaced with a
12 digits number. Note that "__rand_int__" is 12 characters itself.

In order to implement the new semantic, it was needed to change a few
thigns in the internals of redis-benchmark, as new clients are created
cloning old clients, so without a stable prefix such as ":rand:" the old
way of cloning the client was no longer able to understand, from the old
command line, what was the position of the random strings to substitute.

Now instead a client structure is passed as a reference for cloning, so
that we can directly clone the offsets inside the command line.
 2013-08-08 16:47:18 +02:00
antirez 4fe67cc103 redis-benchmark: replace snprintf()+memcpy with faster code. 
This change was profiler-driven, but the actual effect is hard to
measure in real-world redis benchmark runs.
 2013-08-08 14:31:54 +02:00
Jan-Erik Rediger a256b83448 Little typo 2013-08-07 16:09:33 +02:00
antirez a79862de2a redis-benchmark: fix memory leak introduced by 346256f 2013-08-07 16:09:13 +02:00
antirez bcc7965931 redis-benchmark: max pipeline length hardcoded limit removed. 2013-08-07 16:09:09 +02:00
antirez 46c1bafeab redis-benchmark: fix db selection when :rand: feature is used. 2013-08-07 11:22:33 +02:00
antirez 151ad68540 redis-benchmark: ability to SELECT a specifid db number. 2013-08-07 11:22:28 +02:00
antirez fa48b1fa32 Add per-db average TTL information in INFO output. 
Example:

db0:keys=221913,expires=221913,avg_ttl=655

The algorithm uses a running average with only two samples (current and
previous). Keys found to be expired are considered at TTL zero even if
the actual TTL can be negative.

The TTL is reported in milliseconds.
 2013-08-06 15:36:43 +02:00
antirez 31d0f34100 activeExpireCycle(): fix about fast cycle early start. 
We don't want to repeat a fast cycle too soon, the previous code was
broken, we need to wait two times the period *since* the start of the
previous cycle in order to avoid there is an even space between cycles:

.-> start                   .-> second start
|                           |
+-------------+-------------+--------------+
| first cycle |    pause    | second cycle |
+-------------+-------------+--------------+

The second and first start must be PERIOD*2 useconds apart hence the *2
in the new code.
 2013-08-06 15:36:39 +02:00
antirez 00c8cfef74 Some activeExpireCycle() refactoring. 2013-08-06 15:36:35 +02:00
antirez 8686a70303 Remove dead code and fix comments for new expire code. 2013-08-06 15:36:31 +02:00
antirez d54e373b84 Darft #2 for key collection algo: more improvements. 
This commit makes the fast collection cycle time configurable, at
the same time it does not allow to run a new fast collection cycle
for the same amount of time as the max duration of the fast
collection cycle.
 2013-08-06 15:36:27 +02:00
antirez 500155b91b Draft #1 of a new expired keys collection algorithm. 
The main idea here is that when we are no longer to expire keys at the
rate the are created, we can't block more in the normal expire cycle as
this would result in too big latency spikes.

For this reason the commit introduces a "fast" expire cycle that does
not run for more than 1 millisecond but is called in the beforeSleep()
hook of the event loop, so much more often, and with a frequency bound
to the frequency of executed commnads.

The fast expire cycle is only called when the standard expiration
algorithm runs out of time, that is, consumed more than
REDIS_EXPIRELOOKUPS_TIME_PERC of CPU in a given cycle without being able
to take the number of already expired keys that are yet not collected
to a number smaller than 25% of the number of keys.

You can test this commit with different loads, but a simple way is to
use the following:

Extreme load with pipelining:

redis-benchmark -r 100000000 -n 100000000  \
        -P 32 set ele:rand:000000000000 foo ex 2

Remove the -P32 in order to avoid the pipelining for a more real-world
load.

In another terminal tab you can monitor the Redis behavior with:

redis-cli -i 0.1 -r -1 info keyspace

and

redis-cli --latency-history

Note: this commit will make Redis printing a lot of debug messages, it
is not a good idea to use it in production.
 2013-08-06 15:36:23 +02:00
antirez cc94697284 Redis 2.7.102 (2.8 Release Candidate 2). 2013-07-30 20:20:02 +02:00
antirez f8e43aba78 Test: regression test for issue #1221. 2013-07-29 17:40:18 +02:00
antirez 13f7ade551 Fix replicationFeedSlaves() off-by-one bug. 
This fixes issue #1221.
 2013-07-28 12:50:35 +02:00
antirez 78644f5e3e Remove dead variable bothsds from object.c. 
Thanks to @run and @badboy for spotting this.
Triva: clang was not able to provide me a warning about that when
compiling.

This closes #1024 and #1207, committing the change myself as the pull
requests no longer apply cleanly after other changes to the same
function.
 2013-07-28 11:02:20 +02:00
antirez aea8020554 Use latest sds.c in the hiredis library under deps. 2013-07-25 10:34:02 +02:00
antirez c3e1724a57 Ignore sdsrange return value. 2013-07-24 18:59:36 +02:00
antirez f899ab55ca sdsrange() does not need to return a value. 
Actaully the string is modified in-place and a reallocation is never
needed, so there is no need to return the new sds string pointer as
return value of the function, that is now just "void".
 2013-07-24 11:22:52 +02:00
antirez a1d37ba469 Inline protocol improved to accept quoted strings. 2013-07-24 10:38:05 +02:00
antirez 57c8e026b9 Every function inside sds.c is now commented. 2013-07-23 16:36:14 +02:00
antirez f3b44f291d Merge remote-tracking branch 'origin/2.8' into 2.8 2013-07-19 14:55:49 +02:00
antirez 5d0ad2f983 Fixed typo in 2.8 release notes. 2013-07-18 16:10:31 +02:00
Salvatore Sanfilippo 349f6349be Merge pull request #1203 from ronnix/patch-1 
Fix a few typos in release notes
 2013-07-18 03:27:41 -07:00
Ronan Amicel 5b19e9d433 Fix a few typos in release notes 2013-07-18 12:23:47 +02:00
antirez 3cb3714e99 Redis 2.7.101 (2.8 Release Candidate 1). 2013-07-18 11:26:53 +02:00
antirez a50635bb6c addReplyDouble(): format infinite in a libc agnostic way. 
There are systems that when printing +/- infinte with printf-family
functions will not use the usual "inf" "-inf", but different strings.
Handle that explicitly.

Fixes issue #930.
 2013-07-17 15:04:22 +02:00
antirez 2e449d42e0 Fixed typo in rio.h, simgle -> single. 2013-07-16 15:43:41 +02:00
yoav 9f6f436a51 Chunked loading of RDB to prevent redis from stalling reading very large keys. 2013-07-16 15:41:59 +02:00
antirez 112e763618 Make sure that ZADD can accept the full range of double values. 
This fixes issue #1194, that contains many details.

However in short, it was possible for ZADD to not accept as score values
that was however possible to obtain with multiple calls to ZINCRBY, like
in the following example:

redis 127.0.0.1:6379> zadd k 2.5e-308 m
(integer) 1
redis 127.0.0.1:6379> zincrby k -2.4e-308 m
"9.9999999999999694e-310"
redis 127.0.0.1:6379> zscore k m
"9.9999999999999694e-310"
redis 127.0.0.1:6379> zadd k 9.9999999999999694e-310 m1
(error) ERR value is not a valid float

The problem was due to strtod() returning ERANGE in the following case
specified by POSIX:

"If the correct value would cause an underflow, a value whose magnitude
is no greater than the smallest normalized positive number in the return
type shall be returned and errno set to [ERANGE].".

Now instead the returned value is accepted even when ERANGE is returned
as long as the return value of the function is not negative or positive
HUGE_VAL or zero.
 2013-07-16 15:08:57 +02:00
Ted Nyman fe04710908 Make sure the log standardizes on 'timeout' 2013-07-12 23:12:27 +02:00
antirez 98757b4a6e Use the environment locale for strcoll() collation. 2013-07-12 13:39:48 +02:00
antirez 18fabeb264 SORT ALPHA: use collation instead of binary comparison. 
Note that we only do it when STORE is not used, otherwise we want an
absolutely locale independent and binary safe sorting in order to ensure
AOF / replication consistency.

This is probably an unexpected behavior violating the least surprise
rule, but there is currently no other simple / good alternative.
 2013-07-12 13:39:44 +02:00
antirez d8fcbb6645 Fixed compareStringObject() and introduced collateStringObject(). 
compareStringObject was not always giving the same result when comparing
two exact strings, but encoded as integers or as sds strings, since it
switched to strcmp() when at least one of the strings were not sds
encoded.

For instance the two strings "123" and "123\x00456", where the first
string was integer encoded, would result into the old implementation of
compareStringObject() to return 0 as if the strings were equal, while
instead the second string is "greater" than the first in a binary
comparison.

The same compasion, but with "123" encoded as sds string, would instead
return a value < 0, as it is correct. It is not impossible that the
above caused some obscure bug, since the comparison was not always
deterministic, and compareStringObject() is used in the implementation
of skiplists, hash tables, and so forth.

At the same time, collateStringObject() was introduced by this commit, so
that can be used by SORT command to return sorted strings usign
collation instead of binary comparison. See next commit.
 2013-07-12 13:39:40 +02:00
Jan-Erik Rediger 1bcbb7a90c Wrap IPv6 in brackets in the prompt. 2013-07-12 10:58:47 +02:00
antirez 1e23848ed3 Sentinel: embed IPv6 address into [] when naming slave/sentinel instance. 2013-07-11 17:10:09 +02:00
antirez 076f6395b9 Sentinel: use comma as separator to publish hello messages. 
We use comma to play well with IPv6 addresses, but the implementation is
still able to parse the old messages separated by colons.
 2013-07-11 17:09:44 +02:00
antirez 4cfd70e903 hiredis: minimal IPv6 support. 2013-07-11 17:09:35 +02:00
antirez 98d0abcecd Sentinel: make sure published addr/id buffer is large enough. 
With ipv6 support we need more space, so we account for the IP address
max size plus what we need for the Run ID, port, flags.
 2013-07-11 17:09:30 +02:00
antirez 8669e70921 anet.c: save some vertical space. 2013-07-11 17:09:25 +02:00
antirez 9e089e7c5d anet.c: use SO_REUSEADDR when creating listening sockets. 
It used to be ok, but the socket option was removed when adding IPv6
support.
 2013-07-11 17:09:22 +02:00
antirez cdf801d92e Use getClientPeerId() for MONITOR implementation. 2013-07-11 17:09:18 +02:00
antirez 3472d045d8 getClientPeerId() refactored into two functions. 2013-07-11 17:09:14 +02:00
antirez da18366609 getClientPeerId() now reports errors. 
We now also use it in CLIENT KILL implementation.
 2013-07-11 17:09:10 +02:00
antirez 4fa68b2815 getClientPeerID introduced. 
The function returns an unique identifier for the client, as ip:port for
IPv4 and IPv6 clients, or as path:0 for Unix socket clients.

See the top comment in the function for more info.
 2013-07-11 17:09:04 +02:00
antirez a7451c1b6d All IP string repr buffers are now REDIS_IP_STR_LEN bytes. 2013-07-11 17:07:52 +02:00
antirez 2e75d3947c IPv6: bind IPv4 and IPv6 interfaces by default. 2013-07-11 17:07:41 +02:00
antirez b5423b099c Fix old anetPeerToString() API call in replication.c 2013-07-11 17:07:38 +02:00
antirez 1e7949227c Revert "Document port6 and bind6 config options." 
IPv6 support is not going to use IPv6 specific options, just it will be
possible to specify all the ipv4 / ipv6 addresses of the interfaces to
bind, otherwise connections will be accepted from all the interfaces in
both IPv4 and IPv6 addresses.

This reverts commit 93570e179e.
 2013-07-11 17:07:34 +02:00
Geoff Garside 0d8f254359 Add IPv6 support to sentinel.c. 
This has been done by exposing the anetSockName() function anet.c
to be used when the sentinel is publishing its existence to the masters.

This implementation is very unintelligent as it will likely break if used
with IPv6 as the nested colons will break any parsing of the PUBLISH string
by the master.
 2013-07-11 17:07:31 +02:00
Geoff Garside 5938126c98 Cleanup main() and BACKTRACE mistaken pulled while rebasing. 2013-07-11 17:07:27 +02:00
Geoff Garside d3bf85423a Fix calls to anetPeerToString() missing buffer size. 2013-07-11 17:07:22 +02:00
Geoff Garside 47620a3186 Document port6 and bind6 config options. 
Add commented port6 and bind6 options to default redis.conf file.
 2013-07-11 17:07:18 +02:00
Geoff Garside fd8a6ae7a6 Add anetTcp6Server() function. 
Refactor the common code from anetTcpServer into internal function which
can be used by both anetTcpServer and anetTcp6Server.
 2013-07-11 17:07:13 +02:00
Geoff Garside f638236902 Add static anetV6Only() function. 
This function sets the IPV6_V6ONLY option to 1 to use separate stack
IPv6 sockets.
 2013-07-11 17:07:07 +02:00
Geoff Garside 71795d4e7e Change anetTcpGenericConnect to use AF_UNSPEC. 
This allows anetTcpGenericConnect to try to connect to AF_INET6
addresses in addition to any resolved AF_INET addresses.
 2013-07-11 17:07:03 +02:00
Geoff Garside d37d006cd2 Mark places that might want changing for IPv6. 
Any places which I feel might want to be updated to work differently
with IPv6 have been marked with a comment starting "IPV6:".

Currently the only comments address places where an IP address is
combined with a port using the standard : separated form. These may want
to be changed when printing IPv6 addresses to wrap the address in []
such as

	[2001:db8::c0:ffee]:6379

instead of

	2001:db8::c0:ffee:6379

as the latter format is a technically valid IPv6 address and it is hard
to distinguish the IPv6 address component from the port unless you know
the port is supposed to be there.
 2013-07-11 17:06:55 +02:00
Geoff Garside 57e2b23dfb Expand ip char buffers which are too small for v6. 
Increase the size of character buffers being used to store printable IP
addresses so that they can safely store IPv6 addresses.
 2013-07-11 17:06:43 +02:00
Geoff Garside 39d57c057f Mark ip string buffers which could be reduced. 
In two places buffers have been created with a size of 128 bytes which
could be reduced to INET6_ADDRSTRLEN to still hold a full IP address.
These places have been marked as they are presently big enough to handle
the needs of storing a printable IPv6 address.
 2013-07-11 17:06:35 +02:00
Geoff Garside f1151f869e Update anetPeerToString to handle AF_INET6 addrs. 
Change the sockaddr_in to sockaddr_storage which is capable of storing
both AF_INET and AF_INET6 sockets. Uses the sockaddr_storage ss_family
to correctly return the printable IP address and port.
 2013-07-11 17:05:21 +02:00
Geoff Garside fefa14914e Update anetTcpAccept to handle AF_INET6 addresses. 
Change the sockaddr_in to sockaddr_storage which is capable of storing
both AF_INET and AF_INET6 sockets. Uses the sockaddr_storage ss_family
to correctly return the printable IP address and port.
 2013-07-11 17:05:16 +02:00
Geoff Garside 77ddec8582 Update anetResolve to resolve AF_INET6 as well. 
Change the getaddrinfo(3) hints family from AF_INET to AF_UNSPEC to
allow resolution of IPv6 addresses as well as IPv4 addresses. The
function will return the IP address of whichever address family is
preferenced by the operating system. Most current operating systems
will preference AF_INET6 over AF_INET.

Unfortunately without attempting to establish a connection to the
remote address we can't know if the host is capable of using the
returned IP address. It might be desirable to have anetResolve
accept an additional argument specifying the AF_INET/AF_INET6 address
the caller would like to receive. Currently though it does not appear
as though the anetResolve function is ever used within Redis.
 2013-07-11 17:05:12 +02:00
Geoff Garside 4b2e374e4a Update calls to anetResolve to include buffer size 2013-07-11 17:05:08 +02:00
Geoff Garside 3570411def Update calls to anetPeerToString to include ip_len. 2013-07-11 17:05:03 +02:00
Geoff Garside 68d72aa5b1 Add macro to define clusterNode.ip buffer size. 
Add REDIS_CLUSTER_IPLEN macro to define the size of the clusterNode ip
character array. Additionally use this macro in inet_ntop(3) calls where
the size of the array was being defined manually.

The REDIS_CLUSTER_IPLEN is defined as INET_ADDRSTRLEN which defines the
correct size of a buffer to store an IPv4 address in. The
INET_ADDRSTRLEN macro itself is defined in the <netinet/in.h> header
file and should be portable across the majority of systems.
 2013-07-11 17:04:32 +02:00
Geoff Garside 30836721f2 Update anetTcpAccept & anetPeerToString calls. 
Add the additional ip buffer length argument to function calls of
anetTcpAccept and anetPeerToString in network.c and cluster.c
 2013-07-11 17:02:29 +02:00
Geoff Garside 920ab4c926 Use inet_ntop(3) in anet. #apichange 
Replace inet_ntoa(3) calls with the more future proof inet_ntop(3)
function which is capable of handling additional address families.

API Change: anetTcpAccept() & anetPeerToString() additional argument
  additional argument required to specify the length of the character
  buffer the IP address is written to in order to comply with
  inet_ntop(3) function semantics.
 2013-07-11 17:02:12 +02:00
Geoff Garside aee5ee70fa Use getaddrinfo(3) in a anetTcpServer. 
Change anetTcpServer() function to use getaddrinfo(3) to perform
address resolution, socket creation and binding. Resolved addresses
are limited to those reachable by the AF_INET address family.
 2013-07-11 17:02:08 +02:00
Geoff Garside e50f2c93bc Use getaddrinfo(3) in anetTcpGenericConnect. 
Change anetTcpGenericConnect() function to use getaddrinfo(3) to
perform address resolution, socket creation and connection. Resolved
addresses are limited to those reachable by the AF_INET family.
 2013-07-11 17:02:04 +02:00
Geoff Garside d36dc1fc82 Add anetSetReuseAddr(err, fd) static function. 
Extract setting SO_REUSEADDR socket option into separate function
so the same code can be more easily used by anetCreateSocket and
other functions.
 2013-07-11 17:01:59 +02:00
Geoff Garside 855ff8df9f Use getaddrinfo(3) in anetResolve. #apichange 
Change anetResolve() function to use getaddrinfo(3) to resolve hostnames.
Resolved hostnames are limited to those reachable by the AF_INET address
family.

API Change: anetResolve requires additional argument.
  additional argument required to specify the length of the character
  buffer the IP address is written to in order to comply with
  inet_ntop(3) function semantics. inet_ntop(3) replaces inet_ntoa(3)
  as it has been designed to be compatible with more address families.
 2013-07-11 17:01:55 +02:00
antirez 33c5f0a243 Example redis.conf: bind to multiple interfaces documented. 2013-07-08 10:42:37 +02:00
antirez 323e5e2058 Remove cluster stuff from closeListeningSockets() for 2.8. 2013-07-08 10:31:49 +02:00
antirez fc022ca300 Binding multiple IPs done properly with multiple sockets. 2013-07-08 10:30:54 +02:00
antirez 00063bea33 Revert "anet.c: Allow creation of TCP listening sockets bound to N addresses." 
Bind() can't be called multiple times against the same socket, multiple
sockets are required to bind multiple interfaces, silly me.

This reverts commit bd234d62bb.
 2013-07-08 10:28:02 +02:00
antirez 6dabd34ad0 Ability to bind multiple addresses. 2013-07-08 10:27:21 +02:00
antirez 14857decfa anet.c: Allow creation of TCP listening sockets bound to N addresses. 2013-07-08 10:25:51 +02:00
antirez 2150cfebb7 sds.c: new function sdsjoin() to join strings. 2013-07-08 10:25:47 +02:00
antirez 3e3501262b redis-cli: introduced --pipe-timeout. 
When in --pipe mode, after all the data transfer to the server is
complete, now redis-cli waits at max the specified amount of
seconds (30 by default, use 0 to wait forever) without receiving any
reply at all from the server. After this time limit the operation is
aborted with an error.

That's related to issue #681.
 2013-07-03 12:18:55 +02:00
antirez 24b3799238 redis-cli --pipe: send final ECHO in a safer way. 
If the protocol read from stdin happened to contain grabage (invalid
random chars), in the previous implementation it was possible to end
with something like:

dksfjdksjflskfjl*2\r\n$4\r\nECHO....

That is invalid as the *2 should start into a new line. Now we prefix
the ECHO with a CRLF that has no effects on the server but prevents this
issues most of the times.

Of course if the offending wrong sequence is something like:

$3248772349\r\n

No one is going to save us as Redis will wait for data in the context of
a big argument, so this fix does not cover all the cases.

This partially fixes issue #681.
 2013-07-03 12:02:09 +02:00
antirez 2f6560f351 pqsort.c: remove the "switch to insertion sort" optimization. 
It causes catastrophic performance for certain inputs.

Relevant NetBSD commit:

http://cvsweb.netbsd.org/bsdweb.cgi/src/lib/libc/stdlib/qsort.c?rev=1.20&content-type=text/x-cvsweb-markup&only_with_tag=MAIN

This fixes issue #968.
 2013-07-02 17:48:49 +02:00
charsyam 847a40805f fix randstring bug 2013-07-02 12:25:30 +02:00
antirez aaf5623535 Only allow basenames for dbfilename and appendfilename. 
This fixes issue #1094.
 2013-07-02 12:24:12 +02:00
antirez 25a3ad7f74 pathIsBaseName() added to utils.c 
The function is used to test that the specified string looks like just
as the basename of a path, without any absolute or relative path.
 2013-07-02 12:24:08 +02:00
antirez 028fdbb70b getAbsolutePath() moved into utils.c 2013-07-02 12:24:05 +02:00
antirez 8090c37f71 CONFIG SET maxclients. 2013-07-01 10:44:08 +02:00
antirez 75dc48f04a ae.c event loop: API to resize the fd set size on the run. 2013-07-01 10:44:04 +02:00
antirez 0a0fd37e5b Allow SHUTDOWN in loading state. 2013-06-27 12:14:23 +02:00
antirez 7bfbc38068 Redis 2.7.3 2013-06-26 15:26:11 +02:00
antirez bb0d0fd479 function renamed: popcount_binary -> redisPopcount. 2013-06-26 15:24:58 +02:00
antirez cdf79c063f Don't disconnect pre PSYNC replication clients for timeout. 
Clients using SYNC to replicate are older implementations, such as
redis-cli --slave, and are not designed to acknowledge the master with
REPLCONF ACK commands, so we don't have any feedback and should not
disconnect them on timeout.
 2013-06-26 15:24:30 +02:00
antirez 7cca200603 Test: add some AOF testing to EVALSHA replication test. 2013-06-26 15:24:26 +02:00
antirez 29308077a8 Flush the replication script cache after SCRIPT FLUSH. 2013-06-26 15:24:22 +02:00
antirez 9faa6b9dc3 Test: EVALSHA replication. 2013-06-26 15:24:19 +02:00
antirez b0e0f3c5b2 Test: randomInt() behavior commented. 2013-06-26 15:24:16 +02:00
antirez 4e2b97bb89 Test: replication-3 test speedup in master-slave setup. 2013-06-26 15:24:12 +02:00
antirez d34c1ac208 Force propagation of SCRIPT LOAD to AOF. 2013-06-26 15:24:09 +02:00
antirez b7233f40fb SCRIPT FLUSH comment minor pedantic improvement. 2013-06-26 15:24:04 +02:00
antirez 8450d6e24d Move Replication Script Cache initialization in safer place. 
It should be called just one time at startup and not every time the Lua
scripting engine is re-initialized, otherwise memory is leaked.
 2013-06-26 15:23:52 +02:00
antirez 545fe0c318 Use the RSC to replicate EVALSHA unmodified. 
This commit uses the Replication Script Cache in order to avoid
translating EVALSHA into EVAL whenever possible for both the AOF and
slaves.
 2013-06-26 15:23:29 +02:00
antirez 9d894b1b8c Replication of scripts as EVALSHA: sha1 caching implemented. 
This code is only responsible to take an LRU-evicted fixed length cache
of SHA1 that we are sure all the slaves received.

In this commit only the implementation is provided, but the Redis core
does not use it to actually send EVALSHA to slaves when possible.
 2013-06-26 15:23:15 +02:00
antirez 8328d993e1 New API to force propagation. 
The old REDIS_CMD_FORCE_REPLICATION flag was removed from the
implementation of Redis, now there is a new API to force specific
executions of a command to be propagated to AOF / Replication link:

    void forceCommandPropagation(int flags);

The new API is also compatible with Lua scripting, so a script that will
execute commands that are forced to be propagated, will also be
propagated itself accordingly even if no change to data is operated.

As a side effect, this new design fixes the issue with scripts not able
to propagate PUBLISH to slaves (issue #873).
 2013-06-26 15:21:55 +02:00
antirez aeab473dfb Allow PUBSUB NUMSUB without channels. 
The result is an empty list but it is handy to call it programmatically.
 2013-06-26 15:21:12 +02:00
antirez a8f1474dfd PUBSUB command implemented. 
Currently it implements three subcommands:

PUBSUB CHANNELS [<pattern>]    List channels with non-zero subscribers.
PUBSUB NUMSUB [channel_1 ...]  List number of subscribers for channels.
PUBSUB NUMPAT                  Return number of subscribed patterns.
 2013-06-26 15:21:08 +02:00
antirez 72d3fbea65 Fix comment typo in integration/aof.tcl. 2013-06-26 15:21:05 +02:00
YAMAMOTO Takashi 3642557ae6 don't assume time_t == long 
time_t is always 64bit on recent versions of NetBSD.
 2013-06-26 15:20:59 +02:00
YAMAMOTO Takashi 30f3ae03a5 use nanosleep instead of usleep 
SUSv3 says that:
	The useconds argument shall be less than one million. If the value of
	useconds is 0, then the call has no effect.
and actually NetBSD's implementation rejects such a value with EINVAL.
use nanosleep which has no such a limitation instead.
 2013-06-26 15:20:56 +02:00
YAMAMOTO Takashi f27896a17d rename popcount to popcount_binary to avoid a conflict with NetBSD libc 
NetBSD-current's libc has a function named popcount.
hiding these extensions using feature macros is not possible because
redis uses other extensions covered by the same feature macro.
eg. inet_aton
 2013-06-26 15:20:32 +02:00
YAMAMOTO Takashi 9a7b09ed35 don't define _XOPEN_SOURCE for NetBSD 
on NetBSD, defining _XOPEN_SOURCE hides extensions
like inet_aton, strcasecmp, etc.
 2013-06-26 15:20:03 +02:00
Jan-Erik Rediger 90c3a14ac0 Initialize char* to NULL to remove compiler warning 2013-06-20 16:59:04 +02:00
antirez 9af8125c7d Sentinel: parse new INFO replication output correctly. 
Sentinel was not able to detect slaves when connected to a very recent
version of Redis master since a previos non-backward compatible change
to INFO broken the parsing of the slaves ip:port INFO output.

This fixes issue #1164
 2013-06-20 10:24:31 +02:00
antirez be8d5fd954 Test: regression test for #1163. 2013-06-19 18:53:14 +02:00
antirez 86147afd08 Allow writes from scripts called by AOF loading in read-only slaves. 
This fixes issue #1163
 2013-06-19 18:29:05 +02:00
antirez 449ec38fe4 Fix logStackTrace() when logging to stdout. 
When the semantics changed from logfile = NULL to logfile = "" to log
into standard output, no proper change was made to logStackTrace() to
make it able to work with the new setup.

This commit fixes the issue.
 2013-06-19 14:45:38 +02:00
antirez bf8e078aa2 Lua script errors format more unified. 
lua_pcall error handler now formats errors in a way more similar to
luaPushError() so that errors generated in different contexts look alike.
 2013-06-18 19:38:23 +02:00
antirez b3fbc84cc1 Lua scripting: improve error reporting. 
When calling Lua scripts we try to report not just the error but
information about the code line causing the error.
 2013-06-18 19:38:19 +02:00
Marc-Antoine Perennou 1cf839a216 test-server: only listen to 127.0.0.1 
Signed-off-by: Marc-Antoine Perennou <Marc-Antoine@Perennou.com>
 2013-06-18 19:38:16 +02:00
ioddly e34f00842f Try to report source of bad Lua API calls 2013-06-18 19:38:10 +02:00
antirez b435817b17 Binary safe dump of object content in redisLogObjectDebugInfo(). 2013-06-04 15:55:35 +02:00
antirez 054632f551 CONFIG SET: accept slave-priority zero, it is valid. 2013-05-31 19:31:51 +02:00
antirez d34c623e10 Test: avoid a false positive in min-slaves test. 2013-05-31 11:43:23 +02:00
antirez fe4b62fcaf Refresh good slaves count after CONFIG SET min-slaves-... 
This way just after the CONFIG SET enabling the min-slaves feature it is
possible to write to the database without delays.
 2013-05-31 07:20:19 +02:00
antirez 971217c7b4 New INFO field "min_slaves_good_slaves". 
When min-slaves-to-write feature is active, this field reports the
number of slaves considered good (online state, lag within the specified
range).
 2013-05-31 07:19:45 +02:00
antirez f5275da6e9 Refresh good slaves count when setting slave state as online. 2013-05-31 07:19:42 +02:00
antirez 9a7ea3dae9 Tests added for min-slaves feature. 2013-05-30 18:54:39 +02:00
antirez 4333e6ce20 Make tests compatible with new INFO replication output. 2013-05-30 11:43:53 +02:00
antirez 9d1bc3efcf min-replicas-to-write: only deny write commands. 
I guess I needed another coffee...
 2013-05-30 11:32:00 +02:00
antirez d0d67f8d42 min-slaves-to-write: don't accept writes with less than N replicas. 
This feature allows the user to specify the minimum number of
connected replicas having a lag less or equal than the specified
amount of seconds for writes to be accepted.
 2013-05-30 11:31:46 +02:00
antirez fc2587ec6e min-slaves-to-write: initial description of the feature in redis.conf 2013-05-30 11:30:34 +02:00
antirez 75fcf5c936 repl_offset field in INFO replication is now just offset. 2013-05-29 19:56:40 +02:00
antirez c64338aa8f Slaves list in INFO output: lag added, format changed. 
There is a new 'lag' information in the list of slaves, in the
"replication" section of the INFO output.

Also the format was changed in a backward incompatible way in order to
make it more easy to parse if new fields are added in the future, as the
new format is comma separated but has named fields (no longer positional
fields).
 2013-05-29 19:55:11 +02:00
antirez 889d39b5ba Accept REPLCONF in any state. 2013-05-28 15:26:55 +02:00
antirez 1b87b3ef38 A comment about BLPOP timeout did not reflected actual behavior. 2013-05-27 19:34:12 +02:00
antirez 2d497014ea Version bumped to 2.7.2. 2013-05-27 11:43:47 +02:00
antirez 75bf91abea redis.conf updated: repl-timeout now uesd by masters as well. 2013-05-27 11:43:21 +02:00
antirez 308940aa2c Close connection with timedout slaves. 
Now masters, using the time at which the last REPLCONF ACK was received,
are able to explicitly disconnect slaves that are no longer responding.

Previously the only chance was to see a very long output buffer, that
was highly suboptimal.
 2013-05-27 11:43:17 +02:00
antirez 45e6a4023a Send ACK to master once every second. 
ACKs can be also used as a base for synchronous replication. However in
that case they'll be explicitly requested by the master when the client
sends a request that needs to be replicated synchronously.
 2013-05-27 11:43:14 +02:00
antirez a74f8fe1ad Don't ACK the master after every command. 
Sending an ACK is now moved into the replicationSendAck() function.
 2013-05-27 11:43:10 +02:00
antirez 0000d5334d Make sure that REPLCONF ACK really has no return value. 2013-05-27 11:43:06 +02:00
antirez 146f1d7d86 Replication: send REPLCONF ACK to master. 2013-05-27 11:43:03 +02:00
antirez 1e77b77de4 REPLCONF ACK command. 
This special command is used by the slave to inform the master the
amount of replication stream it currently consumed.

it does not return anything so that we not need to consume additional
bandwidth needed by the master to reply something.

The master can do a number of things knowing the amount of stream
processed, such as understanding the "lag" in bytes of the slave, verify
if a given command was already processed by the slave, and so forth.
 2013-05-27 11:43:00 +02:00
antirez 0387f86947 Fixed a bug in no queueing replies to master. 2013-05-25 01:00:16 +02:00
antirez 7ae29d5d9e Replication: don't even queue replies to master commands. 
When master send commands, there is no need for the slave to reply.
Redis used to queue the reply in the output buffer and discard the reply
later, this is a waste of work and it is not clear why it was this way
(I sincerely don't remember).

This commit changes it in order to don't queue the reply at all.

All tests passing.
 2013-05-24 19:48:03 +02:00
antirez 302b0a5c7b Top comment for prepareClientToWrite() clarified. 
We don't write the output buffer to the client socket for slaves only if
the slave is not online.
 2013-05-24 18:42:18 +02:00
antirez 0b398b6ffa Don't stop reading redis.conf if line has no args. 
Should be "continue" and was "return".

This fixes issue #1110
 2013-05-18 16:22:24 +02:00
antirez 059018c4cd CONFIG REWRITE: remove cluster stuff for 2.8 back port. 2013-05-15 12:02:44 +02:00
antirez 414d5db85c Use memtoll() when parsing the backlog-size option. 2013-05-15 12:01:32 +02:00
antirez 604efb02ab CONFIG REWRITE: backlog size is a bytes option. 2013-05-15 12:01:27 +02:00
antirez 7d338fd982 CONFIG REWRITE: bindaddr -> bind. 2013-05-15 12:01:23 +02:00
antirez 2173f30836 CONFIG REWRITE: when rewriting amount of bytes use GB, MB, KB if possible. 2013-05-15 12:01:14 +02:00
antirez 901ceab20b CONFIG REWRITE: correctly escape the notify-keyspace-events option. 2013-05-15 12:01:06 +02:00
antirez b4ce02988c CONFIG REWRITE: "active-rehashing" -> "activerehashing". 2013-05-15 12:01:02 +02:00
antirez 2a29cf430a CONFIG REWRITE: fixed typo in AOF fsync policy. 2013-05-15 12:00:57 +02:00
antirez c95a60960b CONFIG REWRITE: repl-disable-tcp-nodelay is a boolean option. 2013-05-15 12:00:52 +02:00
antirez e8cc959300 Test default config modified to set logfile to empty string. 
Required because of recent changes in the way logfile is set to standard
output.
 2013-05-15 12:00:47 +02:00
antirez 180cfaae8e Added a define for most configuration defaults. 
Also the logfile option was modified to always have an explicit value
and to log to stdout when an empty string is used as log file.

Previously there was special handling of the string "stdout" that set
the logfile to NULL, this always required some special handling.
 2013-05-15 12:00:43 +02:00
antirez 417253afc3 CONFIG REWRITE: Use sane perms when creating redis.conf from scratch. 2013-05-15 12:00:38 +02:00
antirez 8feb3d261a CONFIG REWRITE: actually rewrite the config file, atomically. 2013-05-15 12:00:32 +02:00
antirez a461376ca9 CONFIG REWRITE: remove orphaned lines. 2013-05-15 12:00:28 +02:00
antirez 8cf0b66606 CONFIG REWRITE: strip multiple empty lines. 2013-05-15 12:00:24 +02:00
antirez cb8433f313 CONFIG REWRITE: support for client-output-buffer-limit. 2013-05-15 12:00:21 +02:00
antirez f62851aea9 CONFIG REWRITE: support for dir and slaveof. 2013-05-15 12:00:17 +02:00
antirez 91ef10ef45 CONFIG REWRITE: support for save and syslog-facility. 2013-05-15 12:00:13 +02:00
antirez 973f793b04 CONFIG REWRITE: Initial support code and design. 2013-05-15 12:00:03 +02:00
antirez 7a5d3d91a1 Obtain absoute path of configuration file, expose it in INFO. 2013-05-15 11:59:26 +02:00
antirez d8dd70c2dd redis-cli: help.h updated. 2013-05-14 11:22:58 +02:00
antirez 41babac68d Test: more PSYNC tests (backlog TTL). 2013-05-09 12:52:16 +02:00
antirez cb55ad991b Version bumped to 2.7.1 2013-05-08 15:24:54 +02:00
antirez c847c73e78 Test: check that replication partial sync works if we break the link. 
The test checks both successful syncs and unsuccessful ones by changing
the backlog size.
 2013-05-08 13:02:53 +02:00
antirez 2865bb7f1f Test: various issues with the replication-4.tcl test fixed. 
The test actually worked, but vars for master and slave were inverted
and sometimes used incorrectly.
 2013-05-08 11:59:42 +02:00
Jiahao Huang 3d93dffd5d in 32bit machine, popcount don't work with a input string length up to 512 MB, 
bitcount commant may return negtive integer with string length more than 256 MB
 2013-05-08 09:56:10 +02:00
antirez 85671d0e3b Added more info about 32 bit builds into README. 2013-05-08 09:48:46 +02:00
antirez b05d2516fe CONFIG SET server.masterauth aesthetic change. 
This is just to make the code exactly like the above instance used for
requirepass. No actual change nor the original code violated the Redis
coding style.
 2013-05-02 17:22:30 +02:00
Michel Martens b60465eb78 Reset masterauth if an empty string is configured. 2013-05-02 17:21:55 +02:00
charsyam a94db7908a Fix AOF bug: expire could be removed from key on AOF rewrite. 
There was a race condition in the AOF rewrite code that, with bad enough
timing, could cause a volatile key just about to expire to be turned
into a non-volatile key. The bug was never reported to cause actualy
issues, but was found analytically by an user in the Redis mailing list:

https://groups.google.com/forum/?fromgroups=#!topic/redis-db/Kvh2FAGK4Uk

This commit fixes issue #1079.
 2013-05-02 15:46:56 +02:00
antirez e7bcec829c Sentinel: changes to tilt mode. 
Tilt mode was too aggressive (not processing INFO output), this
resulted in a few problems:

1) Redirections were not followed when in tilt mode. This opened a
   window to misinform clients about the current master when a Sentinel
   was in tilt mode and a fail over happened during the time it was not
   able to update the state.

2) It was possible for a Sentinel exiting tilt mode to detect a false
   fail over start, if a slave rebooted with a wrong configuration
   about at the same time. This used to happen since in tilt mode we
   lose the information that the runid changed (reboot).

   Now instead the Sentinel in tilt mode will still remove the instance
   from the list of slaves if it changes state AND runid at the same
   time.

Both are edge conditions but the changes should overall improve the
reliability of Sentinel.
 2013-04-30 15:09:14 +02:00
antirez 4028a777b6 Sentinel: more sensible delay in master demote after tilt. 2013-04-30 15:09:10 +02:00
antirez 70845320cc Sentinel: only demote old master into slave under certain conditions. 
We used to always turn a master into a slave if the DEMOTE flag was set,
as this was a resurrecting master instance.

However the following race condition is possible for a Sentinel that
got partitioned or internal issues (tilt mode), and was not able to
refresh the state in the meantime:

1) Sentinel X is running, master is instance "A".
3) "A" fails, sentinels will promote slave "B" as master.
2) Sentinel X goes down because of a network partition.
4) "A" returns available, Sentinels will demote it as a slave.
5) "B" fails, other Sentinels will promote slave "A" as master.
6) At this point Sentinel X comes back.

When "X" comes back he thinks that:

"B" is the master.
"A" is the slave to demote.

We want to avoid that Sentinel "X" will demote "A" into a slave.
We also want that Sentinel "X" will detect that the conditions changed
and will reconfigure itself to monitor the right master.

There are two main ways for the Sentinel to reconfigure itself after
this event:

1) If "B" is reachable and already configured as a slave by other
sentinels, "X" will perform a redirection to "A".
2) If there are not the conditions to demote "A", the fact that "A"
reports to be a master will trigger a failover detection in "X", that
will end into a reconfiguraiton to monitor "A".

However if the Sentinel was not reachable, its state may not be updated,
so in case it titled, or was partiitoned from the master instance of the
slave to demote, the new implementation waits some time (enough to
guarantee we can detect the new INFO, and new DOWN conditions).

If after some time still there are not the right condiitons to demote
the instance, the DEMOTE flag is cleared.
 2013-04-30 15:09:06 +02:00
antirez 2b427a89fb Lua updated to version 5.1.5. 2013-04-29 12:02:01 +02:00
antirez d2ff5ed603 Sentinel: always redirect on master->slave transition. 
Sentinel redirected to the master if the instance changed runid or it
was the first time we got INFO, and a role change was detected from
master to slave.

While this is a good idea in case of slave->master, since otherwise we
could detect a failover without good reasons just after a reboot with a
slave with a wrong configuration, in the case of master->slave
transition is much better to always perform the redirection for the
following reasons:

1) A Sentinel may go down for some time. When it is back online there is
no other way to understand there was a failover.
2) Pointing clients to a slave seems to be always the wrong thing to do.
3) There is no good rationale about handling things differently once an
instance is rebooted (runid change) in that case.
 2013-04-24 11:34:02 +02:00
antirez b06f13e7b7 Config option to turn AOF rewrite incremental fsync on/off. 2013-04-24 10:57:35 +02:00
antirez 9ca306d874 AOF: sync data on disk every 32MB when rewriting. 
This prevents the kernel from putting too much stuff in the output
buffers, doing too heavy I/O all at once. So the goal of this commit is
to split the disk pressure due to the AOF rewrite process into smaller
spikes.

Please see issue #1019 for more information.
 2013-04-24 10:27:02 +02:00
antirez 5276996cf1 rio.c: added ability to fdatasync() from time to time while writing. 2013-04-24 10:26:58 +02:00
antirez e426ea5245 Test: fix RDB test checking file permissions. 
When the test is executed using the root account, setting the permission
to 222 does not work as expected, as root can read files with 222
permission.

Now we skip the test if root is detected.

This fixes issue #1034 and the duplicated #1040 issue.

Thanks to Jan-Erik Rediger (@badboy on Github) for finding a way to reproduce the issue.
 2013-04-23 14:16:47 +02:00
antirez d0c9a2a767 Sentinel: turn old master into a slave when it comes back. 2013-04-22 11:26:29 +02:00
antirez a500c120ba Test: split conceptually unrelated comments in RDB test. 2013-04-22 11:26:02 +02:00
antirez 3a865829a7 More explicit panic message on out of memory. 2013-04-19 15:11:55 +02:00
antirez 0e3baa5241 Redis/Jemalloc Gitignore were too aggressive. 
Redis gitignore was too aggressive since simply broken.

Jemalloc gitignore was too agressive because it is conceived to just
keep the files that allow to generate all the rest in development
environments (so for instance the "configure" file is excluded).
 2013-04-18 16:23:12 +02:00
antirez 42ba5d3788 redis-cli: raise error on bad command line switch. 
Previously redis-cli never tried to raise an error when an unrecognized
switch was encountered, as everything after the initial options is to be
transmitted to the server.

However this is too liberal, as there are no commands starting with "-".
So the new behavior is to produce an error if there is an unrecognized
switch starting with "-". This should not break past redis-cli usages
but should prevent broken options to be silently discarded.

As far the first token not starting with "-" is encountered, all the
rest is considered to be part of the command, so you cna still use
strings starting with "-" as values, like in:

    redis-cli --port 6380 set foo --my-value
 2013-04-11 13:19:52 +02:00
antirez 0d7b10ee6e redis-cli: --latency-history mode implemented. 2013-04-11 13:19:48 +02:00
antirez 8d81e68d18 Test: remove useless statements and comments from test default config. 2013-04-04 14:31:30 +02:00
antirez d83d72f380 Make rio.c comment 80-columns friendly. 2013-04-03 12:41:27 +02:00
antirez d6b0c18c51 Throttle BGSAVE attempt on saving error. 
When a BGSAVE fails, Redis used to flood itself trying to BGSAVE at
every next cron call, that is either 10 or 100 times per second
depending on configuration and server version.

This commit does not allow a new automatic BGSAVE attempt to be
performed before a few seconds delay (currently 5).

This avoids both the auto-flood problem and filling the disk with
logs at a serious rate.

The five seconds limit, considering a log entry of 200 bytes, will use
less than 4 MB of disk space per day that is reasonable, the sysadmin
should notice before of catastrofic events especially since by default
Redis will stop serving write queries after the first failed BGSAVE.

This fixes issue #849
 2013-04-02 14:12:28 +02:00
Johan Bergström 0bc7dbda1c use install as default installer (except on SunOS) 2013-04-02 11:54:20 +02:00
Johan Bergström 034d94193d Silence mkdir output 2013-04-02 11:30:16 +02:00
Johan Bergström 2094eb36e7 Only pass -rdynamic as linker option 2013-04-02 11:30:13 +02:00
Johan Bergström 31844fe138 Remove extra spaces 2013-04-02 11:29:56 +02:00
Johan Bergström c72459bf18 make check is a common naming convention for tests 2013-04-02 11:27:38 +02:00
Johan Bergström ca0b1c2436 Inherit CC for Lua 2013-04-02 11:26:50 +02:00
Johan Bergström 49eef7738e Spaces to tabs 2013-04-02 11:26:41 +02:00
Johan Bergström 0c8690d46e Slightly refactor CFLAGS/LDFLAGS/LIBS 
This way, we can avoid -rdynamic and -pthread warnings on darwin.
 2013-04-02 11:26:37 +02:00
charsyam cf8484b723 Support for case unsensitive SET options. 2013-03-29 10:37:29 +01:00
antirez 0c447d3506 Test: Extended SET tests. 2013-03-28 16:39:39 +01:00
antirez 24aa61785d Extended SET command implemented (issue #931). 2013-03-28 16:39:34 +01:00
antirez e6e07042a1 EXPIRE should not resurrect keys. Issue #1026. 2013-03-28 12:48:38 +01:00
antirez be832929c8 Test: regression test for issue #1026. 2013-03-28 12:48:35 +01:00
antirez 4de4080ea5 Better DEBUG error message when num of arguments is wrong. 2013-03-28 12:48:30 +01:00
antirez 978185bf67 Test: verify that lazy-expire works. 2013-03-28 12:48:26 +01:00
antirez 10d8e6a712 DEBUG set-active-expire added. 
We need the ability to disable the activeExpireCycle() (active
expired key collection) call for testing purposes.
 2013-03-28 12:48:19 +01:00
antirez c6a9a20d3b Test: test replication of MULTI/EXEC. 2013-03-27 12:05:22 +01:00
antirez 61b1d6da18 Test: Restore DB back to 9 after testing MULTI/EXEC with DB 5. 2013-03-27 12:05:19 +01:00
antirez 7009ff98b8 Test: new functions to capture and analyze the replication stream. 2013-03-27 12:05:15 +01:00
antirez a452b54849 TTL / PTTL commands: two bugs fixed. 
This commit fixes two corner cases for the TTL command.

1) When the key was already logically expired (expire time older
than current time) the command returned -1 instead of -2.

2) When the key was existing and the expire was found to be exactly 0
(the key was just about to expire), the command reported -1 (that is, no
expire) instead of a TTL of zero (that is, about to expire).
 2013-03-26 13:59:33 +01:00
antirez 10c6195e3f Flag PUBLISH as read-only in the command table. 2013-03-26 13:59:26 +01:00
antirez 2618101ad5 Transactions: propagate MULTI/EXEC only when needed. 
MULTI/EXEC is now propagated to the AOF / Slaves only once we encounter
the first command that is not a read-only one inside the transaction.

The old behavior was to always propagate an empty MULTI/EXEC block when
the transaction was composed just of read only commands, or even
completely empty. This created two problems:

1) It's a bandwidth waste in the replication link and a space waste
   inside the AOF file.

2) We used to always increment server.dirty to force the propagation of
   the EXEC command, resulting into triggering RDB saves more often
   than needed.

Note: even read-only commands may also trigger writes that will be
propagated, when we access a key that is found expired and Redis will
synthesize a DEL operation. However there is no need for this to stay
inside the transaction itself, but only to be ordered.

So for instance something like:

    MULTI
    GET foo
    SET key zap
    EXEC

May be propagated into:

    DEL foo
    MULTI
    SET key zap
    EXEC

While the DEL is outside the transaction, the commands are delivered in
the right order and it is not possible for other commands to be inserted
between DEL and MULTI.
 2013-03-26 13:59:21 +01:00
antirez 82516e8168 Transactions: use discardTransaction() in EXEC implementation. 2013-03-26 13:59:18 +01:00
antirez d91dfaf6dc Transactions: use the propagate() API to propagate MULTI. 
The behavior is the same, but the code is now cleaner and uses the
proper interface instead of dealing directly with AOF/replication
functions.
 2013-03-26 13:59:14 +01:00
antirez 150f7e43ca Allow SELECT while loading the DB. 
Fixes issue #1024.
 2013-03-26 13:59:05 +01:00
NanXiao 382420ec6e Update config.c 
Fix bug in configGetCommand function: get correct masterauth value.
 2013-03-25 19:32:20 +01:00
antirez 6b7562e204 Test: obuf-limits test false positive removed. 
Fixes #621.
 2013-03-25 11:56:31 +01:00
antirez ec18d4bfba redis-cli --stat, stolen from redis-tools. 
Redis-tools is a connection of tools no longer mantained that was
intented as a way to economically make sense of Redis in the pre-vmware
sponsorship era. However there was a nice redis-stat utility, this
commit imports one of the functionalities of this tool here in redis-cli
as it seems to be pretty useful.

Usage: redis-cli --stat

The output is similar to vmstat in the format, but with Redis specific
stuff of course.

From the point of view of the monitored instance, only INFO is used in
order to grab data.
 2013-03-25 11:52:25 +01:00
antirez 58708fa65a Replication: master_link_down_since_seconds initial value should be huge. 
server.repl_down_since used to be initialized to the current time at
startup. This is wrong since the replication never started. Clients
testing this filed to check if data is uptodate should never believe
data is recent if we never ever connected to our master.
 2013-03-13 12:55:02 +01:00
antirez ba8367ee49 Test: make sure broken RDB checksum is detected. 2013-03-13 11:15:31 +01:00
antirez 67a745515d Test: more RDB loading checks. 
A test for issue #1001 is included.
 2013-03-13 10:08:40 +01:00
antirez 7b6fb960cc Test: check that Redis starts empty without an RDB file. 2013-03-13 10:08:36 +01:00
antirez 64b91c1493 rdbLoad(): rework code to save vertical space. 2013-03-13 10:08:30 +01:00
Damian Janowski edb3c10361 Abort when opening the RDB file results in an error other than ENOENT. 
This fixes cases where the RDB file does exist but can't be accessed for
any reason. For instance, when the Redis process doesn't have enough
permissions on the file.
 2013-03-13 10:08:26 +01:00
antirez b2e42a90d3 Set default for stop_writes_on_bgsave_err in initServerConfig(). 
It was placed for error in initServer() that's called after the
configuation is already loaded, causing issue #1000.
 2013-03-12 18:36:02 +01:00
antirez 378fcaad55 redis-cli --bigkeys: don't crash with empty DBs. 2013-03-12 09:57:57 +01:00
antirez 8e7d962291 activeExpireCycle() smarter with many DBs and under expire pressure. 
activeExpireCycle() tries to test just a few DBs per iteration so that
it scales if there are many configured DBs in the Redis instance.
However this commit makes it a bit smarter when one a few of those DBs
are under expiration pressure and there are many many keys to expire.

What we do is to remember if in the last iteration had to return because
we ran out of time. In that case the next iteration we'll test all the
configured DBs so that we are sure we'll test again the DB under
pressure.

Before of this commit after some mass-expire in a given DB the function
tested just a few of the next DBs, possibly empty, a few per iteration,
so it took a long time for the function to reach again the DB under
pressure. This resulted in a lot of memory being used by already expired
keys and never accessed by clients.
 2013-03-11 11:30:13 +01:00
antirez c5afbb6e2a In databasesCron() never test more DBs than we have. 2013-03-11 11:30:09 +01:00
antirez 7e7ee815b0 Make comment name match var name in activeExpireCycle(). 2013-03-11 11:30:04 +01:00
antirez 6a8288766c Optimize inner loop of activeExpireCycle() for no-expires case. 2013-03-11 11:30:01 +01:00
antirez 9afb7789b3 REDIS_DBCRON_DBS_PER_SEC -> REDIS_DBCRON_DBS_PER_CALL 2013-03-11 11:29:57 +01:00
antirez 1a73260dce activeExpireCycle(): process only a small number of DBs per iteration. 
This small number of DBs is set to 16 so actually in the default
configuraiton Redis should behave exactly like in the past.
However the difference is that when the user configures a very large
number of DBs we don't do an O(N) operation, consuming a non trivial
amount of CPU per serverCron() iteration.
 2013-03-11 11:29:53 +01:00
antirez 8abfe5fe8b Use unsigned integers for DB ids, for defined wrap-to-zero. 2013-03-11 11:29:49 +01:00
antirez a4bb4b29fb Only resize/rehash a few databases per cron iteration. 
This is the first step to lower the CPU usage when many databases are
configured. The other is to also process a limited number of DBs per
call in the active expire cycle.
 2013-03-11 11:29:45 +01:00
antirez aa7f74c7e9 Actually call databasesCron() inside serverCron(). 2013-03-11 11:29:41 +01:00
antirez 21822fd0e9 Move Redis databases background processing to databasesCron(). 2013-03-11 11:29:34 +01:00
Gengliang Wang 6ae8fb477a Removed useless "return" statements in pubsub.c 
(original commit message edited)
 2013-03-06 16:50:53 +01:00
antirez bc1b2e8f96 API to lookup commands with their original name. 
A new server.orig_commands table was added to the server structure, this
contains a copy of the commant table unaffected by rename-command
statements in redis.conf.

A new API lookupCommandOrOriginal() was added that checks both tables,
new first, old later, so that rewriteClientCommandVector() and friends
can lookup commands with their new or original name in order to fix the
client->cmd pointer when the argument vector is renamed.

This fixes the segfault of issue #986, but does not fix a wider range of
problems resulting from renaming commands that actually operate on data
and are registered into the AOF file or propagated to slaves... That is
command renaming should be handled with care.
 2013-03-06 16:36:57 +01:00
antirez deccd104d8 Add a warning about command renaming in redis.conf. 2013-03-06 16:36:57 +01:00
antirez 5bcfa2ebbc Handle a non-impossible empty argv in loadServerConfigFromString(). 
Usually this does not happens since we trim for " \t\r\n", but if there
are other chars that return true with isspace(), we may end with an
empty argv. Better to handle the condition in an explicit way.
 2013-03-06 12:43:49 +01:00
antirez 00a5e86e44 redis-cli: use sdsfreesplitres() instead of hand-coding it. 2013-03-06 12:43:45 +01:00
antirez f72a3406a7 sds.c: sdssplitargs_free() removed as it was a duplicate. 2013-03-06 12:43:41 +01:00
antirez 68532cf01e More specific error message in loadServerConfigFromString(). 2013-03-06 12:43:37 +01:00
antirez a70e3ffb16 sdssplitargs(): on error set *argc to 0. 
This makes programs not checking the return value for NULL much safer
since with this change:

1) It is still possible to iterate the zero-length result without
crashes.
2) sdssplitargs_free will work against NULL and 0 count.
 2013-03-06 12:43:32 +01:00
antirez a6f557657c sdssplitargs(): now returns NULL only on error. 
An empty input string also resulted into the function returning NULL
making it harder for the caller to distinguish between error and empty
string without checking the original input string length.
 2013-03-06 12:43:27 +01:00
charsyam 5aeb1f71a3 Don't segfault on unbalanced quotes. 2013-03-06 12:43:22 +01:00
antirez aadb8aaa08 Allow AUTH while loading the DB in memory. 
While Redis is loading the AOF or RDB file in memory only a subset of
commands are allowed. This commit adds AUTH to this subset.
 2013-03-06 11:51:23 +01:00
antirez df3b492045 redis-cli: use keepalive socket option. 
This should improve things in two ways:

1) Prevent timeouts caused by the execution of long commands.
2) Improve detection of real connection errors.

This is mostly effective only on Linux because of the bogus default
keepalive settings. In Linux we have OS-specific calls to set the
keepalive interval to reasonable values.
 2013-03-04 11:16:21 +01:00
0x20h e618eb4fb3 suppress external diff program when using git diff. 2013-03-04 10:58:02 +01:00
Stam He a74056f1e0 point 2 of slave-serve-stale-data miss '-' between 'stale' and 'data' 2013-03-04 10:49:19 +01:00
antirez 02cd545f7d A comment in main() clarified. 2013-02-27 12:47:51 +01:00
antirez b148513fec Remove warning when printing redisBuildId(). 2013-02-27 12:47:38 +01:00
antirez 5e9e96999f Remove too agressive/spamming log in rdb.c. 2013-02-27 12:47:33 +01:00
antirez d2a37badc2 Use GCC printf format attribute for redisLog(). 
This commit also fixes redisLog() statements producing warnings.
 2013-02-27 12:47:16 +01:00
antirez 0d8d7f106a Better panic message for failed time event creation. 2013-02-27 12:00:51 +01:00
Stam He 438dfdd13d add a check for aeCreateTimeEvent 
1) Add a check for aeCreateTimeEvent in function initServer.
 2013-02-27 12:00:48 +01:00
Stam He 6f96ac1c1c Set proctitle: avoid the use of __attribute__((constructor)). 
This cased a segfault in some Linux system and was GCC-specific.

Commit modified by @antirez:

1) Stripped away the part to set the proc title via config for now.
2) Handle initialization of setproctitle only when the replacement
   is used.
3) Don't require GCC now that the attribute constructor is no
   longer used.
 2013-02-27 12:00:43 +01:00
antirez f69b0a0db8 setproctitle.c: declar tmp as static so valgrind will not detect a leak. 2013-02-26 15:51:15 +01:00
antirez ac3100bc3b Set process name in ps output to make operations safer. 
This commit allows Redis to set a process name that includes the binding
address and the port number in order to make operations simpler.

Redis children processes doing AOF rewrites or RDB saving change the
name into redis-aof-rewrite and redis-rdb-bgsave respectively.

This in general makes harder to kill the wrong process because of an
error and makes simpler to identify saving children.

This feature was suggested by Arnaud GRANAL in the Redis Google Group,
Arnaud also pointed me to the setproctitle.c implementation includeed in
this commit.

This feature should work on all the Linux, OSX, and all the three major
BSD systems.
 2013-02-26 12:03:48 +01:00
Arnaud Granal fedcd51185 Fix error "repl-backlog-size must be 1 or greater" 
The parameter repl-backlog-size is not parsed correctly in the configuration file. argv[0] is parsed instead of argv[1].
 2013-02-25 18:38:48 +01:00
antirez ff56772115 PSYNC: another change to unexpected reply from PSYNC. 2013-02-13 18:43:45 +01:00
antirez 964ee00de0 PSYNC: More robust handling of unexpected reply to PSYNC. 2013-02-13 18:34:18 +01:00
antirez 3b4c0d80d5 Avoid compiler warning by casting to match printf() specifier. 2013-02-13 13:38:32 +01:00
antirez d96a66f531 Replication: more strict error checking for master PING reply. 2013-02-12 16:59:27 +01:00
antirez 31f0a6ec50 Replication: added new stats counting full and partial resynchronizations. 2013-02-12 16:26:42 +01:00
antirez 5fe2577a19 Return a specific NOAUTH error if authentication is required. 2013-02-12 16:25:47 +01:00
antirez 9854252798 Test: avoid false positives in CLIENT SETNAME closed connection test. 2013-02-12 13:27:30 +01:00
Steven Penny 5a8cf340a4 Format to fit 80 columns 
This makes it readable on GitHub and editors without auto wrapping.
 2013-02-12 13:11:44 +01:00
antirez 0ae1a6b1c3 Add missing bracket removed for error after rebase of PSYNC. 2013-02-12 12:58:07 +01:00
antirez 2b1398e870 PSYNC: debugging printf() calls are now logs at DEBUG level. 2013-02-12 12:58:03 +01:00
antirez 5a4ef3e5b6 Remove harmless warning in slaveTryPartialResynchronization(). 2013-02-12 12:57:59 +01:00
antirez 4fe7672dfc PSYNC: don't use the client buffer to send +CONTINUE and +FULLRESYNC. 
When we are preparing an handshake with the slave we can't touch the
connection buffer as it'll be used to accumulate differences between
the sent RDB file and what arrives next from clients.

So in short we can't use addReply() family functions.

However we just use write(2) because we know that the socket buffer is
empty, since a prerequisite for SYNC to work is that the static buffer
and the output list are empty, and in general it is not expected that a
client SYNCs after doing some heavy I/O with the master.

However a short write connection is explicitly handled to avoid
fragility (we simply close the connection and the slave will retry).
 2013-02-12 12:57:56 +01:00
antirez 8529dd218f SYNC not allowed with pending data on the static output buffer. 2013-02-12 12:57:52 +01:00
antirez 6726ea5b53 Log the unexpected string received in place of the SYNC payload length. 2013-02-12 12:57:48 +01:00
antirez f5edd535d1 After SLAVEOF <newslave> don't allow chained slaves to PSYNC. 2013-02-12 12:57:44 +01:00
antirez 700e5eb4fc PSYNC: work in progress, preview #2, rebased to unstable. 2013-02-12 12:57:40 +01:00
antirez 4d8655cfd3 Use the new unified protocol to send SELECT to slaves. 
SELECT was still transmitted to slaves using the inline protocol, that
is conceived mostly for humans to type into telnet sessions, and is
notably not understood by redis-cli --slave.

Now the new protocol is used instead.
 2013-02-12 12:57:36 +01:00
antirez 01c21f9943 Use replicationFeedSlaves() to send PING to slaves. 
A Redis master sends PING commands to slaves from time to time: doing
this ensures that even if absence of writes, the master->slave channel
remains active and the slave can feel the master presence, instead of
closing the connection for timeout.

This commit changes the way PINGs are sent to slaves in order to use the
standard interface used to replicate all the other commands, that is,
the function replicationFeedSlaves().

With this change the stream of commands sent to every slave is exactly
the same regardless of their exact state (Transferring RDB for first
synchronization or slave already online). With the previous
implementation the PING was only sent to online slaves, with the result
that the output stream from master to slaves was not identical for all
the slaves: this is a problem if we want to implement partial resyncs in
the future using a global replication stream offset.

TL;DR: this commit should not change the behaviour in practical terms,
but is just something in preparation for partial resynchronization
support.
 2013-02-12 12:57:31 +01:00
antirez 5a35e485f9 Emit SELECT to slaves in a centralized way. 
Before this commit every Redis slave had its own selected database ID
state. This was not actually useful as the emitted stream of commands
is identical for all the slaves.

Now the the currently selected database is a global state that is set to
-1 when a new slave is attached, in order to force the SELECT command to
be re-emitted for all the slaves.

This change is useful in order to implement replication partial
resynchronization in the future, as makes sure that the stream of
commands received by slaves, including SELECT commands, are exactly the
same for every slave connected, at any time.

In this way we could have a global offset that can identify a specific
piece of the master -> slaves stream of commands.
 2013-02-12 12:57:26 +01:00
antirez c970816e4d Makefile: valgrind target added (forces -O0 and libc malloc). 2013-02-11 12:11:26 +01:00
antirez 8a22934c68 TCP keep-alive. Better documentation in redis.conf. 2013-02-11 11:44:35 +01:00
antirez b06b90b5d8 Tcp keep-alive: send three probes before detectin an error. 
Otherwise we end with less reliable connections because it's too easy
that a single packet gets lost.
 2013-02-11 11:44:31 +01:00
antirez 0a14a6542c tcp-keepalive option documented in redis.conf. 2013-02-11 11:44:27 +01:00
antirez 2ed3fc1502 Set SO_KEEPALIVE on client sockets if configured to do so. 2013-02-11 11:44:23 +01:00
antirez f2817cbd9e Add SO_KEEPALIVE support to anet.c. 2013-02-11 11:44:18 +01:00
antirez fcfdbda104 Sentinel: advertise the promoted slave address only after successful setup. 2013-02-11 11:44:14 +01:00
Pierre Chapuis 0d470b4c43 fix comments forgotten in #285 (zipmap -> ziplist) 2013-02-11 11:44:06 +01:00
antirez cc55a4525a Make all WATCHers dirty when the slave reloads the DB. 2013-02-08 10:27:21 +01:00
antirez 5eb0ec142d LASTSAVE is a "random" command. 2013-02-07 19:13:06 +01:00
antirez 5f7dff4d16 TCP_NODELAY after SYNC: changes to the implementation. 2013-02-05 12:05:39 +01:00
charsyam 1d80acae54 Turn off TCP_NODELAY on the slave socket after SYNC. 
Further details from @antirez:

It was reported by @StopForumSpam on Twitter that the Redis replication
link was strangely using multiple TCP packets for multiple commands.
This wastes a lot of bandwidth and is due to the TCP_NODELAY option we
enable on the socket after accepting a new connection.

However the master -> slave channel is a one-way channel since Redis
replication is asynchronous, so there is no point in trying to reduce
the latency, we should aim to reduce the bandwidth. For this reason this
commit introduces the ability to disable the nagle algorithm on the
socket after a successful SYNC.

This feature is off by default because the delay can be up to 40
milliseconds with normally configured Linux kernels.
 2013-02-05 12:05:24 +01:00
antirez 14daa01c52 Test: No clients timeout while testing. 2013-02-05 12:02:41 +01:00
Johan Bergström d2a8bca82b Use info nameofexectuable to find current executable 2013-02-05 11:44:55 +01:00
Johan Bergström 1124cda37a Enforce tcl 8.5 or newer 2013-02-05 11:44:51 +01:00
Johan Bergström e92edd3cee Check available tcl versions 2013-02-05 11:44:44 +01:00
Rock Li 2543fa6476 retval doesn't initalized 
If each if conditions are all fail, variable retval will under uninitlized
 2013-02-05 10:49:24 +01:00
David Celis 44d0eac568 Fix a few typos and improve grammar of redis.conf 
Make several edits to the example redis.conf configuration file for
improved flow and grammar.

Signed-off-by: David Celis <me@davidcel.is>
 2013-02-04 10:34:05 +01:00
Gengliang Wang 57e411c687 Fix a bug in srandmemberWithCountCommand() 
In CASE 2, the call sunionDiffGenericCommand will involve the string "srandmember"
> sadd foo one
(integer 1)
> sadd srandmember two
(integer 2)
> srandmember foo 3
1)"one"
2)"two"
 2013-02-04 10:33:21 +01:00
antirez 8024de7eb0 Z*STORE event fixed: generate del only if resulting sorted set is empty. 2013-01-29 13:50:09 +01:00
antirez ca8e7d4f8a Generate del events when S*STORE commands delete the destination key. 2013-01-29 13:43:24 +01:00
antirez 664de412af Tests for keyspace notifications. 2013-01-28 13:19:04 +01:00
antirez d3cf778d82 Send 'expired' events when a key expires by lookup. 2013-01-28 13:18:57 +01:00
antirez 9db8fbcfdd Keyspace notifications: fixed a leak and a bug introduced in the latest commit. 2013-01-28 13:18:53 +01:00
antirez b9bc4f9132 Keyspace events: it is now possible to select subclasses of events. 
When keyspace events are enabled, the overhead is not sever but
noticeable, so this commit introduces the ability to select subclasses
of events in order to avoid to generate events the user is not
interested in.

The events can be selected using redis.conf or CONFIG SET / GET.
 2013-01-28 13:18:36 +01:00
antirez 6de052cd77 decrRefCount -> decrRefCountVoid in list constructor. 2013-01-28 13:17:37 +01:00
antirez 81af92b023 Enable keyspace events notification when testing. 2013-01-28 13:17:33 +01:00
antirez 4ff138fbbe Keyspace events added for more commands. 2013-01-28 13:17:30 +01:00
antirez 2825f21fd8 Fix decrRefCount() prototype from void to robj pointer. 
decrRefCount used to get its argument as a void* pointer in order to be
used as destructor where a 'void free_object(void*)' prototype is
expected. However this made simpler to introduce bugs by freeing the
wrong pointer. This commit fixes the argument type and introduces a new
wrapper called decrRefCountVoid() that can be used when the void*
argument is needed.
 2013-01-28 13:17:26 +01:00
antirez a64383c861 notifyKeyspaceEvent(): release channel names using the right pointers. 2013-01-28 13:17:22 +01:00
antirez fdfb59beae Initial test events for the new keyspace notification API. 2013-01-28 13:17:18 +01:00
antirez dd2ce743ca Fixed over-80-cols comment in db.c 2013-01-28 13:17:15 +01:00
antirez 6e64525cc2 Two fixes to initial keyspace notifications API. 2013-01-28 13:17:11 +01:00
antirez 212edbc409 Keyspace events notification API. 2013-01-28 13:17:00 +01:00
antirez 153766e137 Lua struct library updated to version 0.2. 
There was a bug in the previous version of this library that caused a
crash under the circumstances described in issue #901.

The newer version of the library appears to be fixed (I tested it
manually with valgrind and everything seems fine now).

For more information about this library please visit this web site:

    http://www.inf.puc-rio.br/~roberto/struct/
 2013-01-23 10:53:31 +01:00
antirez 767a53aa84 redis-cli --bigkeys output is now simpler to understand. 2013-01-21 19:16:11 +01:00
antirez 3ff75e58e8 UNSUBSCRIBE and PUNSUBSCRIBE: always provide a reply. 
UNSUBSCRIBE and PUNSUBSCRIBE commands are designed to mass-unsubscribe
the client respectively all the channels and patters if called without
arguments.

However when these functions are called without arguments, but there are
no channels or patters we are subscribed to, the old behavior was to
don't reply at all.

This behavior is broken, as every command should always reply.
Also it is possible that we are no longer subscribed to a channels but we
are subscribed to patters or the other way around, and the client should
be notified with the correct number of subscriptions.

Also it is not pretty that sometimes we did not receive a reply at all
in a redis-cli session from these commands, blocking redis-cli trying
to read the reply.

This fixes issue #714.
 2013-01-21 19:02:24 +01:00
antirez 850117a8a5 Fixed a bug in memtest progress bar, that had no actual effects. 
This closes issue #859, thanks to @erbenmo.
 2013-01-21 12:38:46 +01:00
Bilal Husain 2f1318ab29 s/adiacent/adjacent/ 
fixed typo in a comment (step 2 memcheck)
 2013-01-21 12:20:07 +01:00
antirez f6f43d7da0 Not every __sun has backtrace(). 
I don't know how to test for Open Solaris that has support for
backtrace() so for now removing the #ifdef that breaks compilation under
other Solaris flavors.
 2013-01-21 12:07:48 +01:00
antirez 635c532c89 Additionally two typos fixed thanks to @jodal 2013-01-19 13:46:22 +01:00
antirez 39f0a33f78 Whitelist SIGUSR1 to avoid auto-triggering errors. 
This commit fixes issue #875 that was caused by the following events:

1) There is an active child doing BGSAVE.
2) flushall is called (or any other condition that makes Redis killing
the saving child process).
3) An error is sensed by Redis as the child exited with an error (killed
by a singal), that stops accepting write commands until a BGSAVE happens
to be executed with success.

Whitelisting SIGUSR1 and making sure Redis always uses this signal in
order to kill its own children fixes the issue.
 2013-01-19 13:30:46 +01:00
antirez 1e20c939fe Clear server.shutdown_asap on failed shutdown. 
When a SIGTERM is received Redis schedules a shutdown. However if it
fails to perform the shutdown it must be clear the shutdown_asap flag
otehrwise it will try again and again possibly making the server
unusable.
 2013-01-19 13:20:56 +01:00
antirez d766907cfb Slowlog: don't log EXEC but just the executed commands. 
The Redis Slow Log always used to log the slow commands executed inside
a MULTI/EXEC block. However also EXEC was logged at the end, which is
perfectly useless.

Now EXEC is no longer logged and a test was added to test this behavior.

This fixes issue #759.
 2013-01-19 12:55:15 +01:00
guiquanz 1caf09399e Fixed many typos. 
Conflicts fixed, mainly because 2.8 has no cluster support / files:
	00-RELEASENOTES
	src/cluster.c
	src/crc16.c
	src/redis-trib.rb
	src/redis.h
 2013-01-19 11:03:19 +01:00
charsyam ff1e4d22a6 redis-cli prompt bug fix 2013-01-19 10:34:22 +01:00
Jan-Erik Rediger a8f9cec16a Always exit if connection fails. 
This avoids unnecessary core dumps. Fixes antirez/redis#894
 2013-01-19 10:29:42 +01:00
bitterb aadcda9977 Fix an error reply for CLIENT command 2013-01-19 10:26:32 +01:00
Nathan Parry f2bc198d40 redis-cli --rdb fails if server sends a ping 
Redis pings slaves in "pre-synchronization stage" with newlines. (See
https://github.com/antirez/redis/blob/2.6.9/src/replication.c#L814)
However, redis-cli does not expect this - it sees the newline as the end
of the bulk length line, and ends up returning 0 as bulk the length.
This manifests as the following when running redis-cli:

    $ ./src/redis-cli --rdb some_file
    SYNC sent to master, writing 0 bytes to 'some_file'
    Transfer finished with success.

With this commit, we just ignore leading newlines while reading the bulk
length line.

To reproduce the problem, load enough data into Redis so that the
preparation of the RDB snapshot takes long enough for a ping to occur
while redis-cli is waiting for the data.
 2013-01-18 11:50:06 +01:00
antirez c48584b85d redis-cli: save an RDB dump from remote server to local file. 2013-01-16 19:48:30 +01:00
antirez 7897e0a0d6 Tests for CLIENT GETNAME/SETNAME. 2013-01-15 13:34:26 +01:00
antirez f31d10b9a6 Typo fixed, ASCI -> ASCII. 2013-01-15 13:34:24 +01:00
antirez 786bd3938e CLIENT GETNAME and CLIENT SETNAME introduced. 
Sometimes it is much simpler to debug complex Redis installations if it
is possible to assign clients a name that is displayed in the CLIENT
LIST output.

This is the case, for example, for "leaked" connections. The ability to
provide a name to the client makes it quite trivial to understand what
is the part of the code implementing the client not releasing the
resources appropriately.

Behavior:

    CLIENT SETNAME: set a name for the client, or remove the current
                    name if an empty name is set.
    CLIENT GETNAME: get the current name, or a nil.
    CLIENT LIST: now displays the client name if any.

Thanks to Mark Gravell for pushing this idea forward.
 2013-01-15 13:34:22 +01:00
antirez aa9497fdb3 Undo slave-master handshake when SLAVEOF sets a new slave. 
Issue #828 shows how Redis was not correctly undoing a non-blocking
connection attempt with the previous master when the master was set to a
new address using the SLAVEOF command.

This was also a result of lack of refactoring, so now there is a
function to cancel the non blocking handshake with the master.
The new function is now used when SLAVEOF NO ONE is called or when
SLAVEOF is used to set the master to a different address.
 2013-01-15 13:33:27 +01:00
antirez 64ee9c861f Makefile.dep updated. 2013-01-11 23:51:12 +01:00
antirez 8a70007f9e Comment in the call() function clarified a bit. 2013-01-10 12:04:24 +01:00
antirez e2e36cf19b Test: added regression for issue #872. 2013-01-10 12:04:19 +01:00
antirez 414e06653c Multiple fixes for EVAL (issue #872). 
1) The event handler was no restored after a timeout condition if the
   command was eventually executed with success.
2) The command was not converted to EVAL in case of errors in the middle
   of the execution.
3) Terrible duplication of code without any apparent reason.
 2013-01-10 12:04:15 +01:00
antirez cefff3a1b3 Better error reporting when fd event creation fails. 2013-01-03 14:32:32 +01:00
antirez 6570d2f89e ae.c: set errno when error is not a failing syscall. 
In this way the caller is able to perform better error checking or to
use strerror() without the risk of meaningless error messages being
displayed.
 2013-01-03 14:32:27 +01:00
antirez 858b5576ac Fix overflow in mstime() in redis-cli and benchmark. 
The problem does not exist in the Redis server implementation of mstime()
but is only limited to redis-cli and redis-benchmark.

Thix fixes issue #839.
 2012-12-20 15:21:40 +01:00
antirez 61fb0aacf5 CONTRIBUTING updated with request to add BSD license. 2012-12-17 11:17:55 +01:00
antirez 6fa2610ac9 Added missing license and copyright in deps/hiredis. 2012-12-17 11:11:36 +01:00
antirez a6d117b6c0 serverCron() frequency is now a runtime parameter (was REDIS_HZ). 
REDIS_HZ is the frequency our serverCron() function is called with.
A more frequent call to this function results into less latency when the
server is trying to handle very expansive background operations like
mass expires of a lot of keys at the same time.

Redis 2.4 used to have an HZ of 10. This was good enough with almost
every setup, but the incremental key expiration algorithm was working a
bit better under *extreme* pressure when HZ was set to 100 for Redis
2.6.

However for most users a latency spike of 30 milliseconds when million
of keys are expiring at the same time is acceptable, on the other hand a
default HZ of 100 in Redis 2.6 was causing idle instances to use some
CPU time compared to Redis 2.4. The CPU usage was in the order of 0.3%
for an idle instance, however this is a shame as more energy is consumed
by the server, if not important resources.

This commit introduces HZ as a runtime parameter, that can be queried by
INFO or CONFIG GET, and can be modified with CONFIG SET. At the same
time the default frequency is set back to 10.

In this way we default to a sane value of 10, but allows users to
easily switch to values up to 500 for near real-time applications if
needed and if they are willing to pay this small CPU usage penalty.
 2012-12-14 17:20:21 +01:00
Patrick TJ McPhee 5207c03c49 Define _XOPEN_SOURCE appropriately on NetBSD. 2012-12-12 18:42:41 +01:00
antirez bbee226ecd Fix config.h endianess detection to work on Linux / PPC64. 
Config.h performs endianess detection including OS-specific headers to
define the endianess macros, or when this is not possible, checking the
processor type via ifdefs.

Sometimes when the OS-specific macro is included, only __BYTE_ORDER is
defined, while BYTE_ORDER remains undefined. There is code at the end of
config.h endianess detection in order to define the macros without the
underscore, but it was not working correctly.

This commit fixes endianess detection fixing Redis on Linux / PPC64 and
possibly other systems.
 2012-12-11 17:19:54 +01:00
Brian J. McManus 17e31eb03a Issue 804 Add Default-Start and Default-Stop LSB tags for RedHat startup and update-rc.d compatability. 2012-12-03 12:31:12 +01:00
antirez 124cb6dda1 Memory leak fixed: release client's bpop->keys dictionary. 
Refactoring performed after issue #801 resolution (see commit
2f87cf8b01) introduced a memory leak that
is fixed by this commit.

I simply forgot to free the new allocated dictionary in the client
structure trusting the output of "make test" on OSX.

However due to changes in the "leaks" utility the test was no longer
testing memory leaks. This problem was also fixed.

Fortunately the CI test running at ci.redis.io spotted the bug in the
valgrind run.

The leak never ended into a stable release.
 2012-12-03 12:17:33 +01:00
antirez e39735214d Test: fixed osx "leaks" support in test. 
Due to changes in recent releases of osx leaks utility, the osx leak
detection no longer worked. Now it is fixed in a way that should be
backward compatible.
 2012-12-03 12:17:29 +01:00
antirez 07a9f85405 Blocking POP: use a dictionary to store keys clinet side. 
To store the keys we block for during a blocking pop operation, in the
case the client is blocked for more data to arrive, we used a simple
linear array of redis objects, in the blockingState structure:

    robj **keys;
    int count;

However in order to fix issue #801 we also use a dictionary in order to
avoid to end in the blocked clients queue for the same key multiple
times with the same client.

The dictionary was only temporary, just to avoid duplicates, but since
we create / destroy it there is no point in doing this duplicated work,
so this commit simply use a dictionary as the main structure to store
the keys we are blocked for. So instead of the previous fields we now
just have:

    dict *keys;

This simplifies the code and reduces the work done by the server during
a blocking POP operation.
 2012-12-02 20:42:59 +01:00
antirez fbf3e33d18 Test: regression for issue #801. 2012-12-02 20:42:56 +01:00
antirez 64e69c0d5f Client should not block multiple times on the same key. 
Sending a command like:

BLPOP foo foo foo foo 0

Resulted into a crash before this commit since the client ended being
inserted in the waiting list for this key multiple times.
This resulted into the function handleClientsBlockedOnLists() to fail
because we have code like that:

    if (de) {
        list *clients = dictGetVal(de);
        int numclients = listLength(clients);

        while(numclients--) {
            listNode *clientnode = listFirst(clients);

            /* server clients here... */
        }
    }

The code to serve clients used to remove the served client from the
waiting list, so if a client is blocking multiple times, eventually the
call to listFirst() will return NULL or worse will access random memory
since the list may no longer exist as it is removed by the function
unblockClientWaitingData() if there are no more clients waiting for this
list.

To avoid making the rest of the implementation more complex, this commit
modifies blockForKeys() so that a client will be put just a single time
into the waiting list for a given key.

Since it is Saturday, I hope this fixes issue #801.
 2012-12-02 20:42:51 +01:00
antirez c1eda786d6 SDIFF fuzz test added. 2012-11-30 17:14:08 +01:00
antirez ccc974d90b SDIFF is now able to select between two algorithms for speed. 
SDIFF used an algorithm that was O(N) where N is the total number
of elements of all the sets involved in the operation.

The algorithm worked like that:

ALGORITHM 1:

1) For the first set, add all the members to an auxiliary set.
2) For all the other sets, remove all the members of the set from the
auxiliary set.

So it is an O(N) algorithm where N is the total number of elements in
all the sets involved in the diff operation.

Cristobal Viedma suggested to modify the algorithm to the following:

ALGORITHM 2:

1) Iterate all the elements of the first set.
2) For every element, check if the element also exists in all the other
remaining sets.
3) Add the element to the auxiliary set only if it does not exist in any
of the other sets.

The complexity of this algorithm on the worst case is O(N*M) where N is
the size of the first set and M the total number of sets involved in the
operation.

However when there are elements in common, with this algorithm we stop
the computation for a given element as long as we find a duplicated
element into another set.

I (antirez) added an additional step to algorithm 2 to make it faster,
that is to sort the set to subtract from the biggest to the
smallest, so that it is more likely to find a duplicate in a larger sets
that are checked before the smaller ones.

WHAT IS BETTER?

None of course, for instance if the first set is much larger than the
other sets the second algorithm does a lot more work compared to the
first algorithm.

Similarly if the first set is much smaller than the other sets, the
original algorithm will less work.

So this commit makes Redis able to guess the number of operations
required by each algorithm, and select the best at runtime according
to the input received.

However, since the second algorithm has better constant times and can do
less work if there are duplicated elements, an advantage is given to the
second algorithm.
 2012-11-30 17:13:34 +01:00
antirez 2572bb13c6 redis-benchmark: seed the PRNG with time() at startup. 2012-11-30 17:13:25 +01:00
antirez e34c14df1a Make an EXEC test more latency proof. 2012-11-29 16:12:23 +01:00
antirez 3b71404d70 Introduced the Build ID in INFO and --version output. 
The idea is to be able to identify a build in a unique way, so for
instance after a bug report we can recognize that the build is the one
of a popular Linux distribution and perform the debugging in the same
environment.
 2012-11-29 14:22:15 +01:00
antirez cd99c14e38 On crash memory test rewrote so that it actaully works. 
1) We no longer test location by location, otherwise the CPU write cache
completely makes our business useless.
2) We still need a memory test that operates in steps from the first to
the last location in order to never hit the cache, but that is still
able to retain the memory content.

This was tested using a Linux box containing a bad memory module with a
zingle bit error (always zero).

So the final solution does has an error propagation step that is:

1) Invert bits at every location.
2) Swap adiacent locations.
3) Swap adiacent locations again.
4) Invert bits at every location.
5) Swap adiacent locations.
6) Swap adiacent locations again.

Before and after these steps, and after step 4, a CRC64 checksum is computed.
If the three CRC64 checksums don't match, a memory error was detected.
 2012-11-29 10:24:41 +01:00
antirez 053f1f357d Jemalloc updated to version 3.2.0. 2012-11-28 18:42:35 +01:00
charsyam 1f0d0b094a Remove unnecessary condition in _dictExpandIfNeeded (dict.c) 2012-11-28 11:45:52 +01:00
Matt Arsenault ccb500cf36 It's a watchdog, not a watchdong. 2012-11-28 11:42:24 +01:00
charsyam ed48550fe6 remove compile warning bioKillThreads 2012-11-28 11:42:20 +01:00
antirez d0570c9693 EVALSHA is now case insensitive. 
EVALSHA used to crash if the SHA1 was not lowercase (Issue #783).
Fixed using a case insensitive dictionary type for the sha -> script
map used for replication of scripts.
 2012-11-22 15:51:03 +01:00
antirez cc85ed41df Fix integer overflow in zunionInterGenericCommand(). 
This fixes issue #761.
 2012-11-22 15:28:48 +01:00
antirez ca9321afe2 Test: MULTI state is cleared after EXECABORT error. 2012-11-22 10:36:06 +01:00
antirez a527235f33 Test: make sure EXEC fails after previous transaction errors. 2012-11-22 10:36:01 +01:00
antirez 53600e34db Test: MULTI/EXEC tests moved into multi.tcl. 2012-11-22 10:35:50 +01:00
antirez 60f9dac672 Safer handling of MULTI/EXEC on errors. 
After the transcation starts with a MULIT, the previous behavior was to
return an error on problems such as maxmemory limit reached. But still
to execute the transaction with the subset of queued commands on EXEC.

While it is true that the client was able to check for errors
distinguish QUEUED by an error reply, MULTI/EXEC in most client
implementations uses pipelining for speed, so all the commands and EXEC
are sent without caring about replies.

With this change:

1) EXEC fails if at least one command was not queued because of an
error. The EXECABORT error is used.
2) A generic error is always reported on EXEC.
3) The client DISCARDs the MULTI state after a failed EXEC, otherwise
pipelining multiple transactions would be basically impossible:
After a failed EXEC the next transaction would be simply queued as
the tail of the previous transaction.
 2012-11-22 10:35:47 +01:00
antirez dff1ec4a36 Make bio.c threads killable ASAP if needed. 
We use this new bio.c feature in order to stop our I/O threads if there
is a memory test to do on crash. In this case we don't want anything
else than the main thread to run, otherwise the other threads may mess
with the heap and the memory test will report a false positive.
 2012-11-22 10:28:53 +01:00
antirez d4b5c4803b Fast memory test on Redis crash. 2012-11-22 10:28:48 +01:00
antirez dd905417ec Use more fine grained HAVE macros instead of HAVE_PROCFS. 2012-11-22 10:28:43 +01:00
antirez c342b075df Children creating AOF or RDB files now report memory used by COW. 
Finally Redis is able to report the amount of memory used by
copy-on-write while saving an RDB or writing an AOF file in background.

Note that this information is currently only logged (at NOTICE level)
and not shown in INFO because this is less trivial (but surely doable
with some minor form of interprocess communication).

The reason we can't capture this information on the parent before we
call wait3() is that the Linux kernel will release the child memory
ASAP, and only retain the minimal state for the process that is useful
to report the child termination to the parent.

The COW size is obtained by summing all the Private_Dirty fields found
in the "smap" file inside the proc filesystem for the process.

All this is Linux specific and is not available on other systems.
 2012-11-19 15:26:02 +01:00
antirez 1d8af6074c zmalloc_get_private_dirty() function added (Linux only). 
For non Linux systmes it just returns 0.

This function is useful to estimate copy-on-write because of childs
saving stuff on disk.
 2012-11-19 15:25:57 +01:00
antirez 1a706c729a zmalloc: kill unused __size parameter in update_zmalloc_stat_alloc() macro. 2012-11-14 13:07:54 +01:00
antirez 50c41de752 TTL API change: TTL returns -2 for non existing keys. 
The previous behavior was to return -1 if:

1) Existing key but without an expire set.
2) Non existing key.

Now the second case is handled in a different, and TTL will return -2
if the key does not exist at all.

PTTL follows the same behavior as well.
 2012-11-12 23:05:56 +01:00
antirez 388cc20418 Copyright date fixed in COPYING file. 2012-11-08 19:14:37 +01:00
antirez cda5c0ba06 Make clear that contributing code to the Redis project means to release it under the terms of the BSD license. 2012-11-08 18:43:41 +01:00
antirez 8ddb23b90c BSD license added to every C source and header file. 2012-11-08 18:34:04 +01:00
antirez 46c5d39660 Type mismatch errors are now prefixed with WRONGTYPE. 
So instead to reply with a generic error like:

-ERR ... wrong kind of value ...

now it replies with:

-WRONGTYPE ... wrong kind of value ...

This makes this particular error easy to check without resorting to
(fragile) pattern matching of the error string (however the error string
used to be consistent already).

Client libraries should return a specific exeption type for this error.

Most of the commit is about fixing unit tests.
 2012-11-06 20:28:15 +01:00
Runzhen Wang 2b28ef0e61 fix a typo in redis.h line 595 comment 2012-11-02 12:12:08 +01:00
antirez 64be5e365a More robust handling of AOF rewrite child. 
After the wait3() syscall we used to do something like that:

    if (pid == server.rdb_child_pid) {
        backgroundSaveDoneHandler(exitcode,bysignal);
    } else {
        ....
    }

So the AOF rewrite was handled in the else branch without actually
checking if the pid really matches. This commit makes the check explicit
and logs at WARNING level if the pid returned by wait3() does not match
neither the RDB or AOF rewrite child.
 2012-11-01 22:41:54 +01:00
Yecheng Fu fecc8797c2 fix typo in comments (redis.c, networking.c) 2012-11-01 22:26:49 +01:00
antirez d775b35715 Unix socket clients properly displayed in MONITOR and CLIENT LIST. 
This also fixes issue #745.
 2012-11-01 22:12:50 +01:00
antirez 655e6838fb 32 bit build fixed on Linux. 
It failed because of the way jemalloc was compiled (without passing the
right flags to make, but just to configure). Now the same set of flags
are also passed to the make command, fixing the issue.

This fixes issue #744
 2012-11-01 15:40:46 +01:00
YAMAMOTO Takashi 428b706308 fix a typo in a comment 2012-10-31 09:30:00 +01:00
antirez 9fab63b334 Invert two sides of if expression in SET to avoid a lookup. 
Because of the short circuit behavior of && inverting the two sides of
the if expression avoids an hash table lookup if the non-EX variant of
SET is called.

Thanks to Weibin Yao (@yaoweibin on github) for spotting this.
 2012-10-31 09:27:46 +01:00
antirez 537235bbbb No longer used macro rdbIsOpcode() removed. 2012-10-30 19:12:01 +01:00
antirez 38272f961a help.h update (adds bitop, bitcount, evalsha...) 2012-10-30 19:11:57 +01:00
antirez 6c7e957d17 Ctrl+w support in linenoise. 2012-10-30 19:11:46 +01:00
antirez 6bd105032c Marginally more robust glibc version test for sync_file_range detection. 2012-10-30 19:11:41 +01:00
charsyam 72f42cf40c patch config.h for sync_file_range 2012-10-30 19:11:34 +01:00
antirez 2a49aa1da2 Fix compilation on Linux kernels or glibc versions lacking sync_file_range(). 
This fixes issue #667.

Many thanks to Didier Spezia for the fix.
 2012-10-30 19:11:29 +01:00
antirez 24ee20e5ca Update memory peak stats while loading RDB / AOF. 2012-10-30 19:11:23 +01:00
antirez f7b42b1749 2.7.0 branch created as a fork of 2.6.0. 2012-10-22 23:32:26 +02:00
antirez 5eec376c2f Redis 2.6.0 2012-10-22 23:27:18 +02:00
antirez 99d7dbe669 A filed called slave_read_only added in INFO output. 
This was an important information missing from the INFO output in the
replication section.

It obviously reflects if the slave is read only or not.
 2012-10-22 19:22:48 +02:00
Greg Hurrell a61705dd95 Fix (cosmetic) typos in dict.h 2012-10-22 11:56:06 +02:00
Schuster 16144589f2 redis-check-dump now understands dumps produced by Redis 2.6 
(Commit message from @antirez as it was missign in the original commits,
also the patch was modified a bit to still work with 2.4 dumps and to
avoid if expressions that are always true due to checked types range)

This commit changes redis-check-dump to account for new encodings and
for the new MSTIME expire format. It also refactors the test for valid
type into a function.

The code is still compatible with Redis 2.4 generated dumps.

This fixes issue #709.
 2012-10-22 11:53:43 +02:00
antirez a25b25f4ef Default memory limit for 32bit instanced moved from 3.5 GB to 3 GB. 
In some system, notably osx, the 3.5 GB limit was too far and not able
to prevent a crash for out of memory. The 3 GB limit works better and it
is still a lot of memory within a 4 GB theorical limit so it's not going
to bore anyone :-)

This fixes issue #711
 2012-10-22 10:45:55 +02:00
antirez ab55180883 Differentiate SCRIPT KILL error replies. 
When calling SCRIPT KILL currently you can get two errors:

* No script in timeout (busy) state.
* The script already performed a write.

It is useful to be able to distinguish the two errors, but right now both
start with "ERR" prefix, so string matching (that is fragile) must be used.

This commit introduces two different prefixes.

-NOTBUSY and -UNKILLABLE respectively to reply with an error when no
script is busy at the moment, and when the script already executed a
write operation and can not be killed.
 2012-10-22 10:31:46 +02:00
NanXiao a03c32702b Update src/redis-benchmark.c 
The code of current implementation:

if (c->pending == 0) clientDone(c);
In clientDone function, the c's memory has been freed, then the loop will continue: while(c->pending). The memory of c has been freed now, so c->pending is invalid (c is an invalid pointer now), and this will cause memory dump in some platforams(eg: Solaris).

So I think the code should be modified as:
if (c->pending == 0)
{
clientDone(c);
break;
}
and this will not lead to while(c->pending).
 2012-10-18 11:05:47 +02:00
antirez 2164523244 Fix MULTI / EXEC rendering in MONITOR output. 
Before of this commit it used to be like this:

MULTI
EXEC
... actual commands of the transaction ...

Because after all that is the natural order of things. Transaction
commands are queued and executed *only after* EXEC is called.

However this makes debugging with MONITOR a mess, so the code was
modified to provide a coherent output.

What happens is that MULTI is rendered in the MONITOR output as far as
possible, instead EXEC is propagated only after the transaction is
executed, or even in the case it fails because of WATCH, so in this case
you'll simply see:

MULTI
EXEC

An empty transaction.
 2012-10-16 17:41:39 +02:00
antirez c3ff470889 Merge remote-tracking branch 'origin/2.6' into 2.6 2012-10-11 18:36:18 +02:00
antirez 0e25c0ccf4 Allow AUTH when Redis is busy because of timedout Lua script. 
If the server is password protected we need to accept AUTH when there is
a server busy (-BUSY) condition, otherwise it will be impossible to send
SHUTDOWN NOSAVE or SCRIPT KILL.

This fixes issue #708.
 2012-10-11 18:35:52 +02:00
antirez f4eb4b3352 CONTRIBUTING file updated. 2012-10-06 12:04:38 +02:00
antirez 500dddc264 Redis 2.6.0 RC8 (2.5.14) 2012-10-05 19:08:55 +02:00
Salvatore Sanfilippo a6305f13d5 Merge pull request #544 from dvirsky/2.6 
fixed install script to rewrite the default config
 2012-10-05 03:30:58 -07:00
antirez 99c3338c23 Hash function switched to murmurhash2. 
The previously used hash function, djbhash, is not secure against
collision attacks even when the seed is randomized as there are simple
ways to find seed-independent collisions.

The new hash function appears to be safe (or much harder to exploit at
least) in this case, and has better distribution.

Better distribution does not always means that's better. For instance in
a fast benchmark with "DEBUG POPULATE 1000000" I obtained the following
results:

    1.6 seconds with djbhash
    2.0 seconds with murmurhash2

This is due to the fact that djbhash will hash objects that follow the
pattern `prefix:<id>` and where the id is numerically near, to near
buckets. This improves the locality.

However in other access patterns with keys that have no relation
murmurhash2 has some (apparently minimal) speed advantage.

On the other hand a better distribution should significantly
improve the quality of the distribution of elements returned with
dictGetRandomKey() that is used in SPOP, SRANDMEMBER, RANDOMKEY, and
other commands.

Everything considered, and under the suspect that this commit fixes a
security issue in Redis, we are switching to the new hash function.
If some serious speed regression will be found in the future we'll be able
to step back easiliy.

This commit fixes issue #663.
 2012-10-05 11:16:40 +02:00
antirez 05e06e1543 Warn when configured maxmemory value seems odd. 
This commit warns the user with a log at "warning" level if:

1) After the server startup the maxmemory limit was found to be < 1MB.
2) After a CONFIG SET command modifying the maxmemory setting the limit
is set to a value that is smaller than the currently used memory.

The behaviour of the Redis server is unmodified, and this wil not make
the CONFIG SET command or a wrong configuration in redis.conf less
likely to create problems, but at least this will make aware most users
about a possbile error they committed without resorting to external
help.

However no warning is issued if, as a result of loading the AOF or RDB
file, we are very near the maxmemory setting, or key eviction will be
needed in order to go under the specified maxmemory setting. The reason
is that in servers configured as a cache with an aggressive
maxmemory-policy most of the times restarting the server will cause this
condition to happen if persistence is not switched off.

This fixes issue #429.
 2012-10-05 10:56:35 +02:00
antirez e2f2dab3ef Include time.h in ae.c as we now use time(). 2012-10-05 10:10:34 +02:00
Jokea e480c279bc Force expire all timer events when system clock skew is detected. 
When system time changes back, the timer will not worker properly
hence some core functionality of redis will stop working(e.g. replication,
bgsave, etc). See issue #633 for details.

The patch saves the previous time and when a system clock skew is detected,
it will force expire all timers.

Modiifed by @antirez: the previous time was moved into the eventLoop
structure to make sure the library is still thread safe as long as you
use different event loops into different threads (otherwise you need
some synchronization). More comments added about the reasoning at the
base of the patch, that's worth reporting here:

/* If the system clock is moved to the future, and then set back to the
 * right value, time events may be delayed in a random way. Often this
 * means that scheduled operations will not be performed soon enough.
 *
 * Here we try to detect system clock skews, and force all the time
 * events to be processed ASAP when this happens: the idea is that
 * processing events earlier is less dangerous than delaying them
 * indefinitely, and practice suggests it is. */
 2012-10-04 19:30:48 +02:00
antirez 0c19880ce6 "Timeout receiving bulk data" error message modified. 
The new message now contains an hint about modifying the repl-timeout
configuration directive if the problem persists.

This should normally not be needed, because while the master generates
the RDB file it makes sure to send newlines to the replication channel
to prevent timeouts. However there are times when masters running on
very slow systems can completely stop for seconds during the RDB saving
process. In such a case enlarging the timeout value can fix the problem.

See issue #695 for an example of this problem in an EC2 deployment.
 2012-10-04 11:49:17 +02:00
antirez 2ba962714a "SORT by nosort" (skip sorting) respect sorted set ordering. 
When SORT is called with the option BY set to a string constant not
inclduing the wildcard character "*", there is no way to sort the output
so any ordering is valid. This allows the SORT internals to optimize its
work and don't really sort the output at all.

However it was odd that this option was not able to retain the natural
order of a sorted set. This feature was requested by users multiple
times as sometimes to call SORT with GET against sorted sets as a way to
mass-fetch objects can be handy.

This commit introduces two things:

1) The ability of SORT to return sorted sets elements in their natural
ordering when `BY nosort` is specified, accordingly to `DESC / ASC` options.
2) The ability of SORT to optimize this case further if LIMIT is passed
as well, avoiding to really fetch the whole sorted set, but directly
obtaining the specified range.

Because in this case the sorting is always deterministic, no
post-sorting activity is performed when SORT is called from a Lua
script.

This commit fixes issue #98.
 2012-10-03 18:46:48 +02:00
antirez 151b606c11 Revert "Scripting: redis.NIL to return nil bulk replies." 
This reverts commit e061d797d7.

Conflicts:

	src/scripting.c
 2012-10-01 10:10:03 +02:00
antirez f1466e11cd Scripting: add helper functions redis.error_reply() and redis.status_reply(). 
A previous commit introduced Redis.NIL. This commit adds similar helper
functions to return tables with a single field set to the specified
string so that instead of using 'return {err="My Error"}' it is possible
to use a more idiomatic form:

    return redis.error_reply("My Error")
    return redis.status_reply("OK")
 2012-09-28 16:58:38 +02:00
antirez e061d797d7 Scripting: redis.NIL to return nil bulk replies. 
Lua arrays can't contain nil elements (see
http://www.lua.org/pil/19.1.html for more information), so Lua scripts
were not able to return a multi-bulk reply containing nil bulk
elements inside.

This commit introduces a special conversion: a table with just
a "nilbulk" field set to a boolean value is converted by Redis as a nil
bulk reply, but at the same time for Lua this type is not a "nil" so can
be used inside Lua arrays.

This type is also assigned to redis.NIL, so the following two forms
are equivalent and will be able to return a nil bulk reply as second
element of a three elements array:

    EVAL "return {1,redis.NIL,3}" 0
    EVAL "return {1,{nilbulk=true},3}" 0

The result in redis-cli will be:

    1) (integer) 1
    2) (nil)
    3) (integer) 3
 2012-09-28 14:19:15 +02:00
Erik Dubbelboer 04779bdfe1 Fixed some spelling errors in the comments 2012-09-27 13:22:42 +02:00
Erik Dubbelboer e04be06e89 Added consts keyword where possible 2012-09-27 13:12:05 +02:00
antirez c4cbffa3a1 Final merge of Sentinel into 2.6. 
After cherry-picking Sentinel commits a few spurious issues remained
about references to Redis Cluster that is not present in the 2.6 branch.
 2012-09-27 13:10:36 +02:00
antirez dfb7194cba Sentinel: Support for AUTH. 2012-09-27 13:06:17 +02:00
antirez b8ce9a84c5 Sentinel: reply -IDONTKNOW to get-master-addr-by-name on lack of info. 
If we don't have any clue about a master since it never replied to INFO
so far, reply with an -IDONTKNOW error to SENTINEL
get-master-addr-by-name requests.
 2012-09-27 13:06:12 +02:00
antirez 1f8bd82332 Sentinel: more easy master redirection if master is a slave. 
Before this commit Sentienl used to redirect master ip/addr if the
current instance reported to be a slave only if this was the first INFO
output received, and the role was found to be slave.

Now instead also if we find that the runid is different, and the
reported role is slave, we also redirect to the reported master ip/addr.

This unifies the behavior of Sentinel in the case of a reboot (where it
will see the first INFO output with the wrong role and will perform the
redirection), with the behavior of Sentinel in the case of a change in
what it sees in the INFO output of the master.
 2012-09-27 13:06:05 +02:00
antirez ef792fc950 Sentinel: do not crash against slaves not publishing the runid. 
Older versions of Redis (before 2.4.17) don't publish the runid field in
INFO. This commit makes Sentinel able to handle that without crashing.
 2012-09-27 13:06:01 +02:00
antirez de499f7f7e Sentinel: INFO command implementation. 2012-09-27 13:05:58 +02:00
antirez b65f3c2176 Sentinel: add Redis execution mode to INFO output. 
The new "redis_mode" field in the INFO output will show if Redis is
running in standalone mode, cluster, or sentinel mode.
 2012-09-27 13:05:53 +02:00
antirez 161e137c55 Sentinel: Sentinel-side support for slave priority. 
The slave priority that is now published by Redis in INFO output is
now used by Sentinel in order to select the slave with minimum priority
for promotion, and in order to consider slaves with priority set to 0 as
not able to play the role of master (they will never be promoted by
Sentinel).

The "slave-priority" field is now one of the fileds that Sentinel
publishes when describing an instance via the SENTINEL commands such as
"SENTINEL slaves mastername".
 2012-09-27 13:05:49 +02:00
antirez d480b9ce7f Sentinel: suppress harmless warning by initializing 'table' to NULL. 
Note that the assertion guarantees that one of the if branches setting
table is always entered.
 2012-09-27 13:05:45 +02:00
antirez fa23fc3363 Sentinel: send SCRIPT KILL on -BUSY reply and SDOWN instance. 
From the point of view of Redis an instance replying -BUSY is down,
since it is effectively not able to reply to user requests. However
a looping script is a recoverable condition in Redis if the script still
did not performed any write to the dataset. In that case performing a
fail over is not optimal, so Sentinel now tries to restore the normal server
condition killing the script with a SCRIPT KILL command.

If the script already performed some write before entering an infinite
(or long enough to timeout) loop, SCRIPT KILL will not work and the
fail over will be triggered anyway.
 2012-09-27 13:05:41 +02:00
antirez fc0a0d4aa7 Sentinel: fixed a crash on script execution. 
The call to sentinelScheduleScriptExecution() lacked the final NULL
argument to signal the end of arguments. This resulted into a crash.
 2012-09-27 13:05:38 +02:00
antirez ea9bec50c6 Sentinel: SENTINEL FAILOVER command implemented. 
This command can be used in order to force a Sentinel instance to start
a failover for the specified master, as leader, forcing the failover
even if the master is up.

The commit also adds some minor refactoring and other improvements to
functions already implemented that make them able to work when the
master is not in SDOWN condition. For instance slave selection
assumed that we ask INFO every second to every slave, this is true
only when the master is in SDOWN condition, so slave selection did not
worked when the master was not in SDOWN condition.
 2012-09-27 13:05:33 +02:00
antirez 26a340095d Sentinel: client reconfiguration script execution. 
This commit adds support to optionally execute a script when one of the
following events happen:

* The failover starts (with a slave already promoted).
* The failover ends.
* The failover is aborted.

The script is called with enough parameters (documented in the example
sentinel.conf file) to provide information about the old and new ip:port
pair of the master, the role of the sentinel (leader or observer) and
the name of the master.

The goal of the script is to inform clients of the configuration change
in a way specific to the environment Sentinel is running, that can't be
implemented in a genereal way inside Sentinel itself.
 2012-09-27 13:05:30 +02:00
antirez 524b79d231 Sentinel: when leader in wait-start, sense another leader as race. 
When we are in wait start, if another leader (or any other external
entity) turns a slave into a master, abort the failover, and detect it
as an observer.

Note that the wait-start state is mainly there for this reason but the
abort was yet not implemented.

This adds a new sentinel event -failover-abort-race.
 2012-09-27 13:05:26 +02:00
antirez 201ed6d42e Sentinel: sentinelRefreshInstanceInfo() comments improved a bit. 2012-09-27 13:05:19 +02:00
antirez 7c9bfe1036 Sentinel: sentinel.conf self-documenation improved. 2012-09-27 13:05:15 +02:00
antirez 3da75e2ca4 Sentinel: abort failover when in wait-start if master is back. 
When we are a Leader Sentinel in wait-start state, starting with this
commit the failover is aborted if the master returns online.

This improves the way we handle a notable case of net split, that is the
split between Sentinels and Redis servers, that will be a very common
case of split becase Sentinels will often be installed in the client's
network and servers can be in a differnt arm of the network.

When Sentinels and Redis servers are isolated the master is in ODOWN
condition since the Sentinels can agree about this state, however the
failover does not start since there are no good slaves to promote (in
this specific case all the slaves are unreachable).

However when the split is resolved, Sentinels may sense the slave back
a moment before they sense the master is back, so the failover may start
without a good reason (since the master is actually working too).

Now this condition is reversible, so the failover will be aborted
immediately after if the master is detected to be working again, that
is, not in SDOWN nor in ODOWN condition.
 2012-09-27 13:05:11 +02:00
antirez e328e41a3a Sentinel: scripts execution engine improved. 
We no longer use a vanilla fork+execve but take a queue of jobs of
scripts to execute, with retry on error, timeouts, and so forth.

Currently this is used only for notifications but soon the ability to
also call clients reconfiguration scripts will be added.
 2012-09-27 13:05:07 +02:00
Jan-Erik Rediger 8a8e560b87 Include sys/wait.h to avoid compiler warning 
gcc warned about an implicit declaration of function 'wait3'.
Including this header fixes this.
 2012-09-27 13:04:51 +02:00
antirez 0d0975f2b6 Sentinel: don't start a failover as leader if there is no good slave. 2012-09-27 13:04:48 +02:00
Jeremy Zawodny af41f6cffb comment fix 
improve English a bit. :-)
 2012-09-27 13:04:44 +02:00
antirez 999fe0d352 Sentinel: ability to execute notification scripts. 2012-09-27 13:04:41 +02:00
mrb f1057534e7 Fix warning in redis.c for sentinel config load 2012-09-27 13:04:37 +02:00
mrb fcc8bf9974 Some cleanup in sentinel.conf 2012-09-27 13:04:32 +02:00
antirez 374eed7d2a Sentinel: abort failover if no good slave is available. 
The previous behavior of the state machine was to wait some time and
retry the slave selection, but this is not robust enough against drastic
changes in the conditions of the monitored instances.

What we do now when the slave selection fails is to abort the failover
and return back monitoring the master. If the ODOWN condition is still
present a new failover will be triggered and so forth.

This commit also refactors the code we use to abort a failover.
 2012-09-27 13:04:28 +02:00
antirez 2085fdb1f4 Sentinel: reset pending_commands in a more generic way. 2012-09-27 13:04:24 +02:00
antirez f8a19e32e2 Prevent a spurious +sdown event on switch. 
When we reset the master we should start with clean timestamps for ping
replies otherwise we'll detect a spurious +sdown event, because on
+master-switch event the previous master instance was probably in +sdown
condition. Since we updated the address we should count time from
scratch again.

Also this commit makes sure to explicitly reset the count of pending
commands, now we can do this because of the new way the hiredis link
is closed.
 2012-09-27 13:04:19 +02:00
antirez 7c39b55d42 Sentinel: debugging message removed. 2012-09-27 13:04:16 +02:00
antirez e47236d8d4 Sentinel: changes to connection handling and redirection. 
We disconnect the Redis instances hiredis link in a more robust way now.
Also we change the way we perform the redirection for the +switch-master
event, that is not just an instance reset with an address change.

Using the same system we now implement the +redirect-to-master event
that is triggered by an instance that is configured to be master but
found to be a slave at the first INFO reply. In that case we monitor the
master instead, logging the incident as an event.
 2012-09-27 13:04:12 +02:00
antirez 8ab7e998d1 Sentinel: check that instance still exists in reply callbacks. 
We can't be sure the instance object still exists when the reply
callback is called.
 2012-09-27 13:04:08 +02:00
antirez e01a415d37 Sentinel: more robust failover detection as observer. 
Sentinel observers detect failover checking if a slave attached to the
monitored master turns into its replication state from slave to master.
However while this change may in theory only happen after a SLAVEOF NO
ONE command, in practie it is very easy to reboot a slave instance with
a wrong configuration that turns it into a master, especially if it was
a past master before a successfull failover.

This commit changes the detection policy so that if an instance goes
from slave to master, but at the same time the runid has changed, we
sense a reboot, and in that case we don't detect a failover at all.

This commit also introduces the "reboot" sentinel event, that is logged
at "warning" level (so this will trigger an admin notification).

The commit also fixes a problem in the disconnect handler that assumed
that the instance object always existed, that is not the case. Now we
no longer assume that redisAsyncFree() will call the disconnection
handler before returning.
 2012-09-27 13:04:02 +02:00
antirez d26a8fb4db Fixed an error in the example sentinel.conf. 2012-09-27 13:03:55 +02:00
antirez 5b5eb192f5 Typo. 2012-09-27 13:03:49 +02:00
antirez 120ba3922d First implementation of Redis Sentinel. 
This commit implements the first, beta quality implementation of Redis
Sentinel, a distributed monitoring system for Redis with notification
and automatic failover capabilities.

More info at http://redis.io/topics/sentinel
 2012-09-27 13:03:41 +02:00
antirez 2812b945f0 Test for SRANDMEMBER with <count>. 2012-09-21 11:56:04 +02:00
antirez 31fe053a62 SRANDMEMBER <count> leak fixed. 
For "CASE 4" (see code) we need to free the element if it's already in
the result dictionary and adding it failed.
 2012-09-21 11:56:00 +02:00
antirez dd94771578 Added the SRANDMEMBER key <count> variant. 
SRANDMEMBER called with just the key argument can just return a single
random element from a Redis Set. However many users need to return
multiple unique elements from a Set, this is not a trivial problem to
handle in the client side, and for truly good performance a C
implementation was required.

After many requests for this feature it was finally implemented.

The problem implementing this command is the strategy to follow when
the number of elements the user asks for is near to the number of
elements that are already inside the set. In this case asking random
elements to the dictionary API, and trying to add it to a temporary set,
may result into an extremely poor performance, as most add operations
will be wasted on duplicated elements.

For this reason this implementation uses a different strategy in this
case: the Set is copied, and random elements are returned to reach the
specified count.

The code actually uses 4 different algorithms optimized for the
different cases.

If the count is negative, the command changes behavior and allows for
duplicated elements in the returned subset.
 2012-09-21 11:55:56 +02:00
antirez 8b6b1b27cc Fix compilation on FreeBSD. Thanks to @koobs on twitter. 2012-09-17 12:45:57 +02:00
antirez 4403862623 Redis 2.5.13 (2.6.0 RC7). 2012-09-17 11:02:49 +02:00
antirez 174518ffb7 .gitignore modified to be more general with less entries. 2012-09-17 10:50:07 +02:00
antirez f444e2afdc A reimplementation of blocking operation internals. 
Redis provides support for blocking operations such as BLPOP or BRPOP.
This operations are identical to normal LPOP and RPOP operations as long
as there are elements in the target list, but if the list is empty they
block waiting for new data to arrive to the list.

All the clients blocked waiting for th same list are served in a FIFO
way, so the first that blocked is the first to be served when there is
more data pushed by another client into the list.

The previous implementation of blocking operations was conceived to
serve clients in the context of push operations. For for instance:

1) There is a client "A" blocked on list "foo".
2) The client "B" performs `LPUSH foo somevalue`.
3) The client "A" is served in the context of the "B" LPUSH,
synchronously.

Processing things in a synchronous way was useful as if "A" pushes a
value that is served by "B", from the point of view of the database is a
NOP (no operation) thing, that is, nothing is replicated, nothing is
written in the AOF file, and so forth.

However later we implemented two things:

1) Variadic LPUSH that could add multiple values to a list in the
context of a single call.
2) BRPOPLPUSH that was a version of BRPOP that also provided a "PUSH"
side effect when receiving data.

This forced us to make the synchronous implementation more complex. If
client "B" is waiting for data, and "A" pushes three elemnents in a
single call, we needed to propagate an LPUSH with a missing argument
in the AOF and replication link. We also needed to make sure to
replicate the LPUSH side of BRPOPLPUSH, but only if in turn did not
happened to serve another blocking client into another list ;)

This were complex but with a few of mutually recursive functions
everything worked as expected... until one day we introduced scripting
in Redis.

Scripting + synchronous blocking operations = Issue #614.

Basically you can't "rewrite" a script to have just a partial effect on
the replicas and AOF file if the script happened to serve a few blocked
clients.

The solution to all this problems, implemented by this commit, is to
change the way we serve blocked clients. Instead of serving the blocked
clients synchronously, in the context of the command performing the PUSH
operation, it is now an asynchronous and iterative process:

1) If a key that has clients blocked waiting for data is the subject of
a list push operation, We simply mark keys as "ready" and put it into a
queue.
2) Every command pushing stuff on lists, as a variadic LPUSH, a script,
or whatever it is, is replicated verbatim without any rewriting.
3) Every time a Redis command, a MULTI/EXEC block, or a script,
completed its execution, we run the list of keys ready to serve blocked
clients (as more data arrived), and process this list serving the
blocked clients.
4) As a result of "3" maybe more keys are ready again for other clients
(as a result of BRPOPLPUSH we may have push operations), so we iterate
back to step "3" if it's needed.

The new code has a much simpler semantics, and a simpler to understand
implementation, with the disadvantage of not being able to "optmize out"
a PUSH+BPOP as a No OP.

This commit will be tested with care before the final merge, more tests
will be added likely.
 2012-09-17 10:26:50 +02:00
antirez b58f03a0e8 Make sure that SELECT argument is an integer or return an error. 
Unfortunately we had still the lame atoi() without any error checking in
place, so "SELECT foo" would work as "SELECT 0". This was not an huge
problem per se but some people expected that DB can be strings and not
just numbers, and without errors you get the feeling that they can be
numbers, but not the behavior.

Now getLongFromObjectOrReply() is used as almost everybody else across
the code, generating an error if the number is not an integer or
overflows the long type.

Thanks to @mipearson for reporting that on Twitter.
 2012-09-11 10:34:48 +02:00
antirez efb54f0593 Match printf format with actual type in genRedisInfoString(). 2012-09-10 12:43:20 +02:00
antirez 58889867bb BITCOUNT regression test for #582 fixed for 32 bit target. 
Bug #582 was not present in 32 bit builds of Redis as
getObjectFromLong() will return an error for overflow.

This commit makes sure that the test does not fail because of the error
returned when running against 32 bit builds.
 2012-09-05 17:50:24 +02:00
Haruto Otake 4c3d419013 BITCOUNT: fix segmentation fault. 
remove unsafe and unnecessary cast.
until now, this cast may lead segmentation fault when end > UINT_MAX

setbit foo 0 1
bitcount  0 4294967295
=> ok
bitcount  0 4294967296
=> cause segmentation fault.

Note by @antirez: the commit was modified a bit to also change the
string length type to long, since it's guaranteed to be at max 512 MB in
size, so we can work with the same type across all the code path.

A regression test was also added.
 2012-09-05 16:20:21 +02:00
Saj Goonatilleke 0671d88cab Bug fix: slaves being pinged every second 
REDIS_REPL_PING_SLAVE_PERIOD controls how often the master should
transmit a heartbeat (PING) to its slaves.  This period, which defaults
to 10, is measured in seconds.

Redis 2.4 masters used to ping their slaves every ten seconds, just like
it says on the tin.

The Redis 2.6 masters I have been experimenting with, on the other hand,
ping their slaves *every second*.  (master_last_io_seconds_ago never
approaches 10.)  I think the ping period was inadvertently slashed to
one-tenth of its nominal value around the time REDIS_HZ was introduced.
This commit reintroduces correct ping schedule behaviour.
 2012-09-05 16:01:01 +02:00
antirez 5ddee9b7d5 Scripting: Force SORT BY constant determinism inside SORT itself. 
SORT is able to return (faster than when ordering) unordered output if
the "BY" clause is used with a constant value. However we try to play
well with scripting requirements of determinism providing always sorted
outputs when SORT (and other similar commands) are called by Lua
scripts.

However we used the general mechanism in place in scripting in order to
reorder SORT output, that is, if the command has the "S" flag set, the
Lua scripting engine will take an additional step when converting a
multi bulk reply to Lua value, calling a Lua sorting function.

This is suboptimal as we can do it faster inside SORT itself.
This is also broken as issue #545 shows us: basically when SORT is used
with a constant BY, and additionally also GET is used, the Lua scripting
engine was trying to order the output as a flat array, while it was
actually a list of key-value pairs.

What we do know is to recognized if the caller of SORT is the Lua client
(since we can check this using the REDIS_LUA_CLIENT flag). If so, and if
a "don't sort" condition is triggered by the BY option with a constant
string, we force the lexicographical sorting.

This commit fixes this bug and improves the performance, and at the same
time simplifies the implementation. This does not mean I'm smart today,
it means I was stupid when I committed the original implementation ;)
 2012-09-05 01:19:47 +02:00
antirez fd2a8951bf Send an async PING before starting replication with master. 
During the first synchronization step of the replication process, a Redis
slave connects with the master in a non blocking way. However once the
connection is established the replication continues sending the REPLCONF
command, and sometimes the AUTH command if needed. Those commands are
send in a partially blocking way (blocking with timeout in the order of
seconds).

Because it is common for a blocked master to accept connections even if
it is actually not able to reply to the slave requests, it was easy for
a slave to block if the master had serious issues, but was still able to
accept connections in the listening socket.

For this reason we now send an asynchronous PING request just after the
non blocking connection ended in a successful way, and wait for the
reply before to continue with the replication process. It is very
unlikely that a master replying to PING can't reply to the other
commands.

This solution was proposed by Didier Spezia (Thanks!) so that we don't
need to turn all the replication process into a non blocking affair, but
still the probability of a slave blocked is minimal even in the event of
a failing master.

Also we now use getsockopt(SO_ERROR) in order to check errors ASAP
in the event handler, instead of waiting for actual I/O to return an
error.

This commit fixes issue #632.
 2012-09-03 11:48:27 +02:00
antirez 42a239b888 Scripting: Reset Lua fake client reply_bytes after command execution. 
Lua scripting uses a fake client in order to run commands in the context
of a client, accumulate the reply, and convert it into a Lua object
to return to the caller. This client is reused again and again, and is
referenced by the server.lua_client globally accessible pointer.

However after every call to redis.call() or redis.pcall(), that is
handled by the luaRedisGenericCommand() function, the reply_bytes field
of the client was not set back to zero. This filed is used to estimate
the amount of memory currently used in the reply. Because of the lack of
reset, script after script executed, this value used to get bigger and
bigger, and in the end on 32 bit systems it triggered the following
assert:

    redisAssert(c->reply_bytes < ULONG_MAX-(1024*64));

On 64 bit systems this does not happen because it takes too much time to
reach values near to 2^64 for users to see the practical effect of the
bug.

Now in the cleanup stage of luaRedisGenericCommand() we reset the
reply_bytes counter to zero, avoiding the issue. It is not practical to
add a test for this bug, but the fix was manually tested using a
debugger.

This commit fixes issue #656.
 2012-08-31 11:08:53 +02:00
antirez 851ac9d072 Sentinel: added documentation about slave-priority in redis.conf 2012-08-31 10:30:51 +02:00
antirez 48d26a483d Sentinel: Redis-side support for slave priority. 
A Redis slave can now be configured with a priority, that is an integer
number that is shown in INFO output and can be get and set using the
redis.conf file or the CONFIG GET/SET command.

This field is used by Sentinel during slave election. A slave with lower
priority is preferred. A slave with priority zero is never elected (and
is considered to be impossible to elect even if it is the only slave
available).

A next commit will add support in the Sentinel side as well.
 2012-08-31 10:30:29 +02:00
antirez edfaa64f49 Scripting: require at least one argument for redis.call(). 
Redis used to crash with a call like the following:

    EVAL "redis.call()" 0

Now the explicit check for at least one argument prevents the problem.

This commit fixes issue #655.
 2012-08-31 10:28:36 +02:00
antirez 13732168a5 Incrementally flush RDB on disk while loading it from a master. 
This fixes issue #539.

Basically if there is enough free memory the OS may buffer the RDB file
that the slave transfers on disk from the master. The file may
actually be flused on disk at once by the operating system when it gets
closed by Redis, causing the close system call to block for a long time.

This patch is a modified version of one provided by yoav-steinberg of
@garantiadata (the original version was posted in the issue #539
comments), and tries to flush the OS buffers incrementally (every 8 MB
of loaded data).
 2012-08-28 12:47:35 +02:00
antirez 06bd3b9acd Fix a forget zmalloc_oom() -> zmalloc_oom_handler() replacement. 2012-08-24 15:41:49 +02:00
antirez 5de75120ba Better Out of Memory handling. 
The previous implementation of zmalloc.c was not able to handle out of
memory in an application-specific way. It just logged an error on
standard error, and aborted.

The result was that in the case of an actual out of memory in Redis
where malloc returned NULL (In Linux this actually happens under
specific overcommit policy settings and/or with no or little swap
configured) the error was not properly logged in the Redis log.

This commit fixes this problem, fixing issue #509.
Now the out of memory is properly reported in the Redis log and a stack
trace is generated.

The approach used is to provide a configurable out of memory handler
to zmalloc (otherwise the default one logging the event on the
standard output is used).
 2012-08-24 13:03:40 +02:00
antirez 32095c4057 redis-benchmark: disable big buffer cleanup in hiredis context. 
This new hiredis features allows us to reuse a previous context reader
buffer even if already very big in order to maximize performances with
big payloads (Usually hiredis re-creates buffers when they are too big
and unused in order to save memory).
 2012-08-22 11:34:03 +02:00
antirez 7fcba9fd9a hiredis library updated. 
This version of hiredis merges modifications of the Redis fork with
latest changes in the hiredis repository.

The same version was pushed on the hiredis repository and will probably
merged into the master branch in short time.
 2012-08-22 11:33:57 +02:00
Pieter Noordhuis 2f44452612 Set p to its new offset before modifying it 2012-08-22 11:33:52 +02:00
Pieter Noordhuis 89bf6f58fd Add ziplist test for deleting next to last entries 2012-08-22 11:33:47 +02:00
Tobias Schwab 013189e7db Fix version numbers 2012-08-02 14:38:42 +02:00
antirez 73d3e8751b Redis 2.5.12 (2.6 RC6). 2012-08-01 12:06:03 +02:00
Michael Parker 628890e43e Use correct variable name for value to convert. 
Note by @antirez: this code was never compiled because utils.c lacked the
float.h include, so we never noticed this variable was mispelled in the
past.

This should provide a noticeable speed boost when saving certain types
of databases with many sorted sets inside.
 2012-07-31 11:50:51 +02:00
Saj Goonatilleke 4c0c1fff5a Truncate short write from the AOF 
If Redis only manages to write out a partial buffer, the AOF file won't
load back into Redis the next time it starts up.  It is better to
discard the short write than waste time running redis-check-aof.
 2012-07-31 10:58:16 +02:00
Saj Goonatilleke f00b0844c9 New in INFO: aof_last_bgrewrite_status 
Behaves like rdb_last_bgsave_status -- even down to reporting 'ok' when
no rewrite has been done yet.  (You might want to check that
aof_last_rewrite_time_sec is not -1.)
 2012-07-31 10:58:12 +02:00
Steeve Lennmark 889e443ce5 Check that we have connection before enabling pipe mode 2012-07-22 17:19:05 +02:00
antirez 3bb3f12539 Allow Pub/Sub in contexts where other commands are blocked. 
Redis loading data from disk, and a Redis slave disconnected from its
master with serve-stale-data disabled, are two conditions where
commands are normally refused by Redis, returning an error.

However there is no reason to disable Pub/Sub commands as well, given
that this layer does not interact with the dataset. To allow Pub/Sub in
as many contexts as possible is especially interesting now that Redis
Sentinel uses Pub/Sub of a Redis master as a communication channel
between Sentinels.

This commit allows Pub/Sub to be used in the above two contexts where
it was previously denied.
 2012-07-22 17:18:12 +02:00
antirez 82675c86a6 Don't assume that "char" is signed. 
For the C standard char can be either signed or unsigned, it's up to the
compiler, but Redis assumed that it was signed in a few places.

The practical effect of this patch is that now Redis 2.6 will run
correctly in every system where char is unsigned, notably the RaspBerry
PI and other ARM systems with GCC.

Thanks to Georgi Marinov (@eesn on twitter) that reported the problem
and allowed me to use his RaspBerry via SSH to trace and fix the issue!
 2012-07-18 12:01:43 +02:00
jokea 8a8e01f4a7 mark fd as writable when EPOLLERR or EPOLLHUP is returned by epoll_wait. 2012-07-09 12:15:07 +02:00
antirez d3d567428a Typo in comment. 2012-07-07 17:24:40 +02:00
antirez dbd8c753c4 REPLCONF internal command introduced. 
The REPLCONF command is an internal command (not designed to be directly
used by normal clients) that allows a slave to set some replication
related state in the master before issuing SYNC to start the
replication.

The initial motivation for this command, and the only reason currently
it is used by the implementation, is to let the slave instance
communicate its listening port to the slave, so that the master can
show all the slaves with their listening ports in the "replication"
section of the INFO output.

This allows clients to auto discover and query all the slaves attached
into a master.

Currently only a single option of the REPLCONF command is supported, and
it is called "listening-port", so the slave now starts the replication
process with something like the following chat:

    REPLCONF listening-prot 6380
    SYNC

Note that this works even if the master is an older version of Redis and
does not understand REPLCONF, because the slave ignores the REPLCONF
error.

In the future REPLCONF can be used for partial replication and other
replication related features where there is the need to exchange
information between master and slave.

NOTE: This commit also fixes a bug: the INFO outout already carried
information about slaves, but the port was broken, and was obtained
with getpeername(2), so it was actually just the ephemeral port used
by the slave to connect to the master as a client.
 2012-07-07 17:24:33 +02:00
antirez b3f28b90d2 Fixed comment typo into time_independent_strcmp(). 2012-06-21 14:26:00 +02:00
antirez 4b3865cbdb Fixed a timing attack on AUTH (Issue #560). 
The way we compared the authentication password using strcmp() allowed
an attacker to gain information about the password using a well known
class of attacks called "timing attacks".

The bug appears to be practically not exploitable in most modern systems
running Redis since even using multiple bytes of differences in the
input at a time instead of one the difference in running time in in the
order of 10 nanoseconds, making it hard to exploit even on LAN. However
attacks always get better so we are providing a fix ASAP.

The new implementation uses two fixed length buffers and a constant time
comparison function, with the goal of:

1) Completely avoid leaking information about the content of the
password, since the comparison is always performed between 512
characters and without conditionals.
2) Partially avoid leaking information about the length of the
password.

About "2" we still have a stage in the code where the real password and
the user provided password are copied in the static buffers, we also run
two strlen() operations against the two inputs, so the running time
of the comparison is a fixed amount plus a time proportional to
LENGTH(A)+LENGTH(B). This means that the absolute time of the operation
performed is still related to the length of the password in some way,
but there is no way to change the input in order to get a difference in
the execution time in the comparison that is not just proportional to
the string provided by the user (because the password length is fixed).

Thus in practical terms the user should try to discover LENGTH(PASSWORD)
looking at the whole execution time of the AUTH command and trying to
guess a proportionality between the whole execution time and the
password length: this appears to be mostly unfeasible in the real world.

Also protecting from this attack is not very useful in the case of Redis
as a brute force attack is anyway feasible if the password is too short,
while with a long password makes it not an issue that the attacker knows
the length.
 2012-06-21 12:01:07 +02:00
antirez 0c9cf45270 Redis 2.5.11 (2.6 RC5). 2012-06-15 13:44:17 +02:00
antirez 6fe9c402a2 Fix c->reply_bytes computation in setDeferredMultiBulkLength() 
In order to implement reply buffer limits introduced in 2.6 and useful
to close the connection under user-selected circumastances of big output
buffers (for instance slow consumers in pub/sub, a blocked slave, and so
forth) Redis takes a counter with the amount of used memory in objects
inside the output list stored into c->reply.

The computation was broken in the function setDeferredMultiBulkLength(),
in the case the object was glued with the next one. This caused the
c->reply_bytes field to go out of sync, be subtracted more than needed,
and wrap back near to ULONG_MAX values.

This commit fixes this bug and adds an assertion that is able to trap
this class of problems.

This problem was discovered looking at the INFO output of an unrelated
issue (issue #547).
 2012-06-15 10:11:27 +02:00
antirez 8361d6c406 ziplistFind(): don't assume that entries are comparable by encoding. 
Because Redis 2.6 introduced new integer encodings it is no longer true
that if two entries have a different encoding they are not equal.

An old ziplist can be loaded from an RDB file generated with Redis 2.4,
in this case for instance a small unsigned integers is encoded with a
16 bit encoding, while in Redis 2.6 a more specific 8 bit encoding
format is used.

Because of this bug hashes ended with duplicated values or fields lookup
failed, causing many bad behaviors.
This in turn caused a crash while converting the ziplist encoded hash into
a real hash table because an assertion was raised on duplicated elements.

This commit fixes issue #547.

Many thanks to Pinterest's Marty Weiner and colleagues for discovering
the problem and helping us in the debugging process.
 2012-06-15 10:11:23 +02:00
Ted Nyman e612508d38 Standardize punctuation in redis-cli help. 
Right there is a mix of help entries ending with periods or
without periods. This standardizes the end of command as without
periods, which seems to be the general custom in most unix tools,
at least.
 2012-06-13 10:27:23 +02:00
antirez 1364395f18 Added a new hash fuzzy tester. 
The new fuzzy tester also removes elements from the hash instead of just
adding random fields. This should increase the probability to find bugs
in the implementations of the hash type internal representations.
 2012-06-12 15:20:16 +02:00
antirez 0b8441c1b2 Dump ziplist hex value on failed assertion. 
The ziplist -> hashtable conversion code is triggered every time an hash
value must be promoted to a full hash table because the number or size of
elements reached the threshold.

If a problem in the ziplist causes the same field to be present
multiple times, the assertion of successful addition of the element
inside the hash table will fail, crashing server with a failed
assertion, but providing little information about the problem.

This code adds a new logging function to perform the hex dump of binary
data, and makes sure that the ziplist -> hashtable conversion code uses
this new logging facility to dump the content of the ziplist when the
assertion fails.

This change was originally made in order to investigate issue #547.
 2012-06-12 00:49:32 +02:00
antirez a85c89854f New test: hash ziplist -> hashtable encoding conversion. 
A new stress test was added to stress test the code converting a ziplist
into an hash table.

In this commit also randomValue helper function was modified to also
return negative values.
 2012-06-11 15:21:44 +02:00
dvir volk 0ea1a9c452 fixed server install script to rewrite the default configuration file and not a template, and removed the old config template 2012-06-08 16:03:18 +03:00
antirez 1a3e9d951b Version bumped to 2.5.10 (2.6 RC4) + Release Notes. 2012-06-07 11:41:38 +02:00
antirez 3f12656781 EVAL replication test: less false positives. 
wait_for_condition is now used instead of the usual "after 1000" (that
is the way to sleep in Tcl). This should avoid to find the replica in
a state where it is loading the RDB in memory, returning -LOADING error.

This test used to fail when running the test over valgrind, due to the
added latencies.
 2012-06-02 23:32:01 +02:00
Alex Mitrofanov 591c9e6543 Fixed RESTORE hash failure (Issue #532) 
(additional commit notes by antirez@gmail.com):

The rdbIsObjectType() macro was not updated when the new RDB object type
of ziplist encoded hashes was added.

As a result RESTORE, that uses rdbLoadObjectType(), failed when a
ziplist encoded hash was loaded.
This does not affected normal RDB loading because in that case we use
the lower-level function rdbLoadType().

The commit also adds a regression test.
 2012-06-02 10:27:31 +02:00
antirez ebdcd723a7 RDB type loading functions clarified in comments. 
Improved comments to make clear that rdbLoadType() just loads a
general TYPE in the context of RDB that can be an object type or an
expire type, end-of-file, and so forth.

While rdbLoadObjectType() enforces that the type is a valid Object Type
otherwise it returns -1.
 2012-06-02 10:27:27 +02:00
antirez 8f0658cdd0 BITOP bug when called against non existing keys fixed. 
In the issue #529 an user reported a bug that can be triggered with the
following code:

flushdb
set a
"\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00"
bitop or x a b

The bug was introduced with the speed optimization in commit 8bbc076
that specializes every BITOP operation loop up to the minimum length of
the input strings.

However the computation of the minimum length contained an error when a
non existing key was present in the input, after a key that was non zero
length.

This commit fixes the bug and adds a regression test for it.
 2012-05-31 21:51:52 +02:00
antirez 9a8d51add5 2.6 Release notes: added more new features to the list. 2012-05-27 17:17:22 +02:00
antirez 473f3090f2 Tests modified to account for INFO fields renaming. 
Commit 33e1db36fa modified the name of a
few INFO fields. This commit changes the Redis test to account for this
changes.
 2012-05-25 16:25:07 +02:00
antirez 69396249e5 Release notes: more info about 2.4 -> 2.6 migration. 
Migration information moved at the end of the file as now they start to
be too long to stay at the top, but a warning was added at the top of
the file to remember the user to check.

Added information about INFO fields that changed name between 2.4 and
2.6.
 2012-05-25 12:24:20 +02:00
antirez 6b4d92e21d Four new persistence fields in INFO. A few renamed. 
The 'persistence' section of INFO output now contains additional four
fields related to RDB and AOF persistence:

 rdb_last_bgsave_time_sec       Duration of latest BGSAVE in sec.
 rdb_current_bgsave_time_sec    Duration of current BGSAVE in sec.
 aof_last_rewrite_time_sec      Duration of latest AOF rewrite in sec.
 aof_current_rewrite_time_sec   Duration of current AOF rewrite in sec.

The 'current' fields are set to -1 if a BGSAVE / AOF rewrite is not in
progress. The 'last' fileds are set to -1 if no previous BGSAVE / AOF
rewrites were performed.

Additionally a few fields in the persistence section were renamed for
consistency:

 changes_since_last_save -> rdb_changes_since_last_save
 bgsave_in_progress -> rdb_bgsave_in_progress
 last_save_time -> rdb_last_save_time
 last_bgsave_status -> rdb_last_bgsave_status
 bgrewriteaof_in_progress -> aof_rewrite_in_progress
 bgrewriteaof_scheduled -> aof_rewrite_scheduled

After the renaming, fields in the persistence section start with rdb_ or
aof_ prefix depending on the persistence method they describe.
The field 'loading' and related fields are not prefixed because they are
unique for both the persistence methods.
 2012-05-25 12:20:35 +02:00
antirez 8bbc0768b8 BITOP command 10x speed improvement. 
This commit adds a fast-path to the BITOP that can be used for all the
bytes from 0 to the minimal length of the string, and if there are
at max 16 input keys.

Often the intersected bitmaps are roughly the same size, so this
optimization can provide a 10x speed boost to most real world usages
of the command.

Bytes are processed four full words at a time, in loops specialized
for the specific BITOP sub-command, without the need to check for
length issues with the inputs (since we run this algorithm only as far
as there is data from all the keys at the same time).

The remaining part of the string is intersected in the usual way using
the slow but generic algorith.

It is possible to do better than this with inputs that are not roughly
the same size, sorting the input keys by length, by initializing the
result string in a smarter way, and noticing that the final part of the
output string composed of only data from the longest string does not
need any proecessing since AND, OR and XOR against an empty string does
not alter the output (zero in the first case, and the original string in
the other two cases).

More implementations will be implemented later likely, but this should
be enough to release Redis 2.6-RC4 with bitops merged in.

Note: this commit also adds better testing for BITOP NOT command, that
is currently the faster and hard to optimize further since it just
flips the bits of a single input string.
 2012-05-24 15:25:13 +02:00
antirez f0d962c0ec BITOP: handle integer encoded objects correctly. 
A bug in the implementation caused BITOP to crash the server if at least
one one of the source objects was integer encoded.

The new implementation takes an additional array of Redis objects
pointers and calls getDecodedObject() to get a reference to a string
encoded object, and then uses decrRefCount() to release the object.

Tests modified to cover the regression and improve coverage.
 2012-05-24 15:25:10 +02:00
antirez c19bfe9fcf BITCOUNT performance improved. 
At Redis's default optimization level the command is now much faster,
always using a constant-time bit manipualtion technique to count bits
instead of GCC builtin popcount, and unrolling the loop.

The current implementation performance is 1.5GB/s in a MBA 11" (1.8 Ghz
i7) compiled with both GCC and clang.

The algorithm used is described here:

http://graphics.stanford.edu/~seander/bithacks.html
 2012-05-24 15:25:06 +02:00
antirez e53a1fdd60 bitop.c renamed bitops.c 
bitop.c contains the "Bit related string operations" so it seems more
logical to call it bitops instead of bitop.
This also makes it matching the name of the test (unit/bitops.tcl).
 2012-05-24 15:24:57 +02:00
antirez 951213e2d2 Bit operations tests improved. 
Fuzzing tests of BITCOUNT / BITOP are iterated multiple times.
The new BITCOUNT fuzzing test uses random strings in a wider interval of
lengths including zero-len strings.
 2012-05-24 15:24:35 +02:00
antirez d12a68d415 popcount() optimization for speed. 
We run the array by 32 bit words instead of processing it byte per byte.
If the code is compiled using GCC __builtin_popcount() builtin function
is used instead.
 2012-05-24 15:24:31 +02:00
antirez 7ffa248c7c BITCOUNT refactoring. 
The low level popualtion counting function is now separated from the
BITCOUNT command implementation, so that the low level function can be
further optimized and eventually used in other contexts if needed.
 2012-05-24 15:24:27 +02:00
antirez 84ec6706cf Bit-related string operations moved to bitop.c 
All the general string operations are implemented in t_string.c, however
the bit operations, while targeting the string type, are better served
in a specific file where we have the implementations of the following
four commands and helper functions:

    GETBIT
    SETBIT
    BITOP
    BITCOUNT

In the future this file will probably contain more code related to
making the BITOP and BITCOUNT operations faster.
 2012-05-24 15:24:14 +02:00
antirez 1f40cdd0e5 BITOP and BITCOUNT tests. 
The Redis implementation is tested against Tcl implementations of the
same operation. Both fuzzing and testing of specific aspects of the
commands behavior are performed.
 2012-05-24 15:22:45 +02:00
antirez 4a789decaa New commands: BITOP and BITCOUNT. 
The motivation for this new commands is to be search in the usage of
Redis for real time statistics. See the article "Fast real time metrics
using Redis".

http://blog.getspool.com/2011/11/29/fast-easy-realtime-metrics-using-redis-bitmaps/

In general Redis strings when used as bitmaps using the SETBIT/GETBIT
command provide a very space-efficient and fast way to store statistics.
For instance in a web application with users, every user can be
associated with a key that shows every day in which the user visited the
web service. This information can be really valuable to extract user
behaviour information.

With Redis bitmaps doing this is very simple just saying that a given
day is 0 (the data the service was put online) and all the next days are
1, 2, 3, and so forth. So with SETBIT it is possible to set the bit
corresponding to the current day every time the user visits the site.

It is possible to take the count of the bit sets on the run, this is
extremely easy using a Lua script. However a fast bit count native
operation can be useful, especially if it can operate on ranges, or when
the string is small like in the case of days (even if you consider many
years it is still extremely little data).

For this reason BITOP was introduced. The command counts the number of
bits set to 1 in a string, with optional range:

BITCOUNT key [start end]

The start/end parameters are similar to GETRANGE. If omitted the whole
string is tested.

Population counting is more useful when bit-level operations like AND,
OR and XOR are avaialble. For instance I can test multiple users to see
the number of days three users visited the site at the same time. To do
this we can take the AND of all the bitmaps, and then count the set bits.

For this reason the BITOP command was introduced:

BITOP [AND|OR|XOR|NOT] dest_key src_key1 src_key2 src_key3 ... src_keyN

In the special case of NOT (that inverts the bits) only one source key
can be passed.

The judicious use of BITCOUNT and BITOP combined can lead to interesting
use cases with very space efficient representation of data.

The implementation provided is still not tested and optimized for speed,
next commits will introduce unit tests. Later the implementation will be
profiled to see if it is possible to gain an important amount of speed
without making the code much more complex.
 2012-05-24 15:22:42 +02:00
antirez bec74b1b9c Add aof_rewrite_buffer_length INFO field. 
The INFO output, persistence section, already contained the field
describing the size of the current AOF buffer to flush on disk. However
the other AOF buffer, used to accumulate changes during an AOF rewrite,
was not mentioned in the INFO output.

This commit introduces a new field called aof_rewrite_buffer_length with
the length of the rewrite buffer.
 2012-05-24 15:22:39 +02:00
antirez edb952532d Allow an AOF rewrite buffer > 2GB (Fix for issue #504). 
During the AOF rewrite process, the parent process needs to accumulate
the new writes in an in-memory buffer: when the child will terminate the
AOF rewriting process this buffer (that ist the difference between the
dataset when the rewrite was started, and the current dataset) is
flushed to the new AOF file.

We used to implement this buffer using an sds.c string, but sds.c has a
2GB limit. Sometimes the dataset can be big enough, the amount of writes
so high, and the rewrite process slow enough that we overflow the 2GB
limit, causing a crash, documented on github by issue #504.

In order to prevent this from happening, this commit introduces a new
system to accumulate writes, implemented by a linked list of blocks of
10 MB each, so that we also avoid paying the reallocation cost.

Note that theoretically modern operating systems may implement realloc()
simply as a remaping of the old pages, thus with very good performances,
see for instance the mremap() syscall on Linux. However this is not
always true, and jemalloc by default avoids doing this because there are
issues with the current implementation of mremap().

For this reason we are using a linked list of blocks instead of a single
block that gets reallocated again and again.

The changes in this commit lacks testing, that will be performed before
merging into the unstable branch. This fix will not enter 2.4 because it
is too invasive. However 2.4 will log a warning when the AOF rewrite
buffer is near to the 2GB limit.
 2012-05-24 15:22:35 +02:00
antirez 8152d0c046 Dead code removed from replication.c. 
The user @jokea noticed that the following line of code into
replication.c made little sense:

    addReplySds(slave,sdsempty());

Investigating a bit I found that this was introduced by commit 6208b3a7
three years ago in the early stages of Redis. The code apparently is not
useful at all, so I'm removing it.

This change will not be backported into 2.4 so that in the rare case
this should introduce a bug, we'll have a chance to detect it into the
development branch. However following the code path it seems like the
code is not useful at all, so the risk is truly small.
 2012-05-24 15:22:31 +02:00
antirez 9b6035bc7f TODO file removed. 
The list of things to do is since long time in two places:

1) Github issues.
2) I've a private TOOD list of random ideas, what makes sense is later
moved to github issues. So github is anyway the true source of things to
do.
 2012-05-23 11:52:47 +02:00
antirez 27fc5bf582 Use comments to split aof.c into sections. 
This makes the code more readable, it is still not the case to split the
file itself into three different files, but the logical separation
improves the readability especially since new commits are going to
introduce an additional section.
 2012-05-23 11:52:40 +02:00
antirez dfa90b5969 Redis test: include bug report on crash. 
Due to a change in the format of the bug report in case of crash of
failed assertion the test suite was no longer able to properly log it.
Instead just a protocol error was logged by the Redis TCL client that
provided no clue about the actual problem.

This commit resolves the issue by logging everything from the first line
of the log including the string REDIS BUG REPORT, till the end of the
file.
 2012-05-23 11:52:35 +02:00
jokea a5a037bf81 Set fd to writable when poll(2) detects POLLERR or POLLHUP event. 2012-05-23 11:33:25 +02:00
antirez 4dada1b5bc Fixed issue #516 (ZINTERSTORE mixing sets and zsets). 
Weeks ago trying to fix an harmless GCC warning I introduced a bug in
the ziplist-encoded implementations of sorted sets.

The bug completely broke zuiNext() iterator, that is used in the
ZINTERSTORE and ZUNIONSTORE implementation, so those two commands are no
longer reliable starting from Redis version 2.4.12 and latest 2.6.0-RC
releases.

This commit fixes the problem and adds a regression test.
 2012-05-23 11:12:38 +02:00
antirez afc6bd1b48 Deleted jemalloc.orig from /deps. 
In the commit upgrading jemalloc to version 3.0.0 I added the old
version of Jemalloc in the 'jemalloc.orig' directory for an error.
This commit removes the not useful version of jemalloc.
 2012-05-16 12:22:29 +02:00
antirez 4934f93dfb Jemalloc updated to 3.0.0. 
Full changelog here:

http://www.canonware.com/cgi-bin/gitweb.cgi?p=jemalloc.git;a=blob_plain;f=ChangeLog;hb=master

Notable improvements from the point of view of Redis:

1) Bugfixing.
2) Support for Valgrind.
3) Support for OSX Lion, FreeBSD.
 2012-05-16 11:20:44 +02:00
Salvatore Sanfilippo 14beba7868 Merge pull request #496 from pietern/2.6-makeinstall 
Fix `install` target on OSX (see #495)
 2012-05-15 02:18:03 -07:00
Salvatore Sanfilippo 404223547c Merge pull request #510 from pietern/2.6-eventport 
Support for Illumos event ports
 2012-05-15 02:16:55 -07:00
Pieter Noordhuis d318803f94 Whitespace 2012-05-14 11:06:34 -07:00
Dave Pacheco 72f30bcd30 use port_getn instead of port_get 2012-05-14 11:01:35 -07:00
Dave Pacheco 2aa1efb8a5 first cut at event port support 2012-05-14 11:01:33 -07:00
antirez e67d014d9a Added time.h include in redis-cli. 
redis-cli.c uses the time() function to seed the PRNG, but time.h was
not included. This was not noticed since sys/time.h is included and was
enough in most systems (but not correct). With Ubuntu 12.04 GCC
generates a warning that made us aware of the issue.
 2012-05-14 17:43:31 +02:00
antirez eee680541b activeExpireCycle(): better precision in max time used. 
activeExpireCycle() can consume no more than a few milliseconds per
iteration. This commit improves the precision of the check for the time
elapsed in two ways:

1) We check every 16 iterations instead of the main loop instead of 256.
2) We reset iterations at the start of the function and not every time
   we switch to the next database, so the check is correctly performed
   every 16 iterations.
 2012-05-14 17:43:26 +02:00
antirez a8a981a834 Impovements for: Redis timer, hashes rehashing, keys collection. 
A previous commit introduced REDIS_HZ define that changes the frequency
of calls to the serverCron() Redis function. This commit improves
different related things:

1) Software watchdog: now the minimal period can be set according to
REDIS_HZ. The minimal period is two times the timer period, that is:

    (1000/REDIS_HZ)*2 milliseconds

2) The incremental rehashing is now performed in the expires dictionary
as well.

3) The activeExpireCycle() function was improved in different ways:

- Now it checks if it already used too much time using microseconds
  instead of milliseconds for better precision.
- The time limit is now calculated correctly, in the previous version
  the division was performed before of the multiplication resulting in
  a timelimit of 0 if HZ was big enough.
- Databases with less than 1% of buckets fill in the hash table are
  skipped, because getting random keys is too expensive in this
  condition.

4) tryResizeHashTables() is now called at every timer call, we need to
   match the number of calls we do to the expired keys colleciton cycle.

5) REDIS_HZ was raised to 100.
 2012-05-14 17:43:23 +02:00
antirez f7f2b2610e Redis timer interrupt frequency configurable as REDIS_HZ. 
Redis uses a function called serverCron() that is very similar to the
timer interrupt of an operating system. This function is used to handle
a number of asynchronous things, like active expired keys collection,
clients timeouts, update of statistics, things related to the cluster
and replication, triggering of BGSAVE and AOF rewrite process, and so
forth.

In the past the timer was called 1 time per second. At some point it was
raised to 10 times per second, but it still was fixed and could not be
changed even at compile time, because different functions called from
serverCron() assumed a given fixed frequency.

This commmit makes the frequency configurable, so that it is simpler to
pick a good tradeoff between overhead of this function (that is usually
very small) and the responsiveness of Redis during a few critical
circumstances where a lot of work is done inside the timer.

An example of such a critical condition is mass-expire of a lot of keys
in the same second. Up to a given percentage of CPU time is used to
perform expired keys collection per expire cylce. Now changing the
REDIS_HZ macro it is possible to do less work but more times per second
in order to block the server for less time.

If this patch will work well in our tests it will enter Redis 2.6-final.
 2012-05-14 17:43:07 +02:00
antirez f078d5628d Comment improved so that the code goal is more clear. Thx to @agladysh. 2012-05-12 09:33:29 +02:00
antirez 3a401464e9 More incremental active expired keys collection process. 
If a large amonut of keys are all expiring about at the same time, the
"active" expired keys collection cycle used to block as far as the
percentage of already expired keys was >= 25% of the total population of
keys with an expire set.

This could block the server even for many seconds in order to reclaim
memory ASAP. The new algorithm uses at max a small amount of
milliseconds per cycle, even if this means reclaiming the memory less
promptly it also means a more responsive server.
 2012-05-12 09:33:24 +02:00
antirez 064223107e If the computer running the Redis test is slow, we revert to --clients 1 to avoid false positives. 2012-05-12 09:33:19 +02:00
antirez 25496f4700 redis-cli pipe mode: handle EINTR properly as well so that SIGSTOP/SIGCONT are handled correctly. 2012-05-12 09:33:15 +02:00
antirez 346825c7ed redis-cli pipe mode: handle EAGAIN while writing to socket. 2012-05-12 09:33:11 +02:00
antirez dd4e8203b2 redis-cli --pipe for mass import. 2012-05-12 09:33:06 +02:00
antirez 91d18504c2 Fix PREFIX typo in Makefile. 2012-05-09 20:45:13 +02:00
antirez f580a3e3a0 Allow PREFIX to be overrided in Makefile. 2012-05-09 10:34:58 +02:00
antirez 8afa5b7072 Fixed typo in RC3 release notes. 2012-05-06 10:22:07 +02:00
antirez af2455be29 More complete release notes for 2.5.9 2012-05-06 10:15:40 +02:00
antirez 184b8e78c6 Redis 2.5.9 (2.6 RC3). 2012-05-06 10:11:54 +02:00
Salvatore Sanfilippo d23d73c746 Merge pull request #500 from pietern/2.6-zipfix 
Compare integers in ziplist regardless of encoding
 2012-05-06 01:03:15 -07:00
Pieter Noordhuis 0ef889274f Compare integers in ziplist regardless of encoding 
Because of the introduction of new integer encoding types for ziplists
in the 2.6 tree, the same integer value may have a different encoding in
different versions of the ziplist implementation. This means that the
encoding can NOT be used as a fast path in comparing integers.
 2012-05-04 17:26:24 -07:00
Pieter Noordhuis 1b1cf8e7bf Fix install target on OSX (see #495) 2012-05-04 11:54:25 -07:00
quiver 7f10703b33 fix several bugs of init.d scripts 
- PIDFILE environ variable was not properly retrieved
- chkconfig command failed
 2012-05-04 13:42:03 +02:00
antirez 0cf10e8e86 syncio.c read / write functions reworked for correctness and performance. 
The new implementation start reading / writing before blocking with
aeWait(), likely the descriptor can accept writes or has buffered data
inside and we can go faster, otherwise we get an error and wait.

This change has effects on speed but also on correctness: on socket
errors when we perform non blocking connect(2) write is performed ASAP
and the error is returned ASAP before waiting.

So the practical effect is that now a Redis slave is more available if it
can not connect to the master, previously the slave continued to block on
syncWrite() trying to send SYNC, and serving commands very slowly.
 2012-05-02 22:45:12 +02:00
antirez 9b43b1ef4d Remove useless trailing space in SYNC command sent to master. 2012-05-02 21:48:08 +02:00
antirez 0b08d64882 Use specific error if master is down and slave-serve-stale-data is set to no. 
We used to reply -ERR ... message ..., now the reply is
instead -MASTERDOWN ... message ... so that it can be distinguished
easily by the other error conditions.
 2012-05-02 17:14:45 +02:00
antirez 96d9c6cb8b Add a note abotu read-only slaves in 2.6 release notes. 2012-05-02 16:35:53 +02:00
antirez 0f07781538 Redis 2.5.8 (2.6.0 RC2). 2012-05-02 12:17:21 +02:00
antirez 1858da2faa Test "Turning off AOF kills the background writing child if any" is now more reliable. 2012-05-02 11:40:55 +02:00
Salvatore Sanfilippo 83b1092cf2 Merge pull request #488 from pietern/2.6-safekeys 
Use safe dictionary iterator from KEYS (fixes #487)
 2012-05-01 01:50:25 -07:00
Pieter Noordhuis 9311d2b527 Use safe dictionary iterator from KEYS 
Every matched key in a KEYS call is checked for expiration. When the key
is set to expire, the call to `getExpire` will assert that the key also
exists in the main dictionary. This in turn causes a rehashing step to
be executed. Rehashing a dictionary when there is an iterator active may
result in the iterator emitting duplicate entries, or not emitting some
entries at all. By using a safe iterator, the rehash step is omitted.
 2012-04-30 10:16:20 -07:00
Harmen 8520066d7b Show problem with 'keys' command with specific command sequence. 2012-04-30 09:51:23 -07:00
antirez 8f984bef29 Properly wait the slave to sync with master in BRPOPLPUSH test. 2012-04-30 11:32:02 +02:00
antirez dd418873db A more lightweight implementation of issue 141 regression test. 2012-04-29 17:16:47 +02:00
antirez b330de57ff Added "read-only slaves" in new features section of 2.6 release notes. 2012-04-27 23:06:02 +02:00
antirez 18759c927a yet another typo fixed in release notes. 2012-04-27 17:04:03 +02:00
antirez 8b97442c20 Fixed release notes typo 2012-04-27 17:01:44 +02:00
antirez 7c5d96d98e Redis 2.5.7 (2.6 RC1) 2012-04-27 16:40:07 +02:00
antirez 603adb2b29 memtest.c fixed to actually use v1 and v2 in memtest_fill_value(). 2012-04-27 16:28:31 +02:00
antirez 748f206e3d Release notes updated with the new 2.6 features. 2012-04-27 16:07:42 +02:00
antirez 2c0aae760f redis-cli commands description in help.h updated. 2012-04-27 15:57:17 +02:00
antirez b1aa7183d7 Update makefile dependencies. 2012-04-27 15:56:16 +02:00
antirez 77a75fdef5 Set LUA_MASKCOUNT hook more selectively. Fixes issue #480. 
An user reported a crash with Redis scripting (see issue #480 on
github), inspection of the kindly provided strack trace showed that
server.lua_caller was probably set to NULL. The stack trace also slowed
that the call to the hook was originating from a point where we just
used to set/get a few global variables in the Lua state.

What was happening is that we did not set the timeout hook selectively
only when the user script was called. Now we set it more selectively,
specifically only in the context of the lua_pcall() call, and make sure
to remove the hook when the call returns. Otherwise the hook can get
called in random contexts every time we do something with the Lua
state.
 2012-04-27 11:47:30 +02:00
antirez e69e76d442 Re-introduce -g -rdynamic -ggdb when linking, fixing strack traces. 
A previous commit removed -g -rdynamic -ggdb as LDFLAGS, not allowing
Redis to produce a stack trace wth symbol names on crash.
This commit fixes the issue.
 2012-04-27 11:47:27 +02:00
antirez 9db4cea235 Produce the stack trace in an async safe way. 2012-04-27 11:47:22 +02:00
antirez a28ab2a92b Don't use an alternative stack for SIGSEGV & co. 
This commit reverts most of c575766202, in
order to use back main stack for signal handling.

The main reason is that otherwise it is completely pointless that we do
a lot of efforts to print the stack trace on crash, and the content of
the stack and registers as well. Using an alternate stack broken this
feature completely.
 2012-04-27 11:47:18 +02:00
antirez b1ee7da75a Redis test: More reliable BRPOPLPUSH replication test. 
Now it uses the new wait_for_condition testing primitive.
Also wait_for_condition implementation was fixed in this commit to properly
escape the expr command and its argument.
 2012-04-27 11:47:15 +02:00
antirez d7bad544dc Redis test: scripting EVALSHA replication test more reliable. 
A new primitive wait_for_condition was introduced in the scripting
engine that makes waiting for events simpler, so that it is simpler to
write tests that are more resistant to timing issues.
 2012-04-27 11:47:12 +02:00
David Tran 8111a803cb Spelling: s/synchrnonization/synchronization 2012-04-27 11:47:08 +02:00
antirez c16bf717de Explicitly use bash for install_server.sh. Fixes issue #397 2012-04-24 19:35:40 +02:00
antirez 12a042f863 redis-check-dump now is RDB version 6 ready. 2012-04-24 19:35:36 +02:00
antirez 717145c1a3 Spurious debugging printf removed. 2012-04-24 19:35:33 +02:00
antirez dcd4efe9ef Added two new encodings to ziplist.c 
1) One integer "immediate" encoding that can encode from 0 to 12 in the
encoding byte itself.
2) One 8 bit signed integer encoding that can encode 8 bit signed small
integers in a single byte.

The idea is to exploit all the not used bits we have around in a
backward compatible way.
 2012-04-24 19:35:29 +02:00
antirez 62bfa66284 rdbLoad() should check REDIS_RDB_VERSION instead of hardcoded number. 2012-04-24 19:35:25 +02:00
antirez dd51571cc3 ziplist.c: added comments about the new 24 bit encoding. 2012-04-24 19:35:22 +02:00
Grisha Trubetskoy ad91404a32 Add a 24bit integer to ziplists to save one byte for ints that can 
fit in 24 bits (thanks to antirez for catching and solving the two's compliment
bug).

Increment REDIS_RDB_VERSION to 6
 2012-04-24 19:35:18 +02:00
antirez 9de5d4600a A few compiler warnings suppressed. 2012-04-24 19:35:03 +02:00
antirez 38b60dea54 Fix and refactoring of code used to get registers on crash. 
This fixes compilation on FreeBSD (and possibly other systems) by
not using ucontext_t at all if HAVE_BACKTRACE is not defined.
Also the ifdefs to get the registers are modified to explicitly test for the
operating system in the first level, and the arch in the second level
of nesting.
 2012-04-24 19:34:15 +02:00
antirez 69b30cfcb6 Ziplist encoding now tested with negative integers as well. 2012-04-24 19:34:08 +02:00
antirez 0a8a1e78dc New time limit for protocol desync test set to 30 seconds to reduce false positives. 2012-04-24 19:34:03 +02:00
antirez 537dafab84 Remove loadfile() access from the scripting engine. 2012-04-24 19:33:58 +02:00
Michael Schlenker 7d6bf7956e Replace unnecessary calls to echo and cat 
Tcl's exec can send data to stdout itself, no need to call cat/echo for
that usually.
 2012-04-24 19:33:54 +02:00
antirez e337b26050 Even inside #if 0 comments are comments. 2012-04-21 21:49:34 +02:00
antirez 590d55a206 Limit memory used by big SLOWLOG entries. 
Two limits are added:

1) Up to SLOWLOG_ENTRY_MAX_ARGV arguments are logged.
2) Up to SLOWLOG_ENTRY_MAX_STRING bytes per argument are logged.
3) slowlog-max-len is set to 128 by default (was 1024).

The number of remaining arguments / bytes is logged in the entry
so that the user can understand better the nature of the logged command.
 2012-04-21 21:00:33 +02:00
antirez c11a01a030 redis.conf AOF section comments improved. 2012-04-21 12:17:33 +02:00
antirez 30e89410d4 README now makes clear that our support for solaris derived systems is "best effort". 2012-04-21 12:17:29 +02:00
antirez abfd08f5ad New tests related to scripts max execution time. 2012-04-19 23:49:46 +02:00
antirez ca577d162a SHUTDOWN NOSAVE now can stop a non returning script. Issue #466. 2012-04-19 23:36:04 +02:00
antirez d54943b76d Currenly not used code in dict.c commented out. 2012-04-18 23:56:15 +02:00
antirez c3312760fe Tests for scripting PRNG. 2012-04-18 23:50:27 +02:00
antirez 1d82bbd432 cr16.c removed from 2.6 branch, was not used. 2012-04-18 23:41:00 +02:00
antirez 0122cc4f42 redis-cli --bigkeys output modified to be simpler to read.. 2012-04-18 21:23:42 +02:00
antirez e10768518c redis-cli --bigkeys 2012-04-18 21:23:39 +02:00
antirez 5c45ae1f7b Test SDIFF with first set empty. 2012-04-18 21:23:35 +02:00
antirez 7a2065ef33 Test SINTER against same integer elements, but different set encoding. 2012-04-18 21:23:31 +02:00
antirez eb624e3416 Test SINTER with non existing key. 2012-04-18 21:23:15 +02:00
antirez ff5e31f74b Added an SMOVE test where src and dest key are the same. 2012-04-18 21:23:07 +02:00
antirez 24982f2bbc New hash fuzzing test. 2012-04-18 21:23:04 +02:00
antirez 60ef787efa Document mostly dead code in RPOPLPUSH implementation. 2012-04-18 21:23:00 +02:00
antirez bec200ec39 Explicit RPOP/LPOP tests. 2012-04-18 21:22:56 +02:00
antirez a00fcaa671 Test LINSERT syntax error. 2012-04-18 21:22:52 +02:00
antirez 8d12645569 Test LINDEX out of range index. 2012-04-18 21:22:48 +02:00
Salvatore Sanfilippo d00f51405b Merge pull request #455 from pietern/2.6-mkopts 
Persist Makefile flags (2.6)
 2012-04-18 07:26:36 -07:00
antirez ae55245d1c Marginally cleaner lookupKeyByPattern() implementation. 
just fieldobj itself as sentinel of the fact a field object is used or
not, instead of using the filed length, that may be confusing both for
people and for the compiler emitting a warning.
 2012-04-18 11:41:06 +02:00
antirez 212bb9ca2e More robust maxclients test. 2012-04-18 11:41:03 +02:00
antirez 727d6dd52a Two small fixes to maxclients handling. 
1) Don't accept maxclients set to < 0
2) Allow maxclients < 1024, it is useful for testing.
 2012-04-18 11:40:59 +02:00
antirez a1090c1193 Added test for SORT corner case: pattern ending with just "->". 2012-04-17 18:26:56 +02:00
antirez 68ee18558a lookupKeyByPattern() used by SORT GET/BY rewritten. Fixes issue #460. 
lookupKeyByPattern() was implemented with a trick to speedup the lookup
process allocating two fake Redis obejcts on the stack. However now that
we propagate expires to the slave as DEL operations the lookup of the
key may result into a call to expireIfNeeded() having the stack
allocated object as argument, that may in turn use it to create the
protocol to send to the slave. But since this fake obejcts are
inherently read-only this is a problem.

As a side effect of this fix there are no longer size limits in the
pattern to be used with GET/BY option of SORT.

See https://github.com/antirez/redis/issues/460 for bug details.
 2012-04-17 18:26:52 +02:00
antirez 96aeca4b9d Less false positives in maxclients test, hopefully. 2012-04-17 10:04:59 +02:00
Pieter Noordhuis d0cd262fdf Persist make settings and trigger rebuild if anything changes 2012-04-13 18:05:51 -07:00
Pieter Noordhuis 864229585a Don't set flags recursively 2012-04-13 18:05:51 -07:00
Pieter Noordhuis 3126e08757 Remove unused LIBS variable 2012-04-13 18:05:51 -07:00
Pieter Noordhuis 61e8825e89 First set defaults, then do composition 2012-04-13 18:05:51 -07:00
Pieter Noordhuis 95bc195158 Question mark assignment is not needed 2012-04-13 18:05:51 -07:00
Pieter Noordhuis cb481f432a Ignore gcov/lcov artifacts 2012-04-13 18:05:51 -07:00
Pieter Noordhuis 0b27a55f30 The lcov target shouldn't clean 
This is not needed because every change in compiler/linker flags
triggers a cleanup.
 2012-04-13 18:05:51 -07:00
Pieter Noordhuis 6cedb4d489 Rename ADD_*FLAGS -> REDIS_*FLAGS, REDIS_*FLAGS -> FINAL_*FLAGS 
This reflects that REDIS_*FLAGS will only be used for compilation of
Redis and not for its dependencies. Similarly, that FINAL_*FLAGS are
composed of other variables and holds the options that are finally
passed to the compiler and linker.
 2012-04-13 18:05:51 -07:00
antirez 59333ffd37 New test for scripting engine: DECR_IF_GT. 2012-04-13 16:23:46 +02:00
antirez 9a2dd1eff9 EVAL errors are more clear now. 2012-04-13 16:23:42 +02:00
antirez c9853f537b Use Lua tostring() before concatenation. 2012-04-13 16:23:38 +02:00
antirez d63a1716eb Tests modified to match the new global protection implementation. 2012-04-13 16:23:34 +02:00
antirez d2906893e8 mt.declared is no longer needed. 
Lua global protection can now be simpified becuase we no longer have the
global() function. It's useless to occupy memory with this table, it is
also not faster because the metamethods we use are only called when a
global object does not exist or we are trying to create it from a
script.
 2012-04-13 16:23:29 +02:00
antirez 97cab30993 Stop access to global vars. Not configurable. 
After considering the interaction between ability to delcare globals in
scripts using the 'global' function, and the complexities related to
hanlding replication and AOF in a sane way with globals AND ability to
turn protection On and Off, we reconsidered the design. The new design
makes clear that there is only one good way to write Redis scripts, that
is not using globals. In the rare cases state must be retained across
calls a Redis key can be used.
 2012-04-13 16:23:21 +02:00
antirez 6255a5ae66 Globals protection global() function modified for speed and correctness. 2012-04-13 16:22:47 +02:00
antirez e387dc52a0 Tests for lua globals protection. 2012-04-13 16:22:43 +02:00
antirez 3e6a4463e0 Scripting: globals protection can now be switched on/off. 2012-04-13 16:22:35 +02:00
antirez 430602b26c Protect globals access in Lua scripting. 2012-04-13 16:21:46 +02:00
antirez 236adc2809 A few var names fixed in Makefile. 
I modified it for error in a previous commit doing search & replace.
 2012-04-13 16:15:39 +02:00
antirez d298825803 Print arch bits with redis-server -v 2012-04-13 16:12:32 +02:00
antirez 206568257a memtest.c: integer overflow fixed. 2012-04-13 16:12:28 +02:00
antirez 50fb330399 Make gcov fixed. 2012-04-13 16:12:24 +02:00
antirez 69ac4d063d Makefile now introduces Redis-specific CFLAGS / LDFLAGS. Gcov target fixed. Added comments to describe how it works. 2012-04-13 16:12:04 +02:00
Erik Dubbelboer b9cd703b5b added explanation for the magic 511 backlog number 2012-04-11 18:32:10 +02:00
antirez 79e3df9d72 Makefile.dep updated. 2012-04-11 18:30:23 +02:00
antirez d5ec389585 make dep: redirect output to Makefile.dep. 2012-04-11 18:29:31 +02:00
antirez 336ba6a152 Make inline functions rioRead/Write/Tell static. This fixes issue #447. 2012-04-11 12:03:25 +02:00
antirez ef278d110f Macros ULONG_ONEZERO / ULONG_ZEROONE were inverted in #ifdef to test 32/64 bit arch. 2012-04-11 12:03:21 +02:00
Pieter Noordhuis de07849e0d Clean up Makefiles 
Remove unused variables. Instead of overriding non-standard variables
such as ARCH and PROF, use standard variables CFLAGS and LDFLAGS to
override Makefile settings. Move dependencies generated by `make dep` to
a separate file.
 2012-04-11 12:01:29 +02:00
Pieter Noordhuis e74bec56fa Everything x86 is little endian 2012-04-11 11:59:53 +02:00
antirez 3ba5eab774 Minor MIGRATE implementation simplification about ttl handling. 2012-04-10 20:04:21 +02:00
antirez b9aa332843 Check write(2) return value to avoid warnings, because in this context failing write is not critical. 2012-04-10 20:02:33 +02:00
antirez 3f64694e71 Version 2.5.6. 2012-04-10 16:34:51 +02:00
antirez fdf8bd4025 Test for maxclients. 2012-04-10 16:28:21 +02:00
antirez e95740392b dump/restore fixed to use the new crc64 API. 2012-04-10 16:27:36 +02:00
antirez 39d1e350d9 It is now possible to enable/disable RDB checksum computation from redis.conf or via CONFIG SET/GET. Also CONFIG SET support added for rdbcompression as well. 2012-04-10 16:27:03 +02:00
antirez 7f4f86f427 RDB files now embed a crc64 checksum. Version of RDB bumped to 5. 2012-04-10 16:26:58 +02:00
antirez 1bcb45d118 Fixed compilation of new rio.c changes (typos and so forth.) 2012-04-10 16:26:52 +02:00
antirez b4b923b04b Add checksum computation to rio.c 2012-04-10 16:26:45 +02:00
antirez bb99f42596 crc64.c modified for incremental computation. 2012-04-10 16:26:41 +02:00
antirez 9ba4d5a3bb rio.c file somewhat documented so that the casual reader can understand what's going on without reading the code. 2012-04-10 16:26:37 +02:00
antirez 08211b25d3 Added new test to check that "CONFIG appendonly no" actually kills the background AOF operation in progress if any. 2012-04-08 10:43:33 +02:00
antirez 55951f9005 For coverage testing use exit() instead of _exit() when termiating saving children. 2012-04-08 10:43:29 +02:00
antirez 2cf3f071a5 Tests for MONITOR. 2012-04-07 11:27:04 +02:00
antirez b162e6f133 New client info field added to CLIENT LIST output: multi, containing the length of the current pipeline. Test modified accordingly. 2012-04-07 11:27:00 +02:00
antirez 4f0bd607d9 Never used function stringObjectEqualsMs() removed. 2012-04-07 11:26:56 +02:00
antirez 7dc1d2ba17 Removed dead code: function rdbSaveTime() is no longer used since RDB now saves expires in milliseconds. 2012-04-07 11:26:52 +02:00
antirez eb6bc2e047 Two new tests for BGREWRTIEAOF. 
Check for scheduled rewrite if a BGSAVAE is in progress.
Check for error if a rewrite is already in progress.
 2012-04-07 11:26:47 +02:00
antirez 3984108474 redis.tcl: no longer leave unread replies if an error happens during a MULTI/EXEC block. 2012-04-07 11:26:42 +02:00
antirez 8d3f5ce9db New INFO field in persistence section: bgrewriteaof_scheduled. 2012-04-06 21:13:33 +02:00
antirez 8382453ecf version bumped to 2.5.5 2012-04-06 12:27:31 +02:00
jokea a3ec16f0c2 implement aeWait using poll(2). Fixes issue #267. 2012-04-06 11:49:16 +02:00
antirez 3aad0de2e9 expireGenericCommand(): better variable names and a top-comment that describes the function's behavior. 2012-04-06 00:40:36 +02:00
Premysl Hruby 81a28fe131 for (p)expireat use absolute time, without double recomputation 2012-04-06 00:40:31 +02:00
Premysl Hruby b518a83525 fix mstime() ommited while comparing if key is already expired 2012-04-06 00:40:26 +02:00
Premysl Hruby 9f899440d2 add support for generation of lcov coverage reports 2012-04-06 00:40:22 +02:00
Premysl Hruby 937abebbe1 future-proof version comparison 2012-04-05 11:04:28 +02:00
antirez c537980d98 On slow computers, 10 seconds are not enough for this heavy replication test. 2012-04-05 11:04:23 +02:00
Premysl Hruby c9adc81e2e allocate alternate signal stack, change of sigaction flags for sigterm 2012-04-05 11:04:19 +02:00
Premysl Hruby b811b334ae in kill_server send the signal once, then wait for up to 5sec before sending lethal SIGKILL 2012-04-05 11:04:14 +02:00
Premysl Hruby 4d57e44839 new option for choosing number of test clients to run 2012-04-05 11:04:09 +02:00
Premysl Hruby 6c52d5ce27 remove mentions of VM in comments 2012-04-05 11:04:03 +02:00
antirez 09eb448746 Structure field controlling the INFO field master_link_down_since_seconds initialized correctly to avoid strange INFO output at startup when a slave has yet to connect to its master. 2012-04-04 18:33:13 +02:00
antirez fa2a27cfeb New "os" field in INFO output providing information about the operating system. 2012-04-04 18:33:09 +02:00
antirez a5b75e9a86 SLAVEOF is not a write command. 2012-04-04 18:33:04 +02:00
antirez df35d87336 Print milliseconds of the current second in log lines timestamps. Sometimes precise timing is very important for debugging. 2012-04-04 18:32:59 +02:00
antirez 3c413d59c8 redis-cli help.h updated. Script to generate it updated as well. 2012-04-03 15:29:47 +02:00
antirez e3fd3ccd72 More MIGRATE tests. 2012-04-03 15:14:03 +02:00
antirez fb8409a5c8 Another fix for MIGRATE. 2012-04-03 15:13:50 +02:00
antirez bde80cb28b Two fixed for MIGRATE: fix TTL propagation and fix transferring of data bigger than 64k. 2012-04-03 14:57:28 +02:00
antirez fbce475270 When the user-provided 'maxclients' value is too big for the max number of files we can open, at least try to search the max the OS is allowing (in steps of 256 filedes). 2012-04-03 14:55:18 +02:00
antirez 31e2156c4b redis-trib.rb removed from 2.6 branch. 2012-04-03 10:31:51 +02:00
antirez b739506936 MIGRATE test modified because the implementation changed. 2012-04-02 16:41:11 +02:00
antirez 37cc07dd41 MIGRATE now let the client distinguish I/O errors and timeouts from other erros. 2012-04-02 16:41:02 +02:00
antirez 285df7b0e0 Version bumped 2.4.5 2012-04-02 14:27:04 +02:00
antirez 8cf8974a03 DUMP, RESTORE, MIGRATE tests. 2012-04-02 14:22:07 +02:00
antirez bff31e129d New verions of DUMP, RESTORE, MIGRATE back ported from unstable to 2.6 2012-04-02 14:21:59 +02:00
antirez 179ee2db78 CRC64 implementation added to Redis code base. 2012-04-02 13:53:43 +02:00
antirez 11dae1711f Write RDB magic using a REDIS_RDB_VERSION define that is defined inside rdb.h 2012-04-02 13:52:42 +02:00
ThePicard f08aa2bf48 Fixed a typo in install_server.sh 2012-04-01 10:15:38 +02:00
antirez fa34ba3908 syncio.c calls in replication.c fixed for the new millisecond timeout API. 2012-03-31 11:26:46 +02:00
antirez 4ccf671cc6 Better syncio.c with millisecond resolution. 2012-03-31 11:26:42 +02:00
antirez 88bd32f1b7 Purely aesthetic code change. 2012-03-30 10:40:04 +02:00
Joseph Jang ae15f75089 Fixed a memory leak with replication 
occurs when two or more dbs are replicated and at least one of them is >db10
 2012-03-30 10:40:01 +02:00
antirez 0f51e3c564 Regression test for issue 417 (memory leak when replicating to DB with id >= 10) 2012-03-30 10:39:56 +02:00
antirez f1e38b352e Fixed typo in comment: "te" -> "the". 2012-03-29 09:34:07 +02:00
antirez ed4d4f1145 Fix for slaves chains. Force resync of slaves (simply disconnecting them) when SLAVEOF turns a master into a slave. 2012-03-29 09:34:03 +02:00
antirez ee704a0ff1 Test for redis.sha1hex(). 2012-03-29 09:33:57 +02:00
Nathan Fritz 2f2e6ad487 added redis.sha1hex(string) as lua scripting function. 
(The original implementation was modified by @antirez to conform Redis
coding standards.)
 2012-03-29 09:33:52 +02:00
antirez e51f7d2c17 Fixes for redisLogFromHandler(). 2012-03-28 14:57:09 +02:00
antirez be4f8cccaa Log from signal handlers is now safer. 2012-03-28 14:57:04 +02:00
antirez f3e159bc93 Redis test: regexp to check if valgrind reported errors modified. Now we no longer look at the total count because this includes "possibly lost" bytes that are not interesting for Redis (tons of false positives because of how sds.c works). 2012-03-28 13:18:32 +02:00
antirez dd896e1b8d define zlibc_free() in a way that is not shadowed by jemalloc. 2012-03-28 13:18:29 +02:00
antirez 59d884af8c Produce the watchlog warning log in a way that is safer from a signal handler. Fix a memory leak in the backtrace generation function. 2012-03-28 13:18:25 +02:00
antirez e4cd5838fc Mask SIGALRM everything but in the main thread. 
This is required to ensure that the signal will be delivered to the main
thread when the watchdog timer expires.
 2012-03-28 13:18:21 +02:00
antirez 1507be5b01 Correctly set the SIGARLM timer for the software watchdog. 2012-03-28 13:18:17 +02:00
antirez fc030ac7de Redis software watchdog. 2012-03-28 13:18:13 +02:00
huangz1990 085aaef325 fix typo 2012-03-27 23:04:40 +02:00
Premysl Hruby 92dd4e4301 fix time() instead of mstime() in expireIfNeeded 2012-03-27 23:04:34 +02:00
Premysl Hruby 56ff70f8e0 use server.unixtime instead of time(NULL) where possible (cluster.c not checked though) 2012-03-27 23:04:27 +02:00
Premysl Hruby 38d6976c03 declare hashDictType as external too 2012-03-27 23:04:21 +02:00
Premysl Hruby efd412f908 remove disk-store related comments 2012-03-27 23:04:16 +02:00
antirez 9ea95e6c64 SIGSEGV handler refactored so that we can reuse stack trace and current client logging functionalities in other contexts. 2012-03-27 10:33:45 +02:00
antirez d0407c2d85 CONFIG RESETSTAT resets two more fields. 2012-03-25 11:43:42 +02:00
antirez 81f32c7b65 New INFO field aof_delayed_fsync introduced. 
This new field counts all the times Redis is configured with AOF enabled and
fsync policy 'everysec', but the previous fsync performed by the
background thread was not able to complete within two seconds, forcing
Redis to perform a write against the AOF file while the fsync is still
in progress (likely a blocking operation).
 2012-03-25 11:43:38 +02:00
antirez 754643c1c7 Version is now 2.5.3. 2012-03-25 11:18:24 +02:00
antirez 0387e9c199 convert-zipmap-hash-on-load false positive fixed. 
Apparently because the sample RDB file was not copied before every test
Redis had a chance to replace it with a newly written one, so that the
next test could fail.
 2012-03-25 10:57:34 +02:00
antirez 29e1976855 When running the test in valgrind mode, pass the right flags to show memory leaks stack traces but only including the "definitely lost" items. 2012-03-24 12:06:56 +01:00
antirez 188a17ed56 Add used allocator in redis-server -v output. 2012-03-24 11:52:56 +01:00
antirez 9542d9d8d7 convert-zipmap-hash-on-load test enabled 2012-03-24 11:42:39 +01:00
antirez 2d04eef425 Contextualize comment. 2012-03-23 20:20:43 +01:00
antirez 925927fc8f Merge remote-tracking branch 'origin/2.6' into 2.6 2012-03-23 20:20:13 +01:00
antirez 27688de10e RDB load of different encodings test added. 2012-03-23 20:22:22 +01:00
antirez 9b962d10a4 Fixed memory leak in hash loading. 2012-03-23 20:21:19 +01:00
antirez ab0603812d RDB load of different encodings test added. 2012-03-23 15:23:01 +01:00
antirez b64281cc0e Big endian fix. The bug was introduced because of a typo. 2012-03-23 12:50:19 +01:00
antirez 52192552bd Replicate HINCRBYFLOAT as HSET. 2012-03-23 10:24:25 +01:00
antirez b54cdfb2b7 Code style hack. 2012-03-22 18:16:41 +01:00
antirez 7b22c44ccd Result of INCRBYFLOAT and HINCRBYFLOAT should never be in exponential form, and also should never contain trailing zeroes. This is not possible with vanilla printf() format specifiers, so we alter the output. 2012-03-22 18:14:32 +01:00
antirez 9aba884b34 Comments about security of slave-read-only in redis.coinf. 2012-03-22 10:09:58 +01:00
antirez 3f7ad83398 Correctly create shared.oomerr as an sds string. 2012-03-22 10:09:54 +01:00
antirez 38bb45223a DEBUG should not be flagged as w otherwise we can not call DEBUG DIGEST and other commands against read only slaves. 2012-03-22 10:09:49 +01:00
antirez 054061685a Support for read-only slaves. Semantical fixes. 
This commit introduces support for read only slaves via redis.conf and CONFIG GET/SET commands. Also various semantical fixes are implemented here:

1) MULTI/EXEC with only read commands now work where the server is into a state where writes (or commands increasing memory usage) are not allowed. Before this patch everything inside a transaction would fail in this conditions.

2) Scripts just calling read-only commands will work against read only
slaves, when the server is out of memory, or when persistence is into an
error condition. Before the patch EVAL always failed in this condition.
 2012-03-22 10:09:43 +01:00
antirez 7c1cec2525 Redis cluster stuff removed from 2.6 redis.conf file. 2012-03-20 13:07:26 +01:00
antirez 0380c13bbe redis_init_script template updated. 2012-03-20 13:06:50 +01:00
antirez 518e7202b2 Fixed typo in 2.6 release notes. 2012-03-19 22:10:18 +01:00
antirez f597910121 Merge remote-tracking branch 'origin/2.6' into 2.6 2012-03-19 19:30:30 +01:00
antirez 24b0942275 Suppress warnings compiling redis-cli with certain gcc versions. 2012-03-19 19:29:06 +01:00
antirez 6e6bbac7a5 Read-only flag removed from PUBLISH command. 2012-03-19 19:18:51 +01:00
antirez a7ef5ce1b0 Memory addressing test implemented. 2012-03-19 14:02:57 +01:00
antirez ea693f0282 More memory tests implemented. Default number of iterations lowered to a more acceptable value of 50. 2012-03-18 18:03:49 +01:00
antirez 74760d3ccd Fixed typo. 2012-03-18 17:28:03 +01:00
antirez 32f62ed6d0 Number of iteration of --test-memory is now 300 (several minutes per gigabyte). Memtest86 and Memtester links are also displayed while running the test. 2012-03-18 17:24:48 +01:00
antirez d4e6ce3e97 On crash suggest to give --test-memory a try. 2012-03-18 11:43:09 +01:00
antirez d605fdabfa Memory test function now less boring thanks to screen-wide progress bar. 2012-03-18 11:43:02 +01:00
antirez f4df22d1c5 Hem... actual memtest.c file added. 2012-03-18 11:42:56 +01:00
antirez 78d6a02b0c First implementation of --test-memory. Still a work in progress. 2012-03-18 11:42:47 +01:00
antirez f1eaf57200 Fix for issue #391. 
Use a simple protocol between clientsCron() and helper functions to
understand if the client is still valind and clientsCron() should
continue processing or if the client was freed and we should continue
with the next one.
 2012-03-15 20:51:39 +01:00
antirez 749817b7c3 Version bumped to 2.5.2 2012-03-14 15:37:47 +01:00
antirez 9fa9ccb04e Reclaim space from the client querybuf if needed. 2012-03-14 15:36:27 +01:00
antirez 6934832e53 sds.c: sdsAllocSize() function added. 2012-03-14 15:36:24 +01:00
antirez f9322fb8ed sds.c new function sdsRemoveFreeSpace(). 
The new function is used in order to resize the string allocation so
that only the minimal allocation possible is used, removing all the free
space at the end of the string normally used to improve efficiency of
concatenation operations.
 2012-03-14 15:36:20 +01:00
antirez 5e473cd8fd Call all the helper functions needed by clientsCron() as clientsCronSomething() for clarity. 2012-03-14 15:36:16 +01:00
antirez 3e8fcb6d03 CLIENT LIST test modified to reflect the new output. 2012-03-13 18:08:59 +01:00
antirez cfa4b57cb0 Process async client checks like client timeouts and BLPOP timeouts incrementally using a circular list. 2012-03-13 18:08:53 +01:00
antirez 57a5e54ddc Added a qbuf-free field to CLIENT LIST output. 2012-03-13 18:08:50 +01:00
antirez 41e8e5cb8e Client creation time in redisClient structure. New age field in CLIENT LIST output. 2012-03-13 18:08:47 +01:00
antirez 56de4964ee c->bufpos initialization moved for aesthetics. 2012-03-13 18:08:43 +01:00
antirez a74ab6478c RDB hashes loading, fixed another bug in the loading of HT-encoded hashes: when the hash entry is too big for ziplist, add the field, then convert. The code used to break before the new entry was inserted, resulting into missing fields in the loaded Hash object. 2012-03-13 11:01:08 +01:00
antirez ee61a4b99e RDB hashes loading fixed removing the assertion that failed every time an HT-encoded hash was loaded. 2012-03-13 09:49:21 +01:00
quiver dfc2545470 fix typo of redis.conf 2012-03-10 13:23:31 +01:00
antirez b014c1f211 RDB4 support in redis-check-dump. 2012-03-10 12:38:59 +01:00
antirez 37180ed9cd RDB version is no 4, because small hashes are now encoded as ziplists, so older versions of Redis will not understand this format. 2012-03-10 12:35:15 +01:00
antirez 7551f2a0b1 Version is now 2.5.1, first unstable release of Redis 2.6 2012-03-10 12:29:47 +01:00
antirez 00b3336334 Build dependencies updated. 2012-03-10 12:28:14 +01:00
antirez 571e257db1 Redis 2.6 branch obtained from unstable removing all the cluster related code. 2012-03-10 12:26:37 +01:00
84 changed files with 2060 additions and 11856 deletions
Expand all files Collapse all files
.gitignore
-1
View File
@@ -19,7 +19,6 @@ src/transfer.sh
src/configs
redis.ds
src/redis.conf
src/nodes.conf
deps/lua/src/lua
deps/lua/src/luac
deps/lua/src/liblua.a
00-RELEASENOTES
+539 -451
View File
File diff suppressed because it is too large Load Diff
deps/lua/src/ldo.c
Vendored
+1 -1
View File
@@ -495,7 +495,7 @@ static void f_parser (lua_State *L, void *ud) {
struct SParser *p = cast(struct SParser *, ud);
int c = luaZ_lookahead(p->z);
luaC_checkGC(L);
tf = ((c == LUA_SIGNATURE[0]) ? luaU_undump : luaY_parser)(L, p->z,
tf = (luaY_parser)(L, p->z,
&p->buff, p->name);
cl = luaF_newLclosure(L, tf->nups, hvalue(gt(L)));
cl->l.p = tf;
deps/lua/src/lua_cmsgpack.c
Vendored
+42 -29
View File
@@ -66,7 +66,7 @@
/* Reverse memory bytes if arch is little endian. Given the conceptual
* simplicity of the Lua build system we prefer check for endianess at runtime.
* The performance difference should be acceptable. */
static void memrevifle(void *ptr, size_t len) {
void memrevifle(void *ptr, size_t len) {
unsigned char *p = (unsigned char *)ptr,
*e = (unsigned char *)p+len-1,
aux;
@@ -96,7 +96,7 @@ typedef struct mp_buf {
size_t len, free;
} mp_buf;
static void *mp_realloc(lua_State *L, void *target, size_t osize,size_t nsize) {
void *mp_realloc(lua_State *L, void *target, size_t osize,size_t nsize) {
void *(*local_realloc) (void *, void *, size_t osize, size_t nsize) = NULL;
void *ud;
@@ -105,7 +105,7 @@ static void *mp_realloc(lua_State *L, void *target, size_t osize,size_t nsize) {
return local_realloc(ud, target, osize, nsize);
}
static mp_buf *mp_buf_new(lua_State *L) {
mp_buf *mp_buf_new(lua_State *L) {
mp_buf *buf = NULL;
/* Old size = 0; new size = sizeof(*buf) */
@@ -117,7 +117,7 @@ static mp_buf *mp_buf_new(lua_State *L) {
return buf;
}
static void mp_buf_append(mp_buf *buf, const unsigned char *s, size_t len) {
void mp_buf_append(mp_buf *buf, const unsigned char *s, size_t len) {
if (buf->free < len) {
size_t newlen = buf->len+len;
@@ -153,7 +153,7 @@ typedef struct mp_cur {
int err;
} mp_cur;
static void mp_cur_init(mp_cur *cursor, const unsigned char *s, size_t len) {
void mp_cur_init(mp_cur *cursor, const unsigned char *s, size_t len) {
cursor->p = s;
cursor->left = len;
cursor->err = MP_CUR_ERROR_NONE;
@@ -173,13 +173,17 @@ static void mp_cur_init(mp_cur *cursor, const unsigned char *s, size_t len) {
/* ------------------------- Low level MP encoding -------------------------- */
static void mp_encode_bytes(mp_buf *buf, const unsigned char *s, size_t len) {
void mp_encode_bytes(mp_buf *buf, const unsigned char *s, size_t len) {
unsigned char hdr[5];
int hdrlen;
if (len < 32) {
hdr[0] = 0xa0 | (len&0xff); /* fix raw */
hdrlen = 1;
} else if (len <= 0xff) {
hdr[0] = 0xd9;
hdr[1] = len;
hdrlen = 2;
} else if (len <= 0xffff) {
hdr[0] = 0xda;
hdr[1] = (len&0xff00)>>8;
@@ -198,7 +202,7 @@ static void mp_encode_bytes(mp_buf *buf, const unsigned char *s, size_t len) {
}
/* we assume IEEE 754 internal format for single and double precision floats. */
static void mp_encode_double(mp_buf *buf, double d) {
void mp_encode_double(mp_buf *buf, double d) {
unsigned char b[9];
float f = d;
@@ -216,7 +220,7 @@ static void mp_encode_double(mp_buf *buf, double d) {
}
}
static void mp_encode_int(mp_buf *buf, int64_t n) {
void mp_encode_int(mp_buf *buf, int64_t n) {
unsigned char b[9];
int enclen;
@@ -288,7 +292,7 @@ static void mp_encode_int(mp_buf *buf, int64_t n) {
mp_buf_append(buf,b,enclen);
}
static void mp_encode_array(mp_buf *buf, int64_t n) {
void mp_encode_array(mp_buf *buf, int64_t n) {
unsigned char b[5];
int enclen;
@@ -311,7 +315,7 @@ static void mp_encode_array(mp_buf *buf, int64_t n) {
mp_buf_append(buf,b,enclen);
}
static void mp_encode_map(mp_buf *buf, int64_t n) {
void mp_encode_map(mp_buf *buf, int64_t n) {
unsigned char b[5];
int enclen;
@@ -336,7 +340,7 @@ static void mp_encode_map(mp_buf *buf, int64_t n) {
/* --------------------------- Lua types encoding --------------------------- */
static void mp_encode_lua_string(lua_State *L, mp_buf *buf) {
void mp_encode_lua_string(lua_State *L, mp_buf *buf) {
size_t len;
const char *s;
@@ -344,13 +348,13 @@ static void mp_encode_lua_string(lua_State *L, mp_buf *buf) {
mp_encode_bytes(buf,(const unsigned char*)s,len);
}
static void mp_encode_lua_bool(lua_State *L, mp_buf *buf) {
void mp_encode_lua_bool(lua_State *L, mp_buf *buf) {
unsigned char b = lua_toboolean(L,-1) ? 0xc3 : 0xc2;
mp_buf_append(buf,&b,1);
}
/* Lua 5.3 has a built in 64-bit integer type */
static void mp_encode_lua_integer(lua_State *L, mp_buf *buf) {
void mp_encode_lua_integer(lua_State *L, mp_buf *buf) {
#if (LUA_VERSION_NUM < 503) && BITS_32
lua_Number i = lua_tonumber(L,-1);
#else
@@ -362,7 +366,7 @@ static void mp_encode_lua_integer(lua_State *L, mp_buf *buf) {
/* Lua 5.2 and lower only has 64-bit doubles, so we need to
* detect if the double may be representable as an int
* for Lua < 5.3 */
static void mp_encode_lua_number(lua_State *L, mp_buf *buf) {
void mp_encode_lua_number(lua_State *L, mp_buf *buf) {
lua_Number n = lua_tonumber(L,-1);
if (IS_INT64_EQUIVALENT(n)) {
@@ -372,10 +376,10 @@ static void mp_encode_lua_number(lua_State *L, mp_buf *buf) {
}
}
static void mp_encode_lua_type(lua_State *L, mp_buf *buf, int level);
void mp_encode_lua_type(lua_State *L, mp_buf *buf, int level);
/* Convert a lua table into a message pack list. */
static void mp_encode_lua_table_as_array(lua_State *L, mp_buf *buf, int level) {
void mp_encode_lua_table_as_array(lua_State *L, mp_buf *buf, int level) {
#if LUA_VERSION_NUM < 502
size_t len = lua_objlen(L,-1), j;
#else
@@ -391,7 +395,7 @@ static void mp_encode_lua_table_as_array(lua_State *L, mp_buf *buf, int level) {
}
/* Convert a lua table into a message pack key-value map. */
static void mp_encode_lua_table_as_map(lua_State *L, mp_buf *buf, int level) {
void mp_encode_lua_table_as_map(lua_State *L, mp_buf *buf, int level) {
size_t len = 0;
/* First step: count keys into table. No other way to do it with the
@@ -418,7 +422,7 @@ static void mp_encode_lua_table_as_map(lua_State *L, mp_buf *buf, int level) {
/* Returns true if the Lua table on top of the stack is exclusively composed
* of keys from numerical keys from 1 up to N, with N being the total number
* of elements, without any hole in the middle. */
static int table_is_an_array(lua_State *L) {
int table_is_an_array(lua_State *L) {
int count = 0, max = 0;
#if LUA_VERSION_NUM < 503
lua_Number n;
@@ -461,14 +465,14 @@ static int table_is_an_array(lua_State *L) {
/* If the length operator returns non-zero, that is, there is at least
* an object at key '1', we serialize to message pack list. Otherwise
* we use a map. */
static void mp_encode_lua_table(lua_State *L, mp_buf *buf, int level) {
void mp_encode_lua_table(lua_State *L, mp_buf *buf, int level) {
if (table_is_an_array(L))
mp_encode_lua_table_as_array(L,buf,level);
else
mp_encode_lua_table_as_map(L,buf,level);
}
static void mp_encode_lua_null(lua_State *L, mp_buf *buf) {
void mp_encode_lua_null(lua_State *L, mp_buf *buf) {
unsigned char b[1];
(void)L;
@@ -476,7 +480,7 @@ static void mp_encode_lua_null(lua_State *L, mp_buf *buf) {
mp_buf_append(buf,b,1);
}
static void mp_encode_lua_type(lua_State *L, mp_buf *buf, int level) {
void mp_encode_lua_type(lua_State *L, mp_buf *buf, int level) {
int t = lua_type(L,-1);
/* Limit the encoding of nested tables to a specified maximum depth, so that
@@ -506,7 +510,7 @@ static void mp_encode_lua_type(lua_State *L, mp_buf *buf, int level) {
* Packs all arguments as a stream for multiple upacking later.
* Returns error if no arguments provided.
*/
static int mp_pack(lua_State *L) {
int mp_pack(lua_State *L) {
int nargs = lua_gettop(L);
int i;
mp_buf *buf;
@@ -687,6 +691,15 @@ void mp_decode_to_lua_type(lua_State *L, mp_cur *c) {
mp_cur_consume(c,9);
}
break;
case 0xd9: /* raw 8 */
mp_cur_need(c,2);
{
size_t l = c->p[1];
mp_cur_need(c,2+l);
lua_pushlstring(L,(char*)c->p+2,l);
mp_cur_consume(c,2+l);
}
break;
case 0xda: /* raw 16 */
mp_cur_need(c,3);
{
@@ -773,7 +786,7 @@ void mp_decode_to_lua_type(lua_State *L, mp_cur *c) {
}
}
static int mp_unpack_full(lua_State *L, int limit, int offset) {
int mp_unpack_full(lua_State *L, int limit, int offset) {
size_t len;
const char *s;
mp_cur c;
@@ -826,18 +839,18 @@ static int mp_unpack_full(lua_State *L, int limit, int offset) {
return cnt;
}
static int mp_unpack(lua_State *L) {
int mp_unpack(lua_State *L) {
return mp_unpack_full(L, 0, 0);
}
static int mp_unpack_one(lua_State *L) {
int mp_unpack_one(lua_State *L) {
int offset = luaL_optinteger(L, 2, 0);
/* Variable pop because offset may not exist */
lua_pop(L, lua_gettop(L)-1);
return mp_unpack_full(L, 1, offset);
}
static int mp_unpack_limit(lua_State *L) {
int mp_unpack_limit(lua_State *L) {
int limit = luaL_checkinteger(L, 2);
int offset = luaL_optinteger(L, 3, 0);
/* Variable pop because offset may not exist */
@@ -846,7 +859,7 @@ static int mp_unpack_limit(lua_State *L) {
return mp_unpack_full(L, limit, offset);
}
static int mp_safe(lua_State *L) {
int mp_safe(lua_State *L) {
int argc, err, total_results;
argc = lua_gettop(L);
@@ -869,7 +882,7 @@ static int mp_safe(lua_State *L) {
}
/* -------------------------------------------------------------------------- */
static const struct luaL_Reg cmds[] = {
const struct luaL_Reg cmds[] = {
{"pack", mp_pack},
{"unpack", mp_unpack},
{"unpack_one", mp_unpack_one},
@@ -877,7 +890,7 @@ static const struct luaL_Reg cmds[] = {
{0}
};
static int luaopen_create(lua_State *L) {
int luaopen_create(lua_State *L) {
int i;
/* Manually construct our module table instead of
* relying on _register or _newlib */
redis.conf
+9 -120
View File
@@ -272,7 +272,7 @@ slave-read-only yes
repl-diskless-sync no
# When diskless replication is enabled, it is possible to configure the delay
# the server waits in order to spawn the child that transfers the RDB via socket
# the server waits in order to spawn the child that trnasfers the RDB via socket
# to the slaves.
#
# This is important since once the transfer starts, it is not possible to serve
@@ -468,18 +468,15 @@ slave-priority 100
#
# The default is:
#
# maxmemory-policy noeviction
# maxmemory-policy volatile-lru
# LRU and minimal TTL algorithms are not precise algorithms but approximated
# algorithms (in order to save memory), so you can tune it for speed or
# accuracy. For default Redis will check five keys and pick the one that was
# used less recently, you can change the sample size using the following
# configuration directive.
# algorithms (in order to save memory), so you can select as well the sample
# size to check. For instance for default Redis will check three keys and
# pick the one that was used less recently, you can change the sample size
# using the following configuration directive.
#
# The default of 5 produces good enough results. 10 Approximates very closely
# true LRU but costs a bit more CPU. 3 is very fast but not very accurate.
#
# maxmemory-samples 5
# maxmemory-samples 3
############################## APPEND ONLY MODE ###############################
@@ -617,114 +614,6 @@ aof-load-truncated yes
# Set it to 0 or a negative value for unlimited execution without warnings.
lua-time-limit 5000
################################ REDIS CLUSTER ###############################
#
# ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
# WARNING EXPERIMENTAL: Redis Cluster is considered to be stable code, however
# in order to mark it as "mature" we need to wait for a non trivial percentage
# of users to deploy it in production.
# ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
#
# Normal Redis instances can't be part of a Redis Cluster; only nodes that are
# started as cluster nodes can. In order to start a Redis instance as a
# cluster node enable the cluster support uncommenting the following:
#
# cluster-enabled yes
# Every cluster node has a cluster configuration file. This file is not
# intended to be edited by hand. It is created and updated by Redis nodes.
# Every Redis Cluster node requires a different cluster configuration file.
# Make sure that instances running in the same system do not have
# overlapping cluster configuration file names.
#
# cluster-config-file nodes-6379.conf
# Cluster node timeout is the amount of milliseconds a node must be unreachable
# for it to be considered in failure state.
# Most other internal time limits are multiple of the node timeout.
#
# cluster-node-timeout 15000
# A slave of a failing master will avoid to start a failover if its data
# looks too old.
#
# There is no simple way for a slave to actually have a exact measure of
# its "data age", so the following two checks are performed:
#
# 1) If there are multiple slaves able to failover, they exchange messages
# in order to try to give an advantage to the slave with the best
# replication offset (more data from the master processed).
# Slaves will try to get their rank by offset, and apply to the start
# of the failover a delay proportional to their rank.
#
# 2) Every single slave computes the time of the last interaction with
# its master. This can be the last ping or command received (if the master
# is still in the "connected" state), or the time that elapsed since the
# disconnection with the master (if the replication link is currently down).
# If the last interaction is too old, the slave will not try to failover
# at all.
#
# The point "2" can be tuned by user. Specifically a slave will not perform
# the failover if, since the last interaction with the master, the time
# elapsed is greater than:
#
# (node-timeout * slave-validity-factor) + repl-ping-slave-period
#
# So for example if node-timeout is 30 seconds, and the slave-validity-factor
# is 10, and assuming a default repl-ping-slave-period of 10 seconds, the
# slave will not try to failover if it was not able to talk with the master
# for longer than 310 seconds.
#
# A large slave-validity-factor may allow slaves with too old data to failover
# a master, while a too small value may prevent the cluster from being able to
# elect a slave at all.
#
# For maximum availability, it is possible to set the slave-validity-factor
# to a value of 0, which means, that slaves will always try to failover the
# master regardless of the last time they interacted with the master.
# (However they'll always try to apply a delay proportional to their
# offset rank).
#
# Zero is the only value able to guarantee that when all the partitions heal
# the cluster will always be able to continue.
#
# cluster-slave-validity-factor 10
# Cluster slaves are able to migrate to orphaned masters, that are masters
# that are left without working slaves. This improves the cluster ability
# to resist to failures as otherwise an orphaned master can't be failed over
# in case of failure if it has no working slaves.
#
# Slaves migrate to orphaned masters only if there are still at least a
# given number of other working slaves for their old master. This number
# is the "migration barrier". A migration barrier of 1 means that a slave
# will migrate only if there is at least 1 other working slave for its master
# and so forth. It usually reflects the number of slaves you want for every
# master in your cluster.
#
# Default is 1 (slaves migrate only if their masters remain with at least
# one slave). To disable migration just set it to a very large value.
# A value of 0 can be set but is useful only for debugging and dangerous
# in production.
#
# cluster-migration-barrier 1
# By default Redis Cluster nodes stop accepting queries if they detect there
# is at least an hash slot uncovered (no available node is serving it).
# This way if the cluster is partially down (for example a range of hash slots
# are no longer covered) all the cluster becomes, eventually, unavailable.
# It automatically returns available as soon as all the slots are covered again.
#
# However sometimes you want the subset of the cluster which is working,
# to continue to accept queries for the part of the key space that is still
# covered. In order to do so, just set the cluster-require-full-coverage
# option to no.
#
# cluster-require-full-coverage yes
# In order to setup your cluster make sure to read the documentation
# available at http://redis.io web site.
################################## SLOW LOG ###################################
# The Redis Slow Log is a system to log queries that exceeded a specified
@@ -766,11 +655,11 @@ slowlog-max-len 128
# By default latency monitoring is disabled since it is mostly not needed
# if you don't have latency issues, and collecting data has a performance
# impact, that while very small, can be measured under big load. Latency
# monitoring can easily be enabled at runtime using the command
# monitoring can easily be enalbed at runtime using the command
# "CONFIG SET latency-monitor-threshold <milliseconds>" if needed.
latency-monitor-threshold 0
############################# EVENT NOTIFICATION ##############################
############################# Event notification ##############################
# Redis can notify Pub/Sub clients about events happening in the key space.
# This feature is documented at http://redis.io/topics/notifications
runtest-cluster
-14
View File
@@ -1,14 +0,0 @@
#!/bin/sh
TCL_VERSIONS="8.5 8.6"
TCLSH=""
for VERSION in $TCL_VERSIONS; do
TCL=`which tclsh$VERSION 2>/dev/null` && TCLSH=$TCL
done
if [ -z $TCLSH ]
then
echo "You need tcl 8.5 or newer in order to run the Redis Sentinel test"
exit 1
fi
$TCLSH tests/cluster/run.tcl $*
src/Makefile
+1 -1
View File
@@ -113,7 +113,7 @@ endif
REDIS_SERVER_NAME=redis-server
REDIS_SENTINEL_NAME=redis-sentinel
REDIS_SERVER_OBJ=adlist.o ae.o anet.o dict.o redis.o sds.o zmalloc.o lzf_c.o lzf_d.o pqsort.o zipmap.o sha1.o ziplist.o release.o networking.o util.o object.o db.o replication.o rdb.o t_string.o t_list.o t_set.o t_zset.o t_hash.o config.o aof.o pubsub.o multi.o debug.o sort.o intset.o syncio.o cluster.o crc16.o endianconv.o slowlog.o scripting.o bio.o rio.o rand.o memtest.o crc64.o bitops.o sentinel.o notify.o setproctitle.o blocked.o hyperloglog.o latency.o sparkline.o
REDIS_SERVER_OBJ=adlist.o ae.o anet.o dict.o redis.o sds.o zmalloc.o lzf_c.o lzf_d.o pqsort.o zipmap.o sha1.o ziplist.o release.o networking.o util.o object.o db.o replication.o rdb.o t_string.o t_list.o t_set.o t_zset.o t_hash.o config.o aof.o pubsub.o multi.o debug.o sort.o intset.o syncio.o migrate.o endianconv.o slowlog.o scripting.o bio.o rio.o rand.o memtest.o crc64.o bitops.o sentinel.o notify.o setproctitle.o hyperloglog.o latency.o sparkline.o
REDIS_CLI_NAME=redis-cli
REDIS_CLI_OBJ=anet.o sds.o adlist.o redis-cli.o zmalloc.o release.o anet.o ae.o crc64.o
REDIS_BENCHMARK_NAME=redis-benchmark
src/Makefile.dep
+77 -87
View File
@@ -1,142 +1,132 @@
adlist.o: adlist.c adlist.h zmalloc.h
ae.o: ae.c ae.h zmalloc.h config.h ae_kqueue.c ae_epoll.c ae_select.c ae_evport.c
ae.o: ae.c ae.h zmalloc.h config.h ae_kqueue.c ae_select.c ae_evport.c ae_epoll.c
ae_epoll.o: ae_epoll.c
ae_evport.o: ae_evport.c
ae_kqueue.o: ae_kqueue.c
ae_select.o: ae_select.c
anet.o: anet.c fmacros.h anet.h
aof.o: aof.c redis.h fmacros.h config.h ../deps/lua/src/lua.h \
../deps/lua/src/luaconf.h ae.h sds.h dict.h adlist.h zmalloc.h anet.h \
ziplist.h intset.h version.h util.h latency.h sparkline.h rdb.h rio.h \
bio.h
../deps/lua/src/luaconf.h ae.h sds.h dict.h adlist.h zmalloc.h anet.h \
ziplist.h intset.h version.h util.h latency.h sparkline.h rdb.h rio.h \
bio.h
bio.o: bio.c redis.h fmacros.h config.h ../deps/lua/src/lua.h \
../deps/lua/src/luaconf.h ae.h sds.h dict.h adlist.h zmalloc.h anet.h \
ziplist.h intset.h version.h util.h latency.h sparkline.h rdb.h rio.h \
bio.h
../deps/lua/src/luaconf.h ae.h sds.h dict.h adlist.h zmalloc.h anet.h \
ziplist.h intset.h version.h util.h latency.h sparkline.h rdb.h rio.h \
bio.h
bitops.o: bitops.c redis.h fmacros.h config.h ../deps/lua/src/lua.h \
../deps/lua/src/luaconf.h ae.h sds.h dict.h adlist.h zmalloc.h anet.h \
ziplist.h intset.h version.h util.h latency.h sparkline.h rdb.h rio.h
blocked.o: blocked.c redis.h fmacros.h config.h ../deps/lua/src/lua.h \
../deps/lua/src/luaconf.h ae.h sds.h dict.h adlist.h zmalloc.h anet.h \
ziplist.h intset.h version.h util.h latency.h sparkline.h rdb.h rio.h
cluster.o: cluster.c redis.h fmacros.h config.h ../deps/lua/src/lua.h \
../deps/lua/src/luaconf.h ae.h sds.h dict.h adlist.h zmalloc.h anet.h \
ziplist.h intset.h version.h util.h latency.h sparkline.h rdb.h rio.h \
cluster.h endianconv.h
../deps/lua/src/luaconf.h ae.h sds.h dict.h adlist.h zmalloc.h anet.h \
ziplist.h intset.h version.h util.h latency.h sparkline.h rdb.h rio.h
config.o: config.c redis.h fmacros.h config.h ../deps/lua/src/lua.h \
../deps/lua/src/luaconf.h ae.h sds.h dict.h adlist.h zmalloc.h anet.h \
ziplist.h intset.h version.h util.h latency.h sparkline.h rdb.h rio.h \
cluster.h
crc16.o: crc16.c redis.h fmacros.h config.h ../deps/lua/src/lua.h \
../deps/lua/src/luaconf.h ae.h sds.h dict.h adlist.h zmalloc.h anet.h \
ziplist.h intset.h version.h util.h latency.h sparkline.h rdb.h rio.h
../deps/lua/src/luaconf.h ae.h sds.h dict.h adlist.h zmalloc.h anet.h \
ziplist.h intset.h version.h util.h latency.h sparkline.h rdb.h rio.h
crc64.o: crc64.c
db.o: db.c redis.h fmacros.h config.h ../deps/lua/src/lua.h \
../deps/lua/src/luaconf.h ae.h sds.h dict.h adlist.h zmalloc.h anet.h \
ziplist.h intset.h version.h util.h latency.h sparkline.h rdb.h rio.h \
cluster.h
../deps/lua/src/luaconf.h ae.h sds.h dict.h adlist.h zmalloc.h anet.h \
ziplist.h intset.h version.h util.h latency.h sparkline.h rdb.h rio.h
debug.o: debug.c redis.h fmacros.h config.h ../deps/lua/src/lua.h \
../deps/lua/src/luaconf.h ae.h sds.h dict.h adlist.h zmalloc.h anet.h \
ziplist.h intset.h version.h util.h latency.h sparkline.h rdb.h rio.h \
sha1.h crc64.h bio.h
../deps/lua/src/luaconf.h ae.h sds.h dict.h adlist.h zmalloc.h anet.h \
ziplist.h intset.h version.h util.h latency.h sparkline.h rdb.h rio.h \
sha1.h crc64.h bio.h
dict.o: dict.c fmacros.h dict.h zmalloc.h redisassert.h
endianconv.o: endianconv.c
hyperloglog.o: hyperloglog.c redis.h fmacros.h config.h \
../deps/lua/src/lua.h ../deps/lua/src/luaconf.h ae.h sds.h dict.h \
adlist.h zmalloc.h anet.h ziplist.h intset.h version.h util.h latency.h \
sparkline.h rdb.h rio.h
../deps/lua/src/lua.h ../deps/lua/src/luaconf.h ae.h sds.h dict.h \
adlist.h zmalloc.h anet.h ziplist.h intset.h version.h util.h \
latency.h sparkline.h rdb.h rio.h
intset.o: intset.c intset.h zmalloc.h endianconv.h config.h
latency.o: latency.c redis.h fmacros.h config.h ../deps/lua/src/lua.h \
../deps/lua/src/luaconf.h ae.h sds.h dict.h adlist.h zmalloc.h anet.h \
ziplist.h intset.h version.h util.h latency.h sparkline.h rdb.h rio.h
../deps/lua/src/luaconf.h ae.h sds.h dict.h adlist.h zmalloc.h anet.h \
ziplist.h intset.h version.h util.h latency.h sparkline.h rdb.h rio.h
lzf_c.o: lzf_c.c lzfP.h
lzf_d.o: lzf_d.c lzfP.h
memtest.o: memtest.c config.h
migrate.o: migrate.c redis.h fmacros.h config.h ../deps/lua/src/lua.h \
../deps/lua/src/luaconf.h ae.h sds.h dict.h adlist.h zmalloc.h anet.h \
ziplist.h intset.h version.h util.h latency.h sparkline.h rdb.h rio.h \
endianconv.h
multi.o: multi.c redis.h fmacros.h config.h ../deps/lua/src/lua.h \
../deps/lua/src/luaconf.h ae.h sds.h dict.h adlist.h zmalloc.h anet.h \
ziplist.h intset.h version.h util.h latency.h sparkline.h rdb.h rio.h
../deps/lua/src/luaconf.h ae.h sds.h dict.h adlist.h zmalloc.h anet.h \
ziplist.h intset.h version.h util.h latency.h sparkline.h rdb.h rio.h
networking.o: networking.c redis.h fmacros.h config.h \
../deps/lua/src/lua.h ../deps/lua/src/luaconf.h ae.h sds.h dict.h \
adlist.h zmalloc.h anet.h ziplist.h intset.h version.h util.h latency.h \
sparkline.h rdb.h rio.h
../deps/lua/src/lua.h ../deps/lua/src/luaconf.h ae.h sds.h dict.h \
adlist.h zmalloc.h anet.h ziplist.h intset.h version.h util.h \
latency.h sparkline.h rdb.h rio.h
notify.o: notify.c redis.h fmacros.h config.h ../deps/lua/src/lua.h \
../deps/lua/src/luaconf.h ae.h sds.h dict.h adlist.h zmalloc.h anet.h \
ziplist.h intset.h version.h util.h latency.h sparkline.h rdb.h rio.h
../deps/lua/src/luaconf.h ae.h sds.h dict.h adlist.h zmalloc.h anet.h \
ziplist.h intset.h version.h util.h latency.h sparkline.h rdb.h rio.h
object.o: object.c redis.h fmacros.h config.h ../deps/lua/src/lua.h \
../deps/lua/src/luaconf.h ae.h sds.h dict.h adlist.h zmalloc.h anet.h \
ziplist.h intset.h version.h util.h latency.h sparkline.h rdb.h rio.h
../deps/lua/src/luaconf.h ae.h sds.h dict.h adlist.h zmalloc.h anet.h \
ziplist.h intset.h version.h util.h latency.h sparkline.h rdb.h rio.h
pqsort.o: pqsort.c
pubsub.o: pubsub.c redis.h fmacros.h config.h ../deps/lua/src/lua.h \
../deps/lua/src/luaconf.h ae.h sds.h dict.h adlist.h zmalloc.h anet.h \
ziplist.h intset.h version.h util.h latency.h sparkline.h rdb.h rio.h
../deps/lua/src/luaconf.h ae.h sds.h dict.h adlist.h zmalloc.h anet.h \
ziplist.h intset.h version.h util.h latency.h sparkline.h rdb.h rio.h
rand.o: rand.c
rdb.o: rdb.c redis.h fmacros.h config.h ../deps/lua/src/lua.h \
../deps/lua/src/luaconf.h ae.h sds.h dict.h adlist.h zmalloc.h anet.h \
ziplist.h intset.h version.h util.h latency.h sparkline.h rdb.h rio.h \
lzf.h zipmap.h endianconv.h
../deps/lua/src/luaconf.h ae.h sds.h dict.h adlist.h zmalloc.h anet.h \
ziplist.h intset.h version.h util.h latency.h sparkline.h rdb.h rio.h \
lzf.h zipmap.h endianconv.h
redis-benchmark.o: redis-benchmark.c fmacros.h ae.h \
../deps/hiredis/hiredis.h sds.h adlist.h zmalloc.h
../deps/hiredis/hiredis.h sds.h adlist.h zmalloc.h
redis-check-aof.o: redis-check-aof.c fmacros.h config.h
redis-check-dump.o: redis-check-dump.c lzf.h crc64.h
redis-cli.o: redis-cli.c fmacros.h version.h ../deps/hiredis/hiredis.h \
sds.h zmalloc.h ../deps/linenoise/linenoise.h help.h anet.h ae.h
sds.h zmalloc.h ../deps/linenoise/linenoise.h help.h anet.h ae.h
redis.o: redis.c redis.h fmacros.h config.h ../deps/lua/src/lua.h \
../deps/lua/src/luaconf.h ae.h sds.h dict.h adlist.h zmalloc.h anet.h \
ziplist.h intset.h version.h util.h latency.h sparkline.h rdb.h rio.h \
cluster.h slowlog.h bio.h asciilogo.h
../deps/lua/src/luaconf.h ae.h sds.h dict.h adlist.h zmalloc.h anet.h \
ziplist.h intset.h version.h util.h latency.h sparkline.h rdb.h rio.h \
slowlog.h bio.h asciilogo.h
release.o: release.c release.h version.h crc64.h
replication.o: replication.c redis.h fmacros.h config.h \
../deps/lua/src/lua.h ../deps/lua/src/luaconf.h ae.h sds.h dict.h \
adlist.h zmalloc.h anet.h ziplist.h intset.h version.h util.h latency.h \
sparkline.h rdb.h rio.h
../deps/lua/src/lua.h ../deps/lua/src/luaconf.h ae.h sds.h dict.h \
adlist.h zmalloc.h anet.h ziplist.h intset.h version.h util.h \
latency.h sparkline.h rdb.h rio.h
rio.o: rio.c fmacros.h rio.h sds.h util.h crc64.h config.h redis.h \
../deps/lua/src/lua.h ../deps/lua/src/luaconf.h ae.h dict.h adlist.h \
zmalloc.h anet.h ziplist.h intset.h version.h latency.h sparkline.h \
rdb.h
../deps/lua/src/lua.h ../deps/lua/src/luaconf.h ae.h dict.h adlist.h \
zmalloc.h anet.h ziplist.h intset.h version.h latency.h sparkline.h \
rdb.h
scripting.o: scripting.c redis.h fmacros.h config.h ../deps/lua/src/lua.h \
../deps/lua/src/luaconf.h ae.h sds.h dict.h adlist.h zmalloc.h anet.h \
ziplist.h intset.h version.h util.h latency.h sparkline.h rdb.h rio.h \
sha1.h rand.h ../deps/lua/src/lauxlib.h ../deps/lua/src/lua.h \
../deps/lua/src/lualib.h
../deps/lua/src/luaconf.h ae.h sds.h dict.h adlist.h zmalloc.h anet.h \
ziplist.h intset.h version.h util.h latency.h sparkline.h rdb.h rio.h \
sha1.h rand.h ../deps/lua/src/lauxlib.h ../deps/lua/src/lualib.h
sds.o: sds.c sds.h zmalloc.h
sentinel.o: sentinel.c redis.h fmacros.h config.h ../deps/lua/src/lua.h \
../deps/lua/src/luaconf.h ae.h sds.h dict.h adlist.h zmalloc.h anet.h \
ziplist.h intset.h version.h util.h latency.h sparkline.h rdb.h rio.h \
../deps/hiredis/hiredis.h ../deps/hiredis/async.h \
../deps/hiredis/hiredis.h
../deps/lua/src/luaconf.h ae.h sds.h dict.h adlist.h zmalloc.h anet.h \
ziplist.h intset.h version.h util.h latency.h sparkline.h rdb.h rio.h \
../deps/hiredis/hiredis.h ../deps/hiredis/async.h
setproctitle.o: setproctitle.c
sha1.o: sha1.c sha1.h config.h
slowlog.o: slowlog.c redis.h fmacros.h config.h ../deps/lua/src/lua.h \
../deps/lua/src/luaconf.h ae.h sds.h dict.h adlist.h zmalloc.h anet.h \
ziplist.h intset.h version.h util.h latency.h sparkline.h rdb.h rio.h \
slowlog.h
../deps/lua/src/luaconf.h ae.h sds.h dict.h adlist.h zmalloc.h anet.h \
ziplist.h intset.h version.h util.h latency.h sparkline.h rdb.h rio.h \
slowlog.h
sort.o: sort.c redis.h fmacros.h config.h ../deps/lua/src/lua.h \
../deps/lua/src/luaconf.h ae.h sds.h dict.h adlist.h zmalloc.h anet.h \
ziplist.h intset.h version.h util.h latency.h sparkline.h rdb.h rio.h \
pqsort.h
../deps/lua/src/luaconf.h ae.h sds.h dict.h adlist.h zmalloc.h anet.h \
ziplist.h intset.h version.h util.h latency.h sparkline.h rdb.h rio.h \
pqsort.h
sparkline.o: sparkline.c redis.h fmacros.h config.h ../deps/lua/src/lua.h \
../deps/lua/src/luaconf.h ae.h sds.h dict.h adlist.h zmalloc.h anet.h \
ziplist.h intset.h version.h util.h latency.h sparkline.h rdb.h rio.h
../deps/lua/src/luaconf.h ae.h sds.h dict.h adlist.h zmalloc.h anet.h \
ziplist.h intset.h version.h util.h latency.h sparkline.h rdb.h rio.h
syncio.o: syncio.c redis.h fmacros.h config.h ../deps/lua/src/lua.h \
../deps/lua/src/luaconf.h ae.h sds.h dict.h adlist.h zmalloc.h anet.h \
ziplist.h intset.h version.h util.h latency.h sparkline.h rdb.h rio.h
../deps/lua/src/luaconf.h ae.h sds.h dict.h adlist.h zmalloc.h anet.h \
ziplist.h intset.h version.h util.h latency.h sparkline.h rdb.h rio.h
t_hash.o: t_hash.c redis.h fmacros.h config.h ../deps/lua/src/lua.h \
../deps/lua/src/luaconf.h ae.h sds.h dict.h adlist.h zmalloc.h anet.h \
ziplist.h intset.h version.h util.h latency.h sparkline.h rdb.h rio.h
../deps/lua/src/luaconf.h ae.h sds.h dict.h adlist.h zmalloc.h anet.h \
ziplist.h intset.h version.h util.h latency.h sparkline.h rdb.h rio.h
t_list.o: t_list.c redis.h fmacros.h config.h ../deps/lua/src/lua.h \
../deps/lua/src/luaconf.h ae.h sds.h dict.h adlist.h zmalloc.h anet.h \
ziplist.h intset.h version.h util.h latency.h sparkline.h rdb.h rio.h
../deps/lua/src/luaconf.h ae.h sds.h dict.h adlist.h zmalloc.h anet.h \
ziplist.h intset.h version.h util.h latency.h sparkline.h rdb.h rio.h
t_set.o: t_set.c redis.h fmacros.h config.h ../deps/lua/src/lua.h \
../deps/lua/src/luaconf.h ae.h sds.h dict.h adlist.h zmalloc.h anet.h \
ziplist.h intset.h version.h util.h latency.h sparkline.h rdb.h rio.h
../deps/lua/src/luaconf.h ae.h sds.h dict.h adlist.h zmalloc.h anet.h \
ziplist.h intset.h version.h util.h latency.h sparkline.h rdb.h rio.h
t_string.o: t_string.c redis.h fmacros.h config.h ../deps/lua/src/lua.h \
../deps/lua/src/luaconf.h ae.h sds.h dict.h adlist.h zmalloc.h anet.h \
ziplist.h intset.h version.h util.h latency.h sparkline.h rdb.h rio.h
../deps/lua/src/luaconf.h ae.h sds.h dict.h adlist.h zmalloc.h anet.h \
ziplist.h intset.h version.h util.h latency.h sparkline.h rdb.h rio.h
t_zset.o: t_zset.c redis.h fmacros.h config.h ../deps/lua/src/lua.h \
../deps/lua/src/luaconf.h ae.h sds.h dict.h adlist.h zmalloc.h anet.h \
ziplist.h intset.h version.h util.h latency.h sparkline.h rdb.h rio.h
../deps/lua/src/luaconf.h ae.h sds.h dict.h adlist.h zmalloc.h anet.h \
ziplist.h intset.h version.h util.h latency.h sparkline.h rdb.h rio.h
util.o: util.c fmacros.h util.h sds.h
ziplist.o: ziplist.c zmalloc.h util.h sds.h ziplist.h endianconv.h \
config.h redisassert.h
config.h redisassert.h
zipmap.o: zipmap.c zmalloc.h endianconv.h config.h
zmalloc.o: zmalloc.c config.h zmalloc.h
src/anet.c
+11 -51
View File
@@ -264,12 +264,11 @@ static int anetCreateSocket(char *err, int domain) {
#define ANET_CONNECT_NONE 0
#define ANET_CONNECT_NONBLOCK 1
static int anetTcpGenericConnect(char *err, char *addr, int port,
char *source_addr, int flags)
static int anetTcpGenericConnect(char *err, char *addr, int port, int flags)
{
int s = ANET_ERR, rv;
char portstr[6]; /* strlen("65535") + 1; */
struct addrinfo hints, *servinfo, *bservinfo, *p, *b;
struct addrinfo hints, *servinfo, *p;
snprintf(portstr,sizeof(portstr),"%d",port);
memset(&hints,0,sizeof(hints));
@@ -289,26 +288,6 @@ static int anetTcpGenericConnect(char *err, char *addr, int port,
if (anetSetReuseAddr(err,s) == ANET_ERR) goto error;
if (flags & ANET_CONNECT_NONBLOCK && anetNonBlock(err,s) != ANET_OK)
goto error;
if (source_addr) {
int bound = 0;
/* Using getaddrinfo saves us from self-determining IPv4 vs IPv6 */
if ((rv = getaddrinfo(source_addr, NULL, &hints, &bservinfo)) != 0)
{
anetSetError(err, "%s", gai_strerror(rv));
goto end;
}
for (b = bservinfo; b != NULL; b = b->ai_next) {
if (bind(s,b->ai_addr,b->ai_addrlen) != -1) {
bound = 1;
break;
}
}
freeaddrinfo(bservinfo);
if (!bound) {
anetSetError(err, "bind: %s", strerror(errno));
goto end;
}
}
if (connect(s,p->ai_addr,p->ai_addrlen) == -1) {
/* If the socket is non-blocking, it is ok for connect() to
* return an EINPROGRESS error here. */
@@ -338,17 +317,12 @@ end:
int anetTcpConnect(char *err, char *addr, int port)
{
return anetTcpGenericConnect(err,addr,port,NULL,ANET_CONNECT_NONE);
return anetTcpGenericConnect(err,addr,port,ANET_CONNECT_NONE);
}
int anetTcpNonBlockConnect(char *err, char *addr, int port)
{
return anetTcpGenericConnect(err,addr,port,NULL,ANET_CONNECT_NONBLOCK);
}
int anetTcpNonBlockBindConnect(char *err, char *addr, int port, char *source_addr)
{
return anetTcpGenericConnect(err,addr,port,source_addr,ANET_CONNECT_NONBLOCK);
return anetTcpGenericConnect(err,addr,port,ANET_CONNECT_NONBLOCK);
}
int anetUnixGenericConnect(char *err, char *path, int flags)
@@ -557,36 +531,22 @@ int anetPeerToString(int fd, char *ip, size_t ip_len, int *port) {
struct sockaddr_storage sa;
socklen_t salen = sizeof(sa);
if (getpeername(fd,(struct sockaddr*)&sa,&salen) == -1) goto error;
if (ip_len == 0) goto error;
if (getpeername(fd,(struct sockaddr*)&sa,&salen) == -1) {
if (port) *port = 0;
ip[0] = '?';
ip[1] = '\0';
return -1;
}
if (sa.ss_family == AF_INET) {
struct sockaddr_in *s = (struct sockaddr_in *)&sa;
if (ip) inet_ntop(AF_INET,(void*)&(s->sin_addr),ip,ip_len);
if (port) *port = ntohs(s->sin_port);
} else if (sa.ss_family == AF_INET6) {
} else {
struct sockaddr_in6 *s = (struct sockaddr_in6 *)&sa;
if (ip) inet_ntop(AF_INET6,(void*)&(s->sin6_addr),ip,ip_len);
if (port) *port = ntohs(s->sin6_port);
} else if (sa.ss_family == AF_UNIX) {
if (ip) strncpy(ip,"/unixsocket",ip_len);
if (port) *port = 0;
} else {
goto error;
}
return 0;
error:
if (ip) {
if (ip_len >= 2) {
ip[0] = '?';
ip[1] = '\0';
} else if (ip_len == 1) {
ip[0] = '\0';
}
}
if (port) *port = 0;
return -1;
}
int anetSockName(int fd, char *ip, size_t ip_len, int *port) {
src/anet.h
-1
View File
@@ -49,7 +49,6 @@
int anetTcpConnect(char *err, char *addr, int port);
int anetTcpNonBlockConnect(char *err, char *addr, int port);
int anetTcpNonBlockBindConnect(char *err, char *addr, int port, char *source_addr);
int anetUnixConnect(char *err, char *path);
int anetUnixNonBlockConnect(char *err, char *path);
int anetRead(int fd, char *buf, int count);
src/aof.c
+9 -201
View File
@@ -40,7 +40,6 @@
#include <sys/wait.h>
void aofUpdateCurrentSize(void);
void aofClosePipes(void);
/* ----------------------------------------------------------------------------
* AOF rewrite buffer implementation.
@@ -76,49 +75,16 @@ void aofRewriteBufferReset(void) {
/* Return the current size of the AOF rewrite buffer. */
unsigned long aofRewriteBufferSize(void) {
listNode *ln;
listIter li;
unsigned long size = 0;
listNode *ln = listLast(server.aof_rewrite_buf_blocks);
aofrwblock *block = ln ? ln->value : NULL;
listRewind(server.aof_rewrite_buf_blocks,&li);
while((ln = listNext(&li))) {
aofrwblock *block = listNodeValue(ln);
size += block->used;
}
if (block == NULL) return 0;
unsigned long size =
(listLength(server.aof_rewrite_buf_blocks)-1) * AOF_RW_BUF_BLOCK_SIZE;
size += block->used;
return size;
}
/* Event handler used to send data to the child process doing the AOF
* rewrite. We send pieces of our AOF differences buffer so that the final
* write when the child finishes the rewrite will be small. */
void aofChildWriteDiffData(aeEventLoop *el, int fd, void *privdata, int mask) {
listNode *ln;
aofrwblock *block;
ssize_t nwritten;
REDIS_NOTUSED(el);
REDIS_NOTUSED(fd);
REDIS_NOTUSED(privdata);
REDIS_NOTUSED(mask);
while(1) {
ln = listFirst(server.aof_rewrite_buf_blocks);
block = ln ? ln->value : NULL;
if (server.aof_stop_sending_diff || !block) {
aeDeleteFileEvent(server.el,server.aof_pipe_write_data_to_child,
AE_WRITABLE);
return;
}
if (block->used > 0) {
nwritten = write(server.aof_pipe_write_data_to_child,
block->buf,block->used);
if (nwritten <= 0) return;
memmove(block->buf,block->buf+nwritten,block->used-nwritten);
block->used -= nwritten;
}
if (block->used == 0) listDelNode(server.aof_rewrite_buf_blocks,ln);
}
}
/* Append data to the AOF rewrite buffer, allocating new blocks if needed. */
void aofRewriteBufferAppend(unsigned char *s, unsigned long len) {
listNode *ln = listLast(server.aof_rewrite_buf_blocks);
@@ -157,13 +123,6 @@ void aofRewriteBufferAppend(unsigned char *s, unsigned long len) {
}
}
}
/* Install a file event to send data to the rewrite child if there is
* not one already. */
if (aeGetFileEvents(server.el,server.aof_pipe_write_data_to_child) == 0) {
aeCreateFileEvent(server.el, server.aof_pipe_write_data_to_child,
AE_WRITABLE, aofChildWriteDiffData, NULL);
}
}
/* Write the buffer (possibly composed of multiple blocks) into the specified
@@ -225,8 +184,6 @@ void stopAppendOnly(void) {
aofRemoveTempFile(server.aof_child_pid);
server.aof_child_pid = -1;
server.aof_rewrite_time_start = -1;
/* close pipes used for IPC between the two processes. */
aofClosePipes();
}
}
@@ -562,7 +519,6 @@ struct redisClient *createFakeClient(void) {
c->argv = NULL;
c->bufpos = 0;
c->flags = 0;
c->btype = REDIS_BLOCKED_NONE;
/* We set the fake client as a slave waiting for the synchronization
* so that Redis will not try to send replies to this client. */
c->replstate = REDIS_REPL_WAIT_BGSAVE_START;
@@ -758,7 +714,7 @@ int rioWriteBulkObject(rio *r, robj *obj) {
* in a child process when this function is called). */
if (obj->encoding == REDIS_ENCODING_INT) {
return rioWriteBulkLongLong(r,(long)obj->ptr);
} else if (sdsEncodedObject(obj)) {
} else if (obj->encoding == REDIS_ENCODING_RAW) {
return rioWriteBulkString(r,obj->ptr,sdslen(obj->ptr));
} else {
redisPanic("Unknown string encoding");
@@ -996,21 +952,6 @@ int rewriteHashObject(rio *r, robj *key, robj *o) {
return 1;
}
/* This function is called by the child rewriting the AOF file to read
* the difference accumulated from the parent into a buffer, that is
* concatenated at the end of the rewrite. */
ssize_t aofReadDiffFromParent(void) {
char buf[65536]; /* Default pipe buffer size on most Linux systems. */
ssize_t nread, total = 0;
while ((nread =
read(server.aof_pipe_read_data_from_parent,buf,sizeof(buf))) > 0) {
server.aof_child_diff = sdscatlen(server.aof_child_diff,buf,nread);
total += nread;
}
return total;
}
/* Write a sequence of commands able to fully rebuild the dataset into
* "filename". Used both by REWRITEAOF and BGREWRITEAOF.
*
@@ -1026,8 +967,6 @@ int rewriteAppendOnlyFile(char *filename) {
char tmpfile[256];
int j;
long long now = mstime();
char byte;
size_t processed = 0;
/* Note that we have to use a different temp name here compared to the
* one used by rewriteAppendOnlyFileBackground() function. */
@@ -1038,7 +977,6 @@ int rewriteAppendOnlyFile(char *filename) {
return REDIS_ERR;
}
server.aof_child_diff = sdsempty();
rioInitWithFile(&aof,fp);
if (server.aof_rewrite_incremental_fsync)
rioSetAutoSync(&aof,REDIS_AOF_AUTOSYNC_BYTES);
@@ -1098,61 +1036,11 @@ int rewriteAppendOnlyFile(char *filename) {
if (rioWriteBulkObject(&aof,&key) == 0) goto werr;
if (rioWriteBulkLongLong(&aof,expiretime) == 0) goto werr;
}
/* Read some diff from the parent process from time to time. */
if (aof.processed_bytes > processed+1024*10) {
processed = aof.processed_bytes;
aofReadDiffFromParent();
}
}
dictReleaseIterator(di);
di = NULL;
}
/* Do an initial slow fsync here while the parent is still sending
* data, in order to make the next final fsync faster. */
if (fflush(fp) == EOF) goto werr;
if (fsync(fileno(fp)) == -1) goto werr;
/* Read again a few times to get more data from the parent.
* We can't read forever (the server may receive data from clients
* faster than it is able to send data to the child), so we try to read
* some more data in a loop as soon as there is a good chance more data
* will come. If it looks like we are wasting time, we abort (this
* happens after 20 ms without new data). */
int nodata = 0;
mstime_t start = mstime();
while(mstime()-start < 1000 && nodata < 20) {
if (aeWait(server.aof_pipe_read_data_from_parent, AE_READABLE, 1) <= 0)
{
nodata++;
continue;
}
nodata = 0; /* Start counting from zero, we stop on N *contiguous*
timeouts. */
aofReadDiffFromParent();
}
/* Ask the master to stop sending diffs. */
if (write(server.aof_pipe_write_ack_to_parent,"!",1) != 1) goto werr;
if (anetNonBlock(NULL,server.aof_pipe_read_ack_from_parent) != ANET_OK)
goto werr;
/* We read the ACK from the server using a 10 seconds timeout. Normally
* it should reply ASAP, but just in case we lose its reply, we are sure
* the child will eventually get terminated. */
if (syncRead(server.aof_pipe_read_ack_from_parent,&byte,1,5000) != 1 ||
byte != '!') goto werr;
redisLog(REDIS_NOTICE,"Parent agreed to stop sending diffs. Finalizing AOF...");
/* Read the final diff if any. */
aofReadDiffFromParent();
/* Write the received diff to the file. */
redisLog(REDIS_NOTICE,
"Concatenating %.2f MB of AOF diff received from parent.",
(double) sdslen(server.aof_child_diff) / (1024*1024));
if (rioWrite(&aof,server.aof_child_diff,sdslen(server.aof_child_diff)) == 0)
goto werr;
/* Make sure data will not remain on the OS's output buffers */
if (fflush(fp) == EOF) goto werr;
if (fsync(fileno(fp)) == -1) goto werr;
@@ -1169,91 +1057,13 @@ int rewriteAppendOnlyFile(char *filename) {
return REDIS_OK;
werr:
redisLog(REDIS_WARNING,"Write error writing append only file on disk: %s", strerror(errno));
fclose(fp);
unlink(tmpfile);
redisLog(REDIS_WARNING,"Write error writing append only file on disk: %s", strerror(errno));
if (di) dictReleaseIterator(di);
return REDIS_ERR;
}
/* ----------------------------------------------------------------------------
* AOF rewrite pipes for IPC
* -------------------------------------------------------------------------- */
/* This event handler is called when the AOF rewriting child sends us a
* single '!' char to signal we should stop sending buffer diffs. The
* parent sends a '!' as well to acknowledge. */
void aofChildPipeReadable(aeEventLoop *el, int fd, void *privdata, int mask) {
char byte;
REDIS_NOTUSED(el);
REDIS_NOTUSED(privdata);
REDIS_NOTUSED(mask);
if (read(fd,&byte,1) == 1 && byte == '!') {
redisLog(REDIS_NOTICE,"AOF rewrite child asks to stop sending diffs.");
server.aof_stop_sending_diff = 1;
if (write(server.aof_pipe_write_ack_to_child,"!",1) != 1) {
/* If we can't send the ack, inform the user, but don't try again
* since in the other side the children will use a timeout if the
* kernel can't buffer our write, or, the children was
* terminated. */
redisLog(REDIS_WARNING,"Can't send ACK to AOF child: %s",
strerror(errno));
}
}
/* Remove the handler since this can be called only one time during a
* rewrite. */
aeDeleteFileEvent(server.el,server.aof_pipe_read_ack_from_child,AE_READABLE);
}
/* Create the pipes used for parent - child process IPC during rewrite.
* We have a data pipe used to send AOF incremental diffs to the child,
* and two other pipes used by the children to signal it finished with
* the rewrite so no more data should be written, and another for the
* parent to acknowledge it understood this new condition. */
int aofCreatePipes(void) {
int fds[6] = {-1, -1, -1, -1, -1, -1};
int j;
if (pipe(fds) == -1) goto error; /* parent -> children data. */
if (pipe(fds+2) == -1) goto error; /* children -> parent ack. */
if (pipe(fds+4) == -1) goto error; /* children -> parent ack. */
/* Parent -> children data is non blocking. */
if (anetNonBlock(NULL,fds[0]) != ANET_OK) goto error;
if (anetNonBlock(NULL,fds[1]) != ANET_OK) goto error;
if (aeCreateFileEvent(server.el, fds[2], AE_READABLE, aofChildPipeReadable, NULL) == AE_ERR) goto error;
server.aof_pipe_write_data_to_child = fds[1];
server.aof_pipe_read_data_from_parent = fds[0];
server.aof_pipe_write_ack_to_parent = fds[3];
server.aof_pipe_read_ack_from_child = fds[2];
server.aof_pipe_write_ack_to_child = fds[5];
server.aof_pipe_read_ack_from_parent = fds[4];
server.aof_stop_sending_diff = 0;
return REDIS_OK;
error:
redisLog(REDIS_WARNING,"Error opening /setting AOF rewrite IPC pipes: %s",
strerror(errno));
for (j = 0; j < 6; j++) if(fds[j] != -1) close(fds[j]);
return REDIS_ERR;
}
void aofClosePipes(void) {
aeDeleteFileEvent(server.el,server.aof_pipe_read_ack_from_child,AE_READABLE);
aeDeleteFileEvent(server.el,server.aof_pipe_write_data_to_child,AE_WRITABLE);
close(server.aof_pipe_write_data_to_child);
close(server.aof_pipe_read_data_from_parent);
close(server.aof_pipe_write_ack_to_parent);
close(server.aof_pipe_read_ack_from_child);
close(server.aof_pipe_write_ack_to_child);
close(server.aof_pipe_read_ack_from_parent);
}
/* ----------------------------------------------------------------------------
* AOF background rewrite
* ------------------------------------------------------------------------- */
/* This is how rewriting of the append only file in background works:
*
* 1) The user calls BGREWRITEAOF
@@ -1271,7 +1081,6 @@ int rewriteAppendOnlyFileBackground(void) {
long long start;
if (server.aof_child_pid != -1) return REDIS_ERR;
if (aofCreatePipes() != REDIS_OK) return REDIS_ERR;
start = ustime();
if ((childpid = fork()) == 0) {
char tmpfile[256];
@@ -1393,7 +1202,7 @@ void backgroundRewriteDoneHandler(int exitcode, int bysignal) {
latencyAddSampleIfNeeded("aof-rewrite-diff-write",latency);
redisLog(REDIS_NOTICE,
"Residual parent diff successfully flushed to the rewritten AOF (%.2f MB)", (double) aofRewriteBufferSize() / (1024*1024));
"Parent diff successfully flushed to the rewritten AOF (%lu bytes)", aofRewriteBufferSize());
/* The only remaining thing to do is to rename the temporary file to
* the configured file and switch the file descriptor used to do AOF
@@ -1494,7 +1303,6 @@ void backgroundRewriteDoneHandler(int exitcode, int bysignal) {
}
cleanup:
aofClosePipes();
aofRewriteBufferReset();
aofRemoveTempFile(server.aof_child_pid);
server.aof_child_pid = -1;
src/bitops.c
+13 -25
View File
@@ -70,19 +70,16 @@ size_t redisPopcount(void *s, long count) {
count--;
}
/* Count bits 28 bytes at a time */
/* Count bits 16 bytes at a time */
p4 = (uint32_t*)p;
while(count>=28) {
uint32_t aux1, aux2, aux3, aux4, aux5, aux6, aux7;
while(count>=16) {
uint32_t aux1, aux2, aux3, aux4;
aux1 = *p4++;
aux2 = *p4++;
aux3 = *p4++;
aux4 = *p4++;
aux5 = *p4++;
aux6 = *p4++;
aux7 = *p4++;
count -= 28;
count -= 16;
aux1 = aux1 - ((aux1 >> 1) & 0x55555555);
aux1 = (aux1 & 0x33333333) + ((aux1 >> 2) & 0x33333333);
@@ -92,19 +89,10 @@ size_t redisPopcount(void *s, long count) {
aux3 = (aux3 & 0x33333333) + ((aux3 >> 2) & 0x33333333);
aux4 = aux4 - ((aux4 >> 1) & 0x55555555);
aux4 = (aux4 & 0x33333333) + ((aux4 >> 2) & 0x33333333);
aux5 = aux5 - ((aux5 >> 1) & 0x55555555);
aux5 = (aux5 & 0x33333333) + ((aux5 >> 2) & 0x33333333);
aux6 = aux6 - ((aux6 >> 1) & 0x55555555);
aux6 = (aux6 & 0x33333333) + ((aux6 >> 2) & 0x33333333);
aux7 = aux7 - ((aux7 >> 1) & 0x55555555);
aux7 = (aux7 & 0x33333333) + ((aux7 >> 2) & 0x33333333);
bits += ((((aux1 + (aux1 >> 4)) & 0x0F0F0F0F) +
((aux2 + (aux2 >> 4)) & 0x0F0F0F0F) +
((aux3 + (aux3 >> 4)) & 0x0F0F0F0F) +
((aux4 + (aux4 >> 4)) & 0x0F0F0F0F) +
((aux5 + (aux5 >> 4)) & 0x0F0F0F0F) +
((aux6 + (aux6 >> 4)) & 0x0F0F0F0F) +
((aux7 + (aux7 >> 4)) & 0x0F0F0F0F))* 0x01010101) >> 24;
bits += ((((aux1 + (aux1 >> 4)) & 0x0F0F0F0F) * 0x01010101) >> 24) +
((((aux2 + (aux2 >> 4)) & 0x0F0F0F0F) * 0x01010101) >> 24) +
((((aux3 + (aux3 >> 4)) & 0x0F0F0F0F) * 0x01010101) >> 24) +
((((aux4 + (aux4 >> 4)) & 0x0F0F0F0F) * 0x01010101) >> 24);
}
/* Count the remaining bytes. */
p = (unsigned char*)p4;
@@ -273,12 +261,12 @@ void getbitCommand(redisClient *c) {
byte = bitoffset >> 3;
bit = 7 - (bitoffset & 0x7);
if (sdsEncodedObject(o)) {
if (byte < sdslen(o->ptr))
bitval = ((uint8_t*)o->ptr)[byte] & (1 << bit);
} else {
if (o->encoding != REDIS_ENCODING_RAW) {
if (byte < (size_t)ll2string(llbuf,sizeof(llbuf),(long)o->ptr))
bitval = llbuf[byte] & (1 << bit);
} else {
if (byte < sdslen(o->ptr))
bitval = ((uint8_t*)o->ptr)[byte] & (1 << bit);
}
addReply(c, bitval ? shared.cone : shared.czero);
@@ -360,7 +348,7 @@ void bitopCommand(redisClient *c) {
* can take a fast path that performs much better than the
* vanilla algorithm. */
j = 0;
if (minlen >= sizeof(unsigned long)*4 && numkeys <= 16) {
if (minlen && numkeys <= 16) {
unsigned long *lp[16];
unsigned long *lres = (unsigned long*) res;
src/blocked.c
-183
View File
@@ -1,183 +0,0 @@
/* blocked.c - generic support for blocking operations like BLPOP & WAIT.
*
* Copyright (c) 2009-2012, Salvatore Sanfilippo <antirez at gmail dot com>
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
* * Neither the name of Redis nor the names of its contributors may be used
* to endorse or promote products derived from this software without
* specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*
* ---------------------------------------------------------------------------
*
* API:
*
* getTimeoutFromObjectOrReply() is just an utility function to parse a
* timeout argument since blocking operations usually require a timeout.
*
* blockClient() set the REDIS_BLOCKED flag in the client, and set the
* specified block type 'btype' filed to one of REDIS_BLOCKED_* macros.
*
* unblockClient() unblocks the client doing the following:
* 1) It calls the btype-specific function to cleanup the state.
* 2) It unblocks the client by unsetting the REDIS_BLOCKED flag.
* 3) It puts the client into a list of just unblocked clients that are
* processed ASAP in the beforeSleep() event loop callback, so that
* if there is some query buffer to process, we do it. This is also
* required because otherwise there is no 'readable' event fired, we
* already read the pending commands. We also set the REDIS_UNBLOCKED
* flag to remember the client is in the unblocked_clients list.
*
* processUnblockedClients() is called inside the beforeSleep() function
* to process the query buffer from unblocked clients and remove the clients
* from the blocked_clients queue.
*
* replyToBlockedClientTimedOut() is called by the cron function when
* a client blocked reaches the specified timeout (if the timeout is set
* to 0, no timeout is processed).
* It usually just needs to send a reply to the client.
*
* When implementing a new type of blocking opeation, the implementation
* should modify unblockClient() and replyToBlockedClientTimedOut() in order
* to handle the btype-specific behavior of this two functions.
* If the blocking operation waits for certain keys to change state, the
* clusterRedirectBlockedClientIfNeeded() function should also be updated.
*/
#include "redis.h"
/* Get a timeout value from an object and store it into 'timeout'.
* The final timeout is always stored as milliseconds as a time where the
* timeout will expire, however the parsing is performed according to
* the 'unit' that can be seconds or milliseconds.
*
* Note that if the timeout is zero (usually from the point of view of
* commands API this means no timeout) the value stored into 'timeout'
* is zero. */
int getTimeoutFromObjectOrReply(redisClient *c, robj *object, mstime_t *timeout, int unit) {
long long tval;
if (getLongLongFromObjectOrReply(c,object,&tval,
"timeout is not an integer or out of range") != REDIS_OK)
return REDIS_ERR;
if (tval < 0) {
addReplyError(c,"timeout is negative");
return REDIS_ERR;
}
if (tval > 0) {
if (unit == UNIT_SECONDS) tval *= 1000;
tval += mstime();
}
*timeout = tval;
return REDIS_OK;
}
/* Block a client for the specific operation type. Once the REDIS_BLOCKED
* flag is set client query buffer is not longer processed, but accumulated,
* and will be processed when the client is unblocked. */
void blockClient(redisClient *c, int btype) {
c->flags |= REDIS_BLOCKED;
c->btype = btype;
server.bpop_blocked_clients++;
}
/* This function is called in the beforeSleep() function of the event loop
* in order to process the pending input buffer of clients that were
* unblocked after a blocking operation. */
void processUnblockedClients(void) {
listNode *ln;
redisClient *c;
while (listLength(server.unblocked_clients)) {
ln = listFirst(server.unblocked_clients);
redisAssert(ln != NULL);
c = ln->value;
listDelNode(server.unblocked_clients,ln);
c->flags &= ~REDIS_UNBLOCKED;
/* Process remaining data in the input buffer. */
if (c->querybuf && sdslen(c->querybuf) > 0) {
server.current_client = c;
processInputBuffer(c);
server.current_client = NULL;
}
}
}
/* Unblock a client calling the right function depending on the kind
* of operation the client is blocking for. */
void unblockClient(redisClient *c) {
if (c->btype == REDIS_BLOCKED_LIST) {
unblockClientWaitingData(c);
} else if (c->btype == REDIS_BLOCKED_WAIT) {
unblockClientWaitingReplicas(c);
} else {
redisPanic("Unknown btype in unblockClient().");
}
/* Clear the flags, and put the client in the unblocked list so that
* we'll process new commands in its query buffer ASAP. */
c->flags &= ~REDIS_BLOCKED;
c->flags |= REDIS_UNBLOCKED;
c->btype = REDIS_BLOCKED_NONE;
server.bpop_blocked_clients--;
listAddNodeTail(server.unblocked_clients,c);
}
/* This function gets called when a blocked client timed out in order to
* send it a reply of some kind. */
void replyToBlockedClientTimedOut(redisClient *c) {
if (c->btype == REDIS_BLOCKED_LIST) {
addReply(c,shared.nullmultibulk);
} else if (c->btype == REDIS_BLOCKED_WAIT) {
addReplyLongLong(c,replicationCountAcksByOffset(c->bpop.reploffset));
} else {
redisPanic("Unknown btype in replyToBlockedClientTimedOut().");
}
}
/* Mass-unblock clients because something changed in the instance that makes
* blocking no longer safe. For example clients blocked in list operations
* in an instance which turns from master to slave is unsafe, so this function
* is called when a master turns into a slave.
*
* The semantics is to send an -UNBLOCKED error to the client, disconnecting
* it at the same time. */
void disconnectAllBlockedClients(void) {
listNode *ln;
listIter li;
listRewind(server.clients,&li);
while((ln = listNext(&li))) {
redisClient *c = listNodeValue(ln);
if (c->flags & REDIS_BLOCKED) {
addReplySds(c,sdsnew(
"-UNBLOCKED force unblock from blocking operation, "
"instance state changed (master -> slave?)\r\n"));
unblockClient(c);
c->flags |= REDIS_CLOSE_AFTER_REPLY;
}
}
}
src/cluster.c
-5007
View File
File diff suppressed because it is too large Load Diff
src/cluster.h
-256
View File
@@ -1,256 +0,0 @@
#ifndef __REDIS_CLUSTER_H
#define __REDIS_CLUSTER_H
/*-----------------------------------------------------------------------------
* Redis cluster data structures, defines, exported API.
*----------------------------------------------------------------------------*/
#define REDIS_CLUSTER_SLOTS 16384
#define REDIS_CLUSTER_OK 0 /* Everything looks ok */
#define REDIS_CLUSTER_FAIL 1 /* The cluster can't work */
#define REDIS_CLUSTER_NAMELEN 40 /* sha1 hex length */
#define REDIS_CLUSTER_PORT_INCR 10000 /* Cluster port = baseport + PORT_INCR */
/* The following defines are amount of time, sometimes expressed as
* multiplicators of the node timeout value (when ending with MULT). */
#define REDIS_CLUSTER_DEFAULT_NODE_TIMEOUT 15000
#define REDIS_CLUSTER_DEFAULT_SLAVE_VALIDITY 10 /* Slave max data age factor. */
#define REDIS_CLUSTER_DEFAULT_REQUIRE_FULL_COVERAGE 1
#define REDIS_CLUSTER_FAIL_REPORT_VALIDITY_MULT 2 /* Fail report validity. */
#define REDIS_CLUSTER_FAIL_UNDO_TIME_MULT 2 /* Undo fail if master is back. */
#define REDIS_CLUSTER_FAIL_UNDO_TIME_ADD 10 /* Some additional time. */
#define REDIS_CLUSTER_FAILOVER_DELAY 5 /* Seconds */
#define REDIS_CLUSTER_DEFAULT_MIGRATION_BARRIER 1
#define REDIS_CLUSTER_MF_TIMEOUT 5000 /* Milliseconds to do a manual failover. */
#define REDIS_CLUSTER_MF_PAUSE_MULT 2 /* Master pause manual failover mult. */
/* Redirection errors returned by getNodeByQuery(). */
#define REDIS_CLUSTER_REDIR_NONE 0 /* Node can serve the request. */
#define REDIS_CLUSTER_REDIR_CROSS_SLOT 1 /* -CROSSSLOT request. */
#define REDIS_CLUSTER_REDIR_UNSTABLE 2 /* -TRYAGAIN redirection required */
#define REDIS_CLUSTER_REDIR_ASK 3 /* -ASK redirection required. */
#define REDIS_CLUSTER_REDIR_MOVED 4 /* -MOVED redirection required. */
#define REDIS_CLUSTER_REDIR_DOWN_STATE 5 /* -CLUSTERDOWN, global state. */
#define REDIS_CLUSTER_REDIR_DOWN_UNBOUND 6 /* -CLUSTERDOWN, unbound slot. */
struct clusterNode;
/* clusterLink encapsulates everything needed to talk with a remote node. */
typedef struct clusterLink {
mstime_t ctime; /* Link creation time */
int fd; /* TCP socket file descriptor */
sds sndbuf; /* Packet send buffer */
sds rcvbuf; /* Packet reception buffer */
struct clusterNode *node; /* Node related to this link if any, or NULL */
} clusterLink;
/* Cluster node flags and macros. */
#define REDIS_NODE_MASTER 1 /* The node is a master */
#define REDIS_NODE_SLAVE 2 /* The node is a slave */
#define REDIS_NODE_PFAIL 4 /* Failure? Need acknowledge */
#define REDIS_NODE_FAIL 8 /* The node is believed to be malfunctioning */
#define REDIS_NODE_MYSELF 16 /* This node is myself */
#define REDIS_NODE_HANDSHAKE 32 /* We have still to exchange the first ping */
#define REDIS_NODE_NOADDR 64 /* We don't know the address of this node */
#define REDIS_NODE_MEET 128 /* Send a MEET message to this node */
#define REDIS_NODE_PROMOTED 256 /* Master was a slave promoted by failover */
#define REDIS_NODE_NULL_NAME "\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000"
#define nodeIsMaster(n) ((n)->flags & REDIS_NODE_MASTER)
#define nodeIsSlave(n) ((n)->flags & REDIS_NODE_SLAVE)
#define nodeInHandshake(n) ((n)->flags & REDIS_NODE_HANDSHAKE)
#define nodeHasAddr(n) (!((n)->flags & REDIS_NODE_NOADDR))
#define nodeWithoutAddr(n) ((n)->flags & REDIS_NODE_NOADDR)
#define nodeTimedOut(n) ((n)->flags & REDIS_NODE_PFAIL)
#define nodeFailed(n) ((n)->flags & REDIS_NODE_FAIL)
/* Reasons why a slave is not able to failover. */
#define REDIS_CLUSTER_CANT_FAILOVER_NONE 0
#define REDIS_CLUSTER_CANT_FAILOVER_DATA_AGE 1
#define REDIS_CLUSTER_CANT_FAILOVER_WAITING_DELAY 2
#define REDIS_CLUSTER_CANT_FAILOVER_EXPIRED 3
#define REDIS_CLUSTER_CANT_FAILOVER_WAITING_VOTES 4
#define REDIS_CLUSTER_CANT_FAILOVER_RELOG_PERIOD (60*5) /* seconds. */
/* This structure represent elements of node->fail_reports. */
typedef struct clusterNodeFailReport {
struct clusterNode *node; /* Node reporting the failure condition. */
mstime_t time; /* Time of the last report from this node. */
} clusterNodeFailReport;
typedef struct clusterNode {
mstime_t ctime; /* Node object creation time. */
char name[REDIS_CLUSTER_NAMELEN]; /* Node name, hex string, sha1-size */
int flags; /* REDIS_NODE_... */
uint64_t configEpoch; /* Last configEpoch observed for this node */
unsigned char slots[REDIS_CLUSTER_SLOTS/8]; /* slots handled by this node */
int numslots; /* Number of slots handled by this node */
int numslaves; /* Number of slave nodes, if this is a master */
struct clusterNode **slaves; /* pointers to slave nodes */
struct clusterNode *slaveof; /* pointer to the master node */
mstime_t ping_sent; /* Unix time we sent latest ping */
mstime_t pong_received; /* Unix time we received the pong */
mstime_t fail_time; /* Unix time when FAIL flag was set */
mstime_t voted_time; /* Last time we voted for a slave of this master */
mstime_t repl_offset_time; /* Unix time we received offset for this node */
long long repl_offset; /* Last known repl offset for this node. */
char ip[REDIS_IP_STR_LEN]; /* Latest known IP address of this node */
int port; /* Latest known port of this node */
clusterLink *link; /* TCP/IP link with this node */
list *fail_reports; /* List of nodes signaling this as failing */
} clusterNode;
typedef struct clusterState {
clusterNode *myself; /* This node */
uint64_t currentEpoch;
int state; /* REDIS_CLUSTER_OK, REDIS_CLUSTER_FAIL, ... */
int size; /* Num of master nodes with at least one slot */
dict *nodes; /* Hash table of name -> clusterNode structures */
dict *nodes_black_list; /* Nodes we don't re-add for a few seconds. */
clusterNode *migrating_slots_to[REDIS_CLUSTER_SLOTS];
clusterNode *importing_slots_from[REDIS_CLUSTER_SLOTS];
clusterNode *slots[REDIS_CLUSTER_SLOTS];
zskiplist *slots_to_keys;
/* The following fields are used to take the slave state on elections. */
mstime_t failover_auth_time; /* Time of previous or next election. */
int failover_auth_count; /* Number of votes received so far. */
int failover_auth_sent; /* True if we already asked for votes. */
int failover_auth_rank; /* This slave rank for current auth request. */
uint64_t failover_auth_epoch; /* Epoch of the current election. */
int cant_failover_reason; /* Why a slave is currently not able to
failover. See the CANT_FAILOVER_* macros. */
/* Manual failover state in common. */
mstime_t mf_end; /* Manual failover time limit (ms unixtime).
It is zero if there is no MF in progress. */
/* Manual failover state of master. */
clusterNode *mf_slave; /* Slave performing the manual failover. */
/* Manual failover state of slave. */
long long mf_master_offset; /* Master offset the slave needs to start MF
or zero if stil not received. */
int mf_can_start; /* If non-zero signal that the manual failover
can start requesting masters vote. */
/* The followign fields are used by masters to take state on elections. */
uint64_t lastVoteEpoch; /* Epoch of the last vote granted. */
int todo_before_sleep; /* Things to do in clusterBeforeSleep(). */
long long stats_bus_messages_sent; /* Num of msg sent via cluster bus. */
long long stats_bus_messages_received; /* Num of msg rcvd via cluster bus.*/
} clusterState;
/* clusterState todo_before_sleep flags. */
#define CLUSTER_TODO_HANDLE_FAILOVER (1<<0)
#define CLUSTER_TODO_UPDATE_STATE (1<<1)
#define CLUSTER_TODO_SAVE_CONFIG (1<<2)
#define CLUSTER_TODO_FSYNC_CONFIG (1<<3)
/* Redis cluster messages header */
/* Note that the PING, PONG and MEET messages are actually the same exact
* kind of packet. PONG is the reply to ping, in the exact format as a PING,
* while MEET is a special PING that forces the receiver to add the sender
* as a node (if it is not already in the list). */
#define CLUSTERMSG_TYPE_PING 0 /* Ping */
#define CLUSTERMSG_TYPE_PONG 1 /* Pong (reply to Ping) */
#define CLUSTERMSG_TYPE_MEET 2 /* Meet "let's join" message */
#define CLUSTERMSG_TYPE_FAIL 3 /* Mark node xxx as failing */
#define CLUSTERMSG_TYPE_PUBLISH 4 /* Pub/Sub Publish propagation */
#define CLUSTERMSG_TYPE_FAILOVER_AUTH_REQUEST 5 /* May I failover? */
#define CLUSTERMSG_TYPE_FAILOVER_AUTH_ACK 6 /* Yes, you have my vote */
#define CLUSTERMSG_TYPE_UPDATE 7 /* Another node slots configuration */
#define CLUSTERMSG_TYPE_MFSTART 8 /* Pause clients for manual failover */
/* Initially we don't know our "name", but we'll find it once we connect
* to the first node, using the getsockname() function. Then we'll use this
* address for all the next messages. */
typedef struct {
char nodename[REDIS_CLUSTER_NAMELEN];
uint32_t ping_sent;
uint32_t pong_received;
char ip[REDIS_IP_STR_LEN]; /* IP address last time it was seen */
uint16_t port; /* port last time it was seen */
uint16_t flags; /* node->flags copy */
uint16_t notused1; /* Some room for future improvements. */
uint32_t notused2;
} clusterMsgDataGossip;
typedef struct {
char nodename[REDIS_CLUSTER_NAMELEN];
} clusterMsgDataFail;
typedef struct {
uint32_t channel_len;
uint32_t message_len;
/* We can't reclare bulk_data as bulk_data[] since this structure is
* nested. The 8 bytes are removed from the count during the message
* length computation. */
unsigned char bulk_data[8];
} clusterMsgDataPublish;
typedef struct {
uint64_t configEpoch; /* Config epoch of the specified instance. */
char nodename[REDIS_CLUSTER_NAMELEN]; /* Name of the slots owner. */
unsigned char slots[REDIS_CLUSTER_SLOTS/8]; /* Slots bitmap. */
} clusterMsgDataUpdate;
union clusterMsgData {
/* PING, MEET and PONG */
struct {
/* Array of N clusterMsgDataGossip structures */
clusterMsgDataGossip gossip[1];
} ping;
/* FAIL */
struct {
clusterMsgDataFail about;
} fail;
/* PUBLISH */
struct {
clusterMsgDataPublish msg;
} publish;
/* UPDATE */
struct {
clusterMsgDataUpdate nodecfg;
} update;
};
#define CLUSTER_PROTO_VER 0 /* Cluster bus protocol version. */
typedef struct {
char sig[4]; /* Siganture "RCmb" (Redis Cluster message bus). */
uint32_t totlen; /* Total length of this message */
uint16_t ver; /* Protocol version, currently set to 0. */
uint16_t notused0; /* 2 bytes not used. */
uint16_t type; /* Message type */
uint16_t count; /* Only used for some kind of messages. */
uint64_t currentEpoch; /* The epoch accordingly to the sending node. */
uint64_t configEpoch; /* The config epoch if it's a master, or the last
epoch advertised by its master if it is a
slave. */
uint64_t offset; /* Master replication offset if node is a master or
processed replication offset if node is a slave. */
char sender[REDIS_CLUSTER_NAMELEN]; /* Name of the sender node */
unsigned char myslots[REDIS_CLUSTER_SLOTS/8];
char slaveof[REDIS_CLUSTER_NAMELEN];
char notused1[32]; /* 32 bytes reserved for future usage. */
uint16_t port; /* Sender TCP base port */
uint16_t flags; /* Sender node flags */
unsigned char state; /* Cluster state from the POV of the sender */
unsigned char mflags[3]; /* Message flags: CLUSTERMSG_FLAG[012]_... */
union clusterMsgData data;
} clusterMsg;
#define CLUSTERMSG_MIN_LEN (sizeof(clusterMsg)-sizeof(union clusterMsgData))
/* Message flags better specify the packet content or are used to
* provide some information about the node state. */
#define CLUSTERMSG_FLAG0_PAUSED (1<<0) /* Master paused for manual failover. */
#define CLUSTERMSG_FLAG0_FORCEACK (1<<1) /* Give ACK to AUTH_REQUEST even if
master is up. */
/* ---------------------- API exported outside cluster.c -------------------- */
clusterNode *getNodeByQuery(redisClient *c, struct redisCommand *cmd, robj **argv, int argc, int *hashslot, int *ask);
int clusterRedirectBlockedClientIfNeeded(redisClient *c);
void clusterRedirectClient(redisClient *c, clusterNode *n, int hashslot, int error_code);
#endif /* __REDIS_CLUSTER_H */
src/config.c
+11 -88
View File
@@ -29,7 +29,6 @@
*/
#include "redis.h"
#include "cluster.h"
#include <fcntl.h>
#include <sys/stat.h>
@@ -82,7 +81,6 @@ void resetServerSaveParams(void) {
void loadServerConfigFromString(char *config) {
char *err = NULL;
int linenum = 0, totlines, i;
int slaveof_linenum = 0;
sds *lines;
lines = sdssplitlen(config,strlen(config),"\n",1,&totlines);
@@ -250,7 +248,6 @@ void loadServerConfigFromString(char *config) {
goto loaderr;
}
} else if (!strcasecmp(argv[0],"slaveof") && argc == 3) {
slaveof_linenum = linenum;
server.masterhost = sdsnew(argv[1]);
server.masterport = atoi(argv[2]);
server.repl_state = REDIS_REPL_CONNECT;
@@ -432,41 +429,6 @@ void loadServerConfigFromString(char *config) {
err = "Target command name already exists"; goto loaderr;
}
}
} else if (!strcasecmp(argv[0],"cluster-enabled") && argc == 2) {
if ((server.cluster_enabled = yesnotoi(argv[1])) == -1) {
err = "argument must be 'yes' or 'no'"; goto loaderr;
}
} else if (!strcasecmp(argv[0],"cluster-config-file") && argc == 2) {
zfree(server.cluster_configfile);
server.cluster_configfile = zstrdup(argv[1]);
} else if (!strcasecmp(argv[0],"cluster-require-full-coverage") &&
argc == 2)
{
if ((server.cluster_require_full_coverage = yesnotoi(argv[1])) == -1)
{
err = "argument must be 'yes' or 'no'"; goto loaderr;
}
} else if (!strcasecmp(argv[0],"cluster-node-timeout") && argc == 2) {
server.cluster_node_timeout = strtoll(argv[1],NULL,10);
if (server.cluster_node_timeout <= 0) {
err = "cluster node timeout must be 1 or greater"; goto loaderr;
}
} else if (!strcasecmp(argv[0],"cluster-migration-barrier")
&& argc == 2)
{
server.cluster_migration_barrier = atoi(argv[1]);
if (server.cluster_migration_barrier < 0) {
err = "cluster migration barrier must zero or positive";
goto loaderr;
}
} else if (!strcasecmp(argv[0],"cluster-slave-validity-factor")
&& argc == 2)
{
server.cluster_slave_validity_factor = atoi(argv[1]);
if (server.cluster_slave_validity_factor < 0) {
err = "cluster slave validity factor must be zero or positive";
goto loaderr;
}
} else if (!strcasecmp(argv[0],"lua-time-limit") && argc == 2) {
server.lua_time_limit = strtoll(argv[1],NULL,10);
} else if (!strcasecmp(argv[0],"slowlog-log-slower-than") &&
@@ -545,15 +507,6 @@ void loadServerConfigFromString(char *config) {
}
sdsfreesplitres(argv,argc);
}
/* Sanity checks. */
if (server.cluster_enabled && server.masterhost) {
linenum = slaveof_linenum;
i = linenum-1;
err = "slaveof directive not allowed in cluster mode";
goto loaderr;
}
sdsfreesplitres(lines,totlines);
return;
@@ -609,9 +562,8 @@ void loadServerConfig(char *filename, char *options) {
void configSetCommand(redisClient *c) {
robj *o;
long long ll;
int err;
redisAssertWithInfo(c,c->argv[2],sdsEncodedObject(c->argv[2]));
redisAssertWithInfo(c,c->argv[3],sdsEncodedObject(c->argv[3]));
redisAssertWithInfo(c,c->argv[2],c->argv[2]->encoding == REDIS_ENCODING_RAW);
redisAssertWithInfo(c,c->argv[2],c->argv[3]->encoding == REDIS_ENCODING_RAW);
o = c->argv[3];
if (!strcasecmp(c->argv[2]->ptr,"dbfilename")) {
@@ -629,8 +581,8 @@ void configSetCommand(redisClient *c) {
zfree(server.masterauth);
server.masterauth = ((char*)o->ptr)[0] ? zstrdup(o->ptr) : NULL;
} else if (!strcasecmp(c->argv[2]->ptr,"maxmemory")) {
ll = memtoll(o->ptr,&err);
if (err || ll < 0) goto badfmt;
if (getLongLongFromObject(o,&ll) == REDIS_ERR ||
ll < 0) goto badfmt;
server.maxmemory = ll;
if (server.maxmemory) {
if (server.maxmemory < zmalloc_used_memory()) {
@@ -857,6 +809,7 @@ void configSetCommand(redisClient *c) {
* whole configuration string or accept it all, even if a single
* error in a single client class is present. */
for (j = 0; j < vlen; j++) {
char *eptr;
long val;
if ((j % 4) == 0) {
@@ -865,8 +818,8 @@ void configSetCommand(redisClient *c) {
goto badfmt;
}
} else {
val = memtoll(v[j], &err);
if (err || val < 0) {
val = strtoll(v[j], &eptr, 10);
if (eptr[0] != '\0' || val < 0) {
sdsfreesplitres(v,vlen);
goto badfmt;
}
@@ -900,8 +853,7 @@ void configSetCommand(redisClient *c) {
if (getLongLongFromObject(o,&ll) == REDIS_ERR || ll <= 0) goto badfmt;
server.repl_timeout = ll;
} else if (!strcasecmp(c->argv[2]->ptr,"repl-backlog-size")) {
ll = memtoll(o->ptr,&err);
if (err || ll < 0) goto badfmt;
if (getLongLongFromObject(o,&ll) == REDIS_ERR || ll <= 0) goto badfmt;
resizeReplicationBacklog(ll);
} else if (!strcasecmp(c->argv[2]->ptr,"repl-backlog-ttl")) {
if (getLongLongFromObject(o,&ll) == REDIS_ERR || ll < 0) goto badfmt;
@@ -950,23 +902,6 @@ void configSetCommand(redisClient *c) {
ll < 0) goto badfmt;
server.repl_min_slaves_max_lag = ll;
refreshGoodSlavesCount();
} else if (!strcasecmp(c->argv[2]->ptr,"cluster-require-full-coverage")) {
int yn = yesnotoi(o->ptr);
if (yn == -1) goto badfmt;
server.cluster_require_full_coverage = yn;
} else if (!strcasecmp(c->argv[2]->ptr,"cluster-node-timeout")) {
if (getLongLongFromObject(o,&ll) == REDIS_ERR ||
ll <= 0) goto badfmt;
server.cluster_node_timeout = ll;
} else if (!strcasecmp(c->argv[2]->ptr,"cluster-migration-barrier")) {
if (getLongLongFromObject(o,&ll) == REDIS_ERR ||
ll < 0) goto badfmt;
server.cluster_migration_barrier = ll;
} else if (!strcasecmp(c->argv[2]->ptr,"cluster-slave-validity-factor")) {
if (getLongLongFromObject(o,&ll) == REDIS_ERR ||
ll < 0) goto badfmt;
server.cluster_slave_validity_factor = ll;
} else {
addReplyErrorFormat(c,"Unsupported CONFIG parameter: %s",
(char*)c->argv[2]->ptr);
@@ -1016,7 +951,7 @@ void configGetCommand(redisClient *c) {
char *pattern = o->ptr;
char buf[128];
int matches = 0;
redisAssertWithInfo(c,o,sdsEncodedObject(o));
redisAssertWithInfo(c,o,o->encoding == REDIS_ENCODING_RAW);
/* String values */
config_get_string_field("dbfilename",server.rdb_filename);
@@ -1071,14 +1006,9 @@ void configGetCommand(redisClient *c) {
config_get_numerical_field("min-slaves-to-write",server.repl_min_slaves_to_write);
config_get_numerical_field("min-slaves-max-lag",server.repl_min_slaves_max_lag);
config_get_numerical_field("hz",server.hz);
config_get_numerical_field("cluster-node-timeout",server.cluster_node_timeout);
config_get_numerical_field("cluster-migration-barrier",server.cluster_migration_barrier);
config_get_numerical_field("cluster-slave-validity-factor",server.cluster_slave_validity_factor);
config_get_numerical_field("repl-diskless-sync-delay",server.repl_diskless_sync_delay);
/* Bool (yes/no) values */
config_get_bool_field("cluster-require-full-coverage",
server.cluster_require_full_coverage);
config_get_bool_field("no-appendfsync-on-rewrite",
server.aof_no_fsync_on_rewrite);
config_get_bool_field("slave-serve-stale-data",
@@ -1572,9 +1502,8 @@ void rewriteConfigSlaveofOption(struct rewriteConfigState *state) {
sds line;
/* If this is a master, we want all the slaveof config options
* in the file to be removed. Note that if this is a cluster instance
* we don't want a slaveof directive inside redis.conf. */
if (server.cluster_enabled || server.masterhost == NULL) {
* in the file to be removed. */
if (server.masterhost == NULL) {
rewriteConfigMarkAsProcessed(state,"slaveof");
return;
}
@@ -1848,12 +1777,6 @@ int rewriteConfig(char *path) {
rewriteConfigNumericalOption(state,"auto-aof-rewrite-percentage",server.aof_rewrite_perc,REDIS_AOF_REWRITE_PERC);
rewriteConfigBytesOption(state,"auto-aof-rewrite-min-size",server.aof_rewrite_min_size,REDIS_AOF_REWRITE_MIN_SIZE);
rewriteConfigNumericalOption(state,"lua-time-limit",server.lua_time_limit,REDIS_LUA_TIME_LIMIT);
rewriteConfigYesNoOption(state,"cluster-enabled",server.cluster_enabled,0);
rewriteConfigStringOption(state,"cluster-config-file",server.cluster_configfile,REDIS_DEFAULT_CLUSTER_CONFIG_FILE);
rewriteConfigYesNoOption(state,"cluster-require-full-coverage",server.cluster_require_full_coverage,REDIS_CLUSTER_DEFAULT_REQUIRE_FULL_COVERAGE);
rewriteConfigNumericalOption(state,"cluster-node-timeout",server.cluster_node_timeout,REDIS_CLUSTER_DEFAULT_NODE_TIMEOUT);
rewriteConfigNumericalOption(state,"cluster-migration-barrier",server.cluster_migration_barrier,REDIS_CLUSTER_DEFAULT_MIGRATION_BARRIER);
rewriteConfigNumericalOption(state,"cluster-slave-validity-factor",server.cluster_slave_validity_factor,REDIS_CLUSTER_DEFAULT_SLAVE_VALIDITY);
rewriteConfigNumericalOption(state,"slowlog-log-slower-than",server.slowlog_log_slower_than,REDIS_SLOWLOG_LOG_SLOWER_THAN);
rewriteConfigNumericalOption(state,"latency-monitor-threshold",server.latency_monitor_threshold,REDIS_DEFAULT_LATENCY_MONITOR_THRESHOLD);
rewriteConfigNumericalOption(state,"slowlog-max-len",server.slowlog_max_len,REDIS_SLOWLOG_MAX_LEN);
src/config.h
+4 -7
View File
@@ -34,11 +34,6 @@
#include <AvailabilityMacros.h>
#endif
#ifdef __linux__
#include <linux/version.h>
#include <features.h>
#endif
/* Define redis_fstat to fstat or fstat64() */
#if defined(__APPLE__) && !defined(MAC_OS_X_VERSION_10_6)
#define redis_fstat fstat64
@@ -62,7 +57,7 @@
#endif
/* Test for backtrace() */
#if defined(__APPLE__) || (defined(__linux__) && defined(__GLIBC__))
#if defined(__APPLE__) || defined(__linux__)
#define HAVE_BACKTRACE 1
#endif
@@ -92,6 +87,8 @@
/* Define rdb_fsync_range to sync_file_range() on Linux, otherwise we use
* the plain fsync() call. */
#ifdef __linux__
#include <linux/version.h>
#include <features.h>
#if defined(__GLIBC__) && defined(__GLIBC_PREREQ)
#if (LINUX_VERSION_CODE >= 0x020611 && __GLIBC_PREREQ(2, 6))
#define HAVE_SYNC_FILE_RANGE 1
@@ -116,7 +113,7 @@
#define USE_SETPROCTITLE
#endif
#if ((defined __linux && defined(__GLIBC__)) || defined __APPLE__)
#if (defined __linux || defined __APPLE__)
#define USE_SETPROCTITLE
#define INIT_SETPROCTITLE_REPLACEMENT
void spt_init(int argc, char *argv[]);
src/crc16.c
-88
View File
@@ -1,88 +0,0 @@
#include "redis.h"
/*
* Copyright 2001-2010 Georges Menie (www.menie.org)
* Copyright 2010-2012 Salvatore Sanfilippo (adapted to Redis coding style)
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* * Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
* * Neither the name of the University of California, Berkeley nor the
* names of its contributors may be used to endorse or promote products
* derived from this software without specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND ANY
* EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
* WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
* DISCLAIMED. IN NO EVENT SHALL THE REGENTS AND CONTRIBUTORS BE LIABLE FOR ANY
* DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
* (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
* LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
* ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
* (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
* SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*/
/* CRC16 implementation according to CCITT standards.
*
* Note by @antirez: this is actually the XMODEM CRC 16 algorithm, using the
* following parameters:
*
* Name : "XMODEM", also known as "ZMODEM", "CRC-16/ACORN"
* Width : 16 bit
* Poly : 1021 (That is actually x^16 + x^12 + x^5 + 1)
* Initialization : 0000
* Reflect Input byte : False
* Reflect Output CRC : False
* Xor constant to output CRC : 0000
* Output for "123456789" : 31C3
*/
static const uint16_t crc16tab[256]= {
0x0000,0x1021,0x2042,0x3063,0x4084,0x50a5,0x60c6,0x70e7,
0x8108,0x9129,0xa14a,0xb16b,0xc18c,0xd1ad,0xe1ce,0xf1ef,
0x1231,0x0210,0x3273,0x2252,0x52b5,0x4294,0x72f7,0x62d6,
0x9339,0x8318,0xb37b,0xa35a,0xd3bd,0xc39c,0xf3ff,0xe3de,
0x2462,0x3443,0x0420,0x1401,0x64e6,0x74c7,0x44a4,0x5485,
0xa56a,0xb54b,0x8528,0x9509,0xe5ee,0xf5cf,0xc5ac,0xd58d,
0x3653,0x2672,0x1611,0x0630,0x76d7,0x66f6,0x5695,0x46b4,
0xb75b,0xa77a,0x9719,0x8738,0xf7df,0xe7fe,0xd79d,0xc7bc,
0x48c4,0x58e5,0x6886,0x78a7,0x0840,0x1861,0x2802,0x3823,
0xc9cc,0xd9ed,0xe98e,0xf9af,0x8948,0x9969,0xa90a,0xb92b,
0x5af5,0x4ad4,0x7ab7,0x6a96,0x1a71,0x0a50,0x3a33,0x2a12,
0xdbfd,0xcbdc,0xfbbf,0xeb9e,0x9b79,0x8b58,0xbb3b,0xab1a,
0x6ca6,0x7c87,0x4ce4,0x5cc5,0x2c22,0x3c03,0x0c60,0x1c41,
0xedae,0xfd8f,0xcdec,0xddcd,0xad2a,0xbd0b,0x8d68,0x9d49,
0x7e97,0x6eb6,0x5ed5,0x4ef4,0x3e13,0x2e32,0x1e51,0x0e70,
0xff9f,0xefbe,0xdfdd,0xcffc,0xbf1b,0xaf3a,0x9f59,0x8f78,
0x9188,0x81a9,0xb1ca,0xa1eb,0xd10c,0xc12d,0xf14e,0xe16f,
0x1080,0x00a1,0x30c2,0x20e3,0x5004,0x4025,0x7046,0x6067,
0x83b9,0x9398,0xa3fb,0xb3da,0xc33d,0xd31c,0xe37f,0xf35e,
0x02b1,0x1290,0x22f3,0x32d2,0x4235,0x5214,0x6277,0x7256,
0xb5ea,0xa5cb,0x95a8,0x8589,0xf56e,0xe54f,0xd52c,0xc50d,
0x34e2,0x24c3,0x14a0,0x0481,0x7466,0x6447,0x5424,0x4405,
0xa7db,0xb7fa,0x8799,0x97b8,0xe75f,0xf77e,0xc71d,0xd73c,
0x26d3,0x36f2,0x0691,0x16b0,0x6657,0x7676,0x4615,0x5634,
0xd94c,0xc96d,0xf90e,0xe92f,0x99c8,0x89e9,0xb98a,0xa9ab,
0x5844,0x4865,0x7806,0x6827,0x18c0,0x08e1,0x3882,0x28a3,
0xcb7d,0xdb5c,0xeb3f,0xfb1e,0x8bf9,0x9bd8,0xabbb,0xbb9a,
0x4a75,0x5a54,0x6a37,0x7a16,0x0af1,0x1ad0,0x2ab3,0x3a92,
0xfd2e,0xed0f,0xdd6c,0xcd4d,0xbdaa,0xad8b,0x9de8,0x8dc9,
0x7c26,0x6c07,0x5c64,0x4c45,0x3ca2,0x2c83,0x1ce0,0x0cc1,
0xef1f,0xff3e,0xcf5d,0xdf7c,0xaf9b,0xbfba,0x8fd9,0x9ff8,
0x6e17,0x7e36,0x4e55,0x5e74,0x2e93,0x3eb2,0x0ed1,0x1ef0
};
uint16_t crc16(const char *buf, int len) {
int counter;
uint16_t crc = 0;
for (counter = 0; counter < len; counter++)
crc = (crc<<8) ^ crc16tab[((crc>>8) ^ *buf++)&0x00FF];
return crc;
}
src/db.c
+30 -211
View File
@@ -28,14 +28,12 @@
*/
#include "redis.h"
#include "cluster.h"
#include <signal.h>
#include <ctype.h>
void slotToKeyAdd(robj *key);
void slotToKeyDel(robj *key);
void slotToKeyFlush(void);
void SlotToKeyAdd(robj *key);
void SlotToKeyDel(robj *key);
/*-----------------------------------------------------------------------------
* C-level DB API
@@ -50,7 +48,7 @@ robj *lookupKey(redisDb *db, robj *key) {
* Don't do it if we have a saving child, as this will trigger
* a copy on write madness. */
if (server.rdb_child_pid == -1 && server.aof_child_pid == -1)
val->lru = LRU_CLOCK();
val->lru = server.lruclock;
return val;
} else {
return NULL;
@@ -96,7 +94,6 @@ void dbAdd(redisDb *db, robj *key, robj *val) {
redisAssertWithInfo(NULL,key,retval == REDIS_OK);
if (val->type == REDIS_LIST) signalListAsReady(db, key);
if (server.cluster_enabled) slotToKeyAdd(key);
}
/* Overwrite an existing key with a new value. Incrementing the reference
@@ -105,7 +102,7 @@ void dbAdd(redisDb *db, robj *key, robj *val) {
*
* The program is aborted if the key was not already present. */
void dbOverwrite(redisDb *db, robj *key, robj *val) {
dictEntry *de = dictFind(db->dict,key->ptr);
struct dictEntry *de = dictFind(db->dict,key->ptr);
redisAssertWithInfo(NULL,key,de != NULL);
dictReplace(db->dict, key->ptr, val);
@@ -137,7 +134,7 @@ int dbExists(redisDb *db, robj *key) {
*
* The function makes sure to return keys not already expired. */
robj *dbRandomKey(redisDb *db) {
dictEntry *de;
struct dictEntry *de;
while(1) {
sds key;
@@ -164,7 +161,6 @@ int dbDelete(redisDb *db, robj *key) {
* the key, because it is shared with the main dictionary. */
if (dictSize(db->expires) > 0) dictDelete(db->expires,key->ptr);
if (dictDelete(db->dict,key->ptr) == DICT_OK) {
if (server.cluster_enabled) slotToKeyDel(key);
return 1;
} else {
return 0;
@@ -202,7 +198,7 @@ robj *dbUnshareStringValue(redisDb *db, robj *key, robj *o) {
redisAssert(o->type == REDIS_STRING);
if (o->refcount != 1 || o->encoding != REDIS_ENCODING_RAW) {
robj *decoded = getDecodedObject(o);
o = createRawStringObject(decoded->ptr, sdslen(decoded->ptr));
o = createStringObject(decoded->ptr, sdslen(decoded->ptr));
decrRefCount(decoded);
dbOverwrite(db,key,o);
}
@@ -218,7 +214,6 @@ long long emptyDb(void(callback)(void*)) {
dictEmpty(server.db[j].dict,callback);
dictEmpty(server.db[j].expires,callback);
}
if (server.cluster_enabled) slotToKeyFlush();
return removed;
}
@@ -255,7 +250,6 @@ void flushdbCommand(redisClient *c) {
signalFlushedDb(c->db->id);
dictEmpty(c->db->dict,NULL);
dictEmpty(c->db->expires,NULL);
if (server.cluster_enabled) slotToKeyFlush();
addReply(c,shared.ok);
}
@@ -309,10 +303,6 @@ void selectCommand(redisClient *c) {
"invalid DB index") != REDIS_OK)
return;
if (server.cluster_enabled && id != 0) {
addReplyError(c,"SELECT is not allowed in cluster mode");
return;
}
if (selectDb(c,id) == REDIS_ERR) {
addReplyError(c,"invalid DB index");
} else {
@@ -543,16 +533,16 @@ void scanGenericCommand(redisClient *c, robj *o, unsigned long cursor) {
/* Filter element if it does not match the pattern. */
if (!filter && use_pattern) {
if (sdsEncodedObject(kobj)) {
if (!stringmatchlen(pat, patlen, kobj->ptr, sdslen(kobj->ptr), 0))
filter = 1;
} else {
if (kobj->encoding == REDIS_ENCODING_INT) {
char buf[REDIS_LONGSTR_SIZE];
int len;
redisAssert(kobj->encoding == REDIS_ENCODING_INT);
len = ll2string(buf,sizeof(buf),(long)kobj->ptr);
if (!stringmatchlen(pat, patlen, buf, len, 0)) filter = 1;
} else {
if (!stringmatchlen(pat, patlen, kobj->ptr, sdslen(kobj->ptr), 0))
filter = 1;
}
}
@@ -712,11 +702,6 @@ void moveCommand(redisClient *c) {
int srcid;
long long dbid;
if (server.cluster_enabled) {
addReplyError(c,"MOVE is not allowed in cluster mode");
return;
}
/* Obtain source and target DB pointers */
src = c->db;
srcid = c->db->id;
@@ -978,8 +963,6 @@ void persistCommand(redisClient *c) {
* API to get key arguments from commands
* ---------------------------------------------------------------------------*/
/* The base case is to use the keys position as given in the command table
* (firstkey, lastkey, step). */
int *getKeysUsingCommandTable(struct redisCommand *cmd,robj **argv, int argc, int *numkeys) {
int j, i = 0, last, *keys;
REDIS_NOTUSED(argv);
@@ -999,65 +982,42 @@ int *getKeysUsingCommandTable(struct redisCommand *cmd,robj **argv, int argc, in
return keys;
}
/* Return all the arguments that are keys in the command passed via argc / argv.
*
* The command returns the positions of all the key arguments inside the array,
* so the actual return value is an heap allocated array of integers. The
* length of the array is returned by reference into *numkeys.
*
* 'cmd' must be point to the corresponding entry into the redisCommand
* table, according to the command name in argv[0].
*
* This function uses the command table if a command-specific helper function
* is not required, otherwise it calls the command-specific function. */
int *getKeysFromCommand(struct redisCommand *cmd, robj **argv, int argc, int *numkeys) {
int *getKeysFromCommand(struct redisCommand *cmd,robj **argv, int argc, int *numkeys, int flags) {
if (cmd->getkeys_proc) {
return cmd->getkeys_proc(cmd,argv,argc,numkeys);
return cmd->getkeys_proc(cmd,argv,argc,numkeys,flags);
} else {
return getKeysUsingCommandTable(cmd,argv,argc,numkeys);
}
}
/* Free the result of getKeysFromCommand. */
void getKeysFreeResult(int *result) {
zfree(result);
}
/* Helper function to extract keys from following commands:
* ZUNIONSTORE <destkey> <num-keys> <key> <key> ... <key> <options>
* ZINTERSTORE <destkey> <num-keys> <key> <key> ... <key> <options> */
int *zunionInterGetKeys(struct redisCommand *cmd, robj **argv, int argc, int *numkeys) {
int i, num, *keys;
REDIS_NOTUSED(cmd);
num = atoi(argv[2]->ptr);
/* Sanity check. Don't return any key if the command is going to
* reply with syntax error. */
if (num > (argc-3)) {
int *noPreloadGetKeys(struct redisCommand *cmd,robj **argv, int argc, int *numkeys, int flags) {
if (flags & REDIS_GETKEYS_PRELOAD) {
*numkeys = 0;
return NULL;
} else {
return getKeysUsingCommandTable(cmd,argv,argc,numkeys);
}
/* Keys in z{union,inter}store come from two places:
* argv[1] = storage key,
* argv[3...n] = keys to intersect */
keys = zmalloc(sizeof(int)*(num+1));
/* Add all key positions for argv[3...n] to keys[] */
for (i = 0; i < num; i++) keys[i] = 3+i;
/* Finally add the argv[1] key position (the storage key target). */
keys[num] = 1;
*numkeys = num+1; /* Total keys = {union,inter} keys + storage key */
return keys;
}
/* Helper function to extract keys from the following commands:
* EVAL <script> <num-keys> <key> <key> ... <key> [more stuff]
* EVALSHA <script> <num-keys> <key> <key> ... <key> [more stuff] */
int *evalGetKeys(struct redisCommand *cmd, robj **argv, int argc, int *numkeys) {
int *renameGetKeys(struct redisCommand *cmd,robj **argv, int argc, int *numkeys, int flags) {
if (flags & REDIS_GETKEYS_PRELOAD) {
int *keys = zmalloc(sizeof(int));
*numkeys = 1;
keys[0] = 1;
return keys;
} else {
return getKeysUsingCommandTable(cmd,argv,argc,numkeys);
}
}
int *zunionInterGetKeys(struct redisCommand *cmd,robj **argv, int argc, int *numkeys, int flags) {
int i, num, *keys;
REDIS_NOTUSED(cmd);
REDIS_NOTUSED(flags);
num = atoi(argv[2]->ptr);
/* Sanity check. Don't return any key if the command is going to
@@ -1066,149 +1026,8 @@ int *evalGetKeys(struct redisCommand *cmd, robj **argv, int argc, int *numkeys)
*numkeys = 0;
return NULL;
}
keys = zmalloc(sizeof(int)*num);
*numkeys = num;
/* Add all key positions for argv[3...n] to keys[] */
for (i = 0; i < num; i++) keys[i] = 3+i;
*numkeys = num;
return keys;
}
/* Helper function to extract keys from the SORT command.
*
* SORT <sort-key> ... STORE <store-key> ...
*
* The first argument of SORT is always a key, however a list of options
* follow in SQL-alike style. Here we parse just the minimum in order to
* correctly identify keys in the "STORE" option. */
int *sortGetKeys(struct redisCommand *cmd, robj **argv, int argc, int *numkeys) {
int i, j, num, *keys, found_store = 0;
REDIS_NOTUSED(cmd);
num = 0;
keys = zmalloc(sizeof(int)*2); /* Alloc 2 places for the worst case. */
keys[num++] = 1; /* <sort-key> is always present. */
/* Search for STORE option. By default we consider options to don't
* have arguments, so if we find an unknown option name we scan the
* next. However there are options with 1 or 2 arguments, so we
* provide a list here in order to skip the right number of args. */
struct {
char *name;
int skip;
} skiplist[] = {
{"limit", 2},
{"get", 1},
{"by", 1},
{NULL, 0} /* End of elements. */
};
for (i = 2; i < argc; i++) {
for (j = 0; skiplist[j].name != NULL; j++) {
if (!strcasecmp(argv[i]->ptr,skiplist[j].name)) {
i += skiplist[j].skip;
break;
} else if (!strcasecmp(argv[i]->ptr,"store") && i+1 < argc) {
/* Note: we don't increment "num" here and continue the loop
* to be sure to process the *last* "STORE" option if multiple
* ones are provided. This is same behavior as SORT. */
found_store = 1;
keys[num] = i+1; /* <store-key> */
break;
}
}
}
*numkeys = num + found_store;
return keys;
}
/* Slot to Key API. This is used by Redis Cluster in order to obtain in
* a fast way a key that belongs to a specified hash slot. This is useful
* while rehashing the cluster. */
void slotToKeyAdd(robj *key) {
unsigned int hashslot = keyHashSlot(key->ptr,sdslen(key->ptr));
zslInsert(server.cluster->slots_to_keys,hashslot,key);
incrRefCount(key);
}
void slotToKeyDel(robj *key) {
unsigned int hashslot = keyHashSlot(key->ptr,sdslen(key->ptr));
zslDelete(server.cluster->slots_to_keys,hashslot,key);
}
void slotToKeyFlush(void) {
zslFree(server.cluster->slots_to_keys);
server.cluster->slots_to_keys = zslCreate();
}
unsigned int getKeysInSlot(unsigned int hashslot, robj **keys, unsigned int count) {
zskiplistNode *n;
zrangespec range;
int j = 0;
range.min = range.max = hashslot;
range.minex = range.maxex = 0;
n = zslFirstInRange(server.cluster->slots_to_keys, &range);
while(n && n->score == hashslot && count--) {
keys[j++] = n->obj;
n = n->level[0].forward;
}
return j;
}
/* Remove all the keys in the specified hash slot.
* The number of removed items is returned. */
unsigned int delKeysInSlot(unsigned int hashslot) {
zskiplistNode *n;
zrangespec range;
int j = 0;
range.min = range.max = hashslot;
range.minex = range.maxex = 0;
n = zslFirstInRange(server.cluster->slots_to_keys, &range);
while(n && n->score == hashslot) {
robj *key = n->obj;
n = n->level[0].forward; /* Go to the next item before freeing it. */
incrRefCount(key); /* Protect the object while freeing it. */
dbDelete(&server.db[0],key);
decrRefCount(key);
j++;
}
return j;
}
unsigned int countKeysInSlot(unsigned int hashslot) {
zskiplist *zsl = server.cluster->slots_to_keys;
zskiplistNode *zn;
zrangespec range;
int rank, count = 0;
range.min = range.max = hashslot;
range.minex = range.maxex = 0;
/* Find first element in range */
zn = zslFirstInRange(zsl, &range);
/* Use rank of first element, if any, to determine preliminary count */
if (zn != NULL) {
rank = zslGetRank(zsl, zn->score, zn->obj);
count = (zsl->length - (rank - 1));
/* Find last element in range */
zn = zslLastInRange(zsl, &range);
/* Use rank of last element, if any, to determine the actual count */
if (zn != NULL) {
rank = zslGetRank(zsl, zn->score, zn->obj);
count -= (zsl->length - rank);
}
}
return count;
}
src/debug.c
+7 -5
View File
@@ -298,10 +298,10 @@ void debugCommand(redisClient *c) {
addReplyStatusFormat(c,
"Value at:%p refcount:%d "
"encoding:%s serializedlength:%lld "
"lru:%d lru_seconds_idle:%llu",
"lru:%d lru_seconds_idle:%lu",
(void*)val, val->refcount,
strenc, (long long) rdbSavedObjectLen(val),
val->lru, estimateObjectIdleTime(val)/1000);
val->lru, estimateObjectIdleTime(val));
} else if (!strcasecmp(c->argv[1]->ptr,"sdslen") && c->argc == 3) {
dictEntry *de;
robj *val;
@@ -314,7 +314,7 @@ void debugCommand(redisClient *c) {
val = dictGetVal(de);
key = dictGetKey(de);
if (val->type != REDIS_STRING || !sdsEncodedObject(val)) {
if (val->type != REDIS_STRING || val->encoding != REDIS_ENCODING_RAW) {
addReplyError(c,"Not an sds encoded string.");
} else {
addReplyStatusFormat(c,
@@ -412,7 +412,9 @@ void _redisAssertPrintClientInfo(redisClient *c) {
char buf[128];
char *arg;
if (c->argv[j]->type == REDIS_STRING && sdsEncodedObject(c->argv[j])) {
if (c->argv[j]->type == REDIS_STRING &&
c->argv[j]->encoding == REDIS_ENCODING_RAW)
{
arg = (char*) c->argv[j]->ptr;
} else {
snprintf(buf,sizeof(buf),"Object type: %d, encoding: %d",
@@ -428,7 +430,7 @@ void redisLogObjectDebugInfo(robj *o) {
redisLog(REDIS_WARNING,"Object type: %d", o->type);
redisLog(REDIS_WARNING,"Object encoding: %d", o->encoding);
redisLog(REDIS_WARNING,"Object refcount: %d", o->refcount);
if (o->type == REDIS_STRING && sdsEncodedObject(o)) {
if (o->type == REDIS_STRING && o->encoding == REDIS_ENCODING_RAW) {
redisLog(REDIS_WARNING,"Object raw string len: %zu", sdslen(o->ptr));
if (sdslen(o->ptr) < 4096) {
sds repr = sdscatrepr(sdsempty(),o->ptr,sdslen(o->ptr));
src/dict.c
+14 -121
View File
@@ -211,9 +211,6 @@ int dictExpand(dict *d, unsigned long size)
if (dictIsRehashing(d) || d->ht[0].used > size)
return DICT_ERR;
/* Rehashing to the same table size is not useful. */
if (realsize == d->ht[0].size) return DICT_ERR;
/* Allocate the new hash table and initialize all pointers to NULL */
n.size = realsize;
n.sizemask = realsize-1;
@@ -235,27 +232,27 @@ int dictExpand(dict *d, unsigned long size)
/* Performs N steps of incremental rehashing. Returns 1 if there are still
* keys to move from the old to the new hash table, otherwise 0 is returned.
*
* Note that a rehashing step consists in moving a bucket (that may have more
* than one key as we use chaining) from the old to the new hash table, however
* since part of the hash table may be composed of empty spaces, it is not
* guaranteed that this function will rehash even a single bucket, since it
* will visit at max N*10 empty buckets in total, otherwise the amount of
* work it does would be unbound and the function may block for a long time. */
* than one key as we use chaining) from the old to the new hash table. */
int dictRehash(dict *d, int n) {
int empty_visits = n*10; /* Max number of empty buckets to visit. */
if (!dictIsRehashing(d)) return 0;
while(n-- && d->ht[0].used != 0) {
while(n--) {
dictEntry *de, *nextde;
/* Check if we already rehashed the whole table... */
if (d->ht[0].used == 0) {
zfree(d->ht[0].table);
d->ht[0] = d->ht[1];
_dictReset(&d->ht[1]);
d->rehashidx = -1;
return 0;
}
/* Note that rehashidx can't overflow as we are sure there are more
* elements because ht[0].used != 0 */
assert(d->ht[0].size > (unsigned long)d->rehashidx);
while(d->ht[0].table[d->rehashidx] == NULL) {
d->rehashidx++;
if (--empty_visits == 0) return 1;
}
while(d->ht[0].table[d->rehashidx] == NULL) d->rehashidx++;
de = d->ht[0].table[d->rehashidx];
/* Move all the keys in this bucket from the old to the new hash HT */
while(de) {
@@ -273,17 +270,6 @@ int dictRehash(dict *d, int n) {
d->ht[0].table[d->rehashidx] = NULL;
d->rehashidx++;
}
/* Check if we already rehashed the whole table... */
if (d->ht[0].used == 0) {
zfree(d->ht[0].table);
d->ht[0] = d->ht[1];
_dictReset(&d->ht[1]);
d->rehashidx = -1;
return 0;
}
/* More to rehash... */
return 1;
}
@@ -332,7 +318,7 @@ int dictAdd(dict *d, void *key, void *val)
* a value returns the dictEntry structure to the user, that will make
* sure to fill the value field as he wishes.
*
* This function is also directly exposed to the user API to be called
* This function is also directly exposed to user API to be called
* mainly in order to store non-pointers inside the hash value, example:
*
* entry = dictAddRaw(dict,mykey);
@@ -630,11 +616,7 @@ dictEntry *dictGetRandomKey(dict *d)
if (dictIsRehashing(d)) _dictRehashStep(d);
if (dictIsRehashing(d)) {
do {
/* We are sure there are no elements in indexes from 0
* to rehashidx-1 */
h = d->rehashidx + (random() % (d->ht[0].size +
d->ht[1].size -
d->rehashidx));
h = random() % (d->ht[0].size+d->ht[1].size);
he = (h >= d->ht[0].size) ? d->ht[1].table[h - d->ht[0].size] :
d->ht[0].table[h];
} while(he == NULL);
@@ -661,95 +643,6 @@ dictEntry *dictGetRandomKey(dict *d)
return he;
}
/* This function samples the dictionary to return a few keys from random
* locations.
*
* It does not guarantee to return all the keys specified in 'count', nor
* it does guarantee to return non-duplicated elements, however it will make
* some effort to do both things.
*
* Returned pointers to hash table entries are stored into 'des' that
* points to an array of dictEntry pointers. The array must have room for
* at least 'count' elements, that is the argument we pass to the function
* to tell how many random elements we need.
*
* The function returns the number of items stored into 'des', that may
* be less than 'count' if the hash table has less than 'count' elements
* inside, or if not enough elements were found in a reasonable amount of
* steps.
*
* Note that this function is not suitable when you need a good distribution
* of the returned items, but only when you need to "sample" a given number
* of continuous elements to run some kind of algorithm or to produce
* statistics. However the function is much faster than dictGetRandomKey()
* at producing N elements. */
unsigned int dictGetSomeKeys(dict *d, dictEntry **des, unsigned int count) {
unsigned int j; /* internal hash table id, 0 or 1. */
unsigned int tables; /* 1 or 2 tables? */
unsigned int stored = 0, maxsizemask;
unsigned int maxsteps;
if (dictSize(d) < count) count = dictSize(d);
maxsteps = count*10;
/* Try to do a rehashing work proportional to 'count'. */
for (j = 0; j < count; j++) {
if (dictIsRehashing(d))
_dictRehashStep(d);
else
break;
}
tables = dictIsRehashing(d) ? 2 : 1;
maxsizemask = d->ht[0].sizemask;
if (tables > 1 && maxsizemask < d->ht[1].sizemask)
maxsizemask = d->ht[1].sizemask;
/* Pick a random point inside the larger table. */
unsigned int i = random() & maxsizemask;
unsigned int emptylen = 0; /* Continuous empty entries so far. */
while(stored < count && maxsteps--) {
for (j = 0; j < tables; j++) {
/* Invariant of the dict.c rehashing: up to the indexes already
* visited in ht[0] during the rehashing, there are no populated
* buckets, so we can skip ht[0] for indexes between 0 and idx-1. */
if (tables == 2 && j == 0 && i < (unsigned int) d->rehashidx) {
/* Moreover, if we are currently out of range in the second
* table, there will be no elements in both tables up to
* the current rehashing index, so we jump if possible.
* (this happens when going from big to small table). */
if (i >= d->ht[1].size) i = d->rehashidx;
continue;
}
if (i >= d->ht[j].size) continue; /* Out of range for this table. */
dictEntry *he = d->ht[j].table[i];
/* Count contiguous empty buckets, and jump to other
* locations if they reach 'count' (with a minimum of 5). */
if (he == NULL) {
emptylen++;
if (emptylen >= 5 && emptylen > count) {
i = random() & maxsizemask;
emptylen = 0;
}
} else {
emptylen = 0;
while (he) {
/* Collect all the elements of the buckets found non
* empty while iterating. */
*des = he;
des++;
he = he->next;
stored++;
if (stored == count) return stored;
}
}
}
i = (i+1) & maxsizemask;
}
return stored;
}
/* Function to reverse bits. Algorithm from:
* http://graphics.stanford.edu/~seander/bithacks.html#ReverseParallel */
static unsigned long rev(unsigned long v) {
src/dict.h
-1
View File
@@ -164,7 +164,6 @@ dictIterator *dictGetSafeIterator(dict *d);
dictEntry *dictNext(dictIterator *iter);
void dictReleaseIterator(dictIterator *iter);
dictEntry *dictGetRandomKey(dict *d);
unsigned int dictGetSomeKeys(dict *d, dictEntry **des, unsigned int count);
void dictPrintStats(dict *d);
unsigned int dictGenHashFunction(const void *key, int len);
unsigned int dictGenCaseHashFunction(const unsigned char *buf, int len);
src/endianconv.h
-10
View File
@@ -61,14 +61,4 @@ uint64_t intrev64(uint64_t v);
#define intrev64ifbe(v) intrev64(v)
#endif
/* The functions htonu64() and ntohu64() convert the specified value to
* network byte ordering and back. In big endian systems they are no-ops. */
#if (BYTE_ORDER == BIG_ENDIAN)
#define htonu64(v) (v)
#define ntohu64(v) (v)
#else
#define htonu64(v) intrev64(v)
#define ntohu64(v) intrev64(v)
#endif
#endif
src/latency.c
+1 -11
View File
@@ -228,7 +228,6 @@ sds createLatencyReport(void) {
int advise_write_load_info = 0; /* Print info about AOF and write load. */
int advise_hz = 0; /* Use higher HZ. */
int advise_large_objects = 0; /* Deletion of large objects. */
int advise_mass_eviction = 0; /* Avoid mass eviction of keys. */
int advise_relax_fsync_policy = 0; /* appendfsync always is slow. */
int advise_disable_thp = 0; /* AnonHugePages detected. */
int advices = 0;
@@ -365,13 +364,8 @@ sds createLatencyReport(void) {
}
/* Eviction cycle. */
if (!strcasecmp(event,"eviction-del")) {
advise_large_objects = 1;
advices++;
}
if (!strcasecmp(event,"eviction-cycle")) {
advise_mass_eviction = 1;
advise_large_objects = 1;
advices++;
}
@@ -458,10 +452,6 @@ sds createLatencyReport(void) {
report = sdscat(report,"- Deleting, expiring or evicting (because of maxmemory policy) large objects is a blocking operation. If you have very large objects that are often deleted, expired, or evicted, try to fragment those objects into multiple smaller objects.\n");
}
if (advise_mass_eviction) {
report = sdscat(report,"- Sudden changes to the 'maxmemory' setting via 'CONFIG SET', or allocation of large objects via sets or sorted sets intersections, STORE option of SORT, Redis Cluster large keys migrations (RESTORE command), may create sudden memory pressure forcing the server to block trying to evict keys. \n");
}
if (advise_disable_thp) {
report = sdscat(report,"- I detected a non zero amount of anonymous huge pages used by your process. This creates very serious latency events in different conditions, especially when Redis is persisting on disk. To disable THP support use the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled', make sure to also add it into /etc/rc.local so that the command will be executed again after a reboot. Note that even if you have already disabled THP, you still need to restart the Redis process to get rid of the huge pages already created.\n");
}
src/latency.h
-4
View File
@@ -86,8 +86,4 @@ int THPIsEnabled(void);
(var) >= server.latency_monitor_threshold) \
latencyAddSample((event),(var));
/* Remove time from a nested event. */
#define latencyRemoveNestedEvent(event_var,nested_var) \
event_var += nested_var;
#endif /* __LATENCY_H */
src/migrate.c
+247
View File
@@ -0,0 +1,247 @@
#include "redis.h"
#include "endianconv.h"
/* -----------------------------------------------------------------------------
* DUMP, RESTORE and MIGRATE commands
* -------------------------------------------------------------------------- */
/* Generates a DUMP-format representation of the object 'o', adding it to the
* io stream pointed by 'rio'. This function can't fail. */
void createDumpPayload(rio *payload, robj *o) {
unsigned char buf[2];
uint64_t crc;
/* Serialize the object in a RDB-like format. It consist of an object type
* byte followed by the serialized object. This is understood by RESTORE. */
rioInitWithBuffer(payload,sdsempty());
redisAssert(rdbSaveObjectType(payload,o));
redisAssert(rdbSaveObject(payload,o));
/* Write the footer, this is how it looks like:
* ----------------+---------------------+---------------+
* ... RDB payload | 2 bytes RDB version | 8 bytes CRC64 |
* ----------------+---------------------+---------------+
* RDB version and CRC are both in little endian.
*/
/* RDB version */
buf[0] = REDIS_RDB_VERSION & 0xff;
buf[1] = (REDIS_RDB_VERSION >> 8) & 0xff;
payload->io.buffer.ptr = sdscatlen(payload->io.buffer.ptr,buf,2);
/* CRC64 */
crc = crc64(0,(unsigned char*)payload->io.buffer.ptr,
sdslen(payload->io.buffer.ptr));
memrev64ifbe(&crc);
payload->io.buffer.ptr = sdscatlen(payload->io.buffer.ptr,&crc,8);
}
/* Verify that the RDB version of the dump payload matches the one of this Redis
* instance and that the checksum is ok.
* If the DUMP payload looks valid REDIS_OK is returned, otherwise REDIS_ERR
* is returned. */
int verifyDumpPayload(unsigned char *p, size_t len) {
unsigned char *footer;
uint16_t rdbver;
uint64_t crc;
/* At least 2 bytes of RDB version and 8 of CRC64 should be present. */
if (len < 10) return REDIS_ERR;
footer = p+(len-10);
/* Verify RDB version */
rdbver = (footer[1] << 8) | footer[0];
if (rdbver != REDIS_RDB_VERSION) return REDIS_ERR;
/* Verify CRC64 */
crc = crc64(0,p,len-8);
memrev64ifbe(&crc);
return (memcmp(&crc,footer+2,8) == 0) ? REDIS_OK : REDIS_ERR;
}
/* DUMP keyname
* DUMP is actually not used by Redis Cluster but it is the obvious
* complement of RESTORE and can be useful for different applications. */
void dumpCommand(redisClient *c) {
robj *o, *dumpobj;
rio payload;
/* Check if the key is here. */
if ((o = lookupKeyRead(c->db,c->argv[1])) == NULL) {
addReply(c,shared.nullbulk);
return;
}
/* Create the DUMP encoded representation. */
createDumpPayload(&payload,o);
/* Transfer to the client */
dumpobj = createObject(REDIS_STRING,payload.io.buffer.ptr);
addReplyBulk(c,dumpobj);
decrRefCount(dumpobj);
return;
}
/* RESTORE key ttl serialized-value */
void restoreCommand(redisClient *c) {
long long ttl;
rio payload;
int type;
robj *obj;
/* Make sure this key does not already exist here... */
if (lookupKeyWrite(c->db,c->argv[1]) != NULL) {
addReplyError(c,"Target key name is busy.");
return;
}
/* Check if the TTL value makes sense */
if (getLongLongFromObjectOrReply(c,c->argv[2],&ttl,NULL) != REDIS_OK) {
return;
} else if (ttl < 0) {
addReplyError(c,"Invalid TTL value, must be >= 0");
return;
}
/* Verify RDB version and data checksum. */
if (verifyDumpPayload(c->argv[3]->ptr,sdslen(c->argv[3]->ptr)) == REDIS_ERR)
{
addReplyError(c,"DUMP payload version or checksum are wrong");
return;
}
rioInitWithBuffer(&payload,c->argv[3]->ptr);
if (((type = rdbLoadObjectType(&payload)) == -1) ||
((obj = rdbLoadObject(type,&payload)) == NULL))
{
addReplyError(c,"Bad data format");
return;
}
/* Create the key and set the TTL if any */
dbAdd(c->db,c->argv[1],obj);
if (ttl) setExpire(c->db,c->argv[1],mstime()+ttl);
signalModifiedKey(c->db,c->argv[1]);
addReply(c,shared.ok);
server.dirty++;
}
/* MIGRATE host port key dbid timeout */
void migrateCommand(redisClient *c) {
int fd;
long timeout;
long dbid;
long long ttl = 0, expireat;
robj *o;
rio cmd, payload;
/* Sanity check */
if (getLongFromObjectOrReply(c,c->argv[5],&timeout,NULL) != REDIS_OK)
return;
if (getLongFromObjectOrReply(c,c->argv[4],&dbid,NULL) != REDIS_OK)
return;
if (timeout <= 0) timeout = 1;
/* Check if the key is here. If not we reply with success as there is
* nothing to migrate (for instance the key expired in the meantime), but
* we include such information in the reply string. */
if ((o = lookupKeyRead(c->db,c->argv[3])) == NULL) {
addReplySds(c,sdsnew("+NOKEY\r\n"));
return;
}
/* Connect */
fd = anetTcpNonBlockConnect(server.neterr,c->argv[1]->ptr,
atoi(c->argv[2]->ptr));
if (fd == -1) {
addReplyErrorFormat(c,"Can't connect to target node: %s",
server.neterr);
return;
}
if ((aeWait(fd,AE_WRITABLE,timeout*1000) & AE_WRITABLE) == 0) {
addReplySds(c,sdsnew("-IOERR error or timeout connecting to the client\r\n"));
return;
}
/* Create RESTORE payload and generate the protocol to call the command. */
rioInitWithBuffer(&cmd,sdsempty());
redisAssertWithInfo(c,NULL,rioWriteBulkCount(&cmd,'*',2));
redisAssertWithInfo(c,NULL,rioWriteBulkString(&cmd,"SELECT",6));
redisAssertWithInfo(c,NULL,rioWriteBulkLongLong(&cmd,dbid));
expireat = getExpire(c->db,c->argv[3]);
if (expireat != -1) {
ttl = expireat-mstime();
if (ttl < 1) ttl = 1;
}
redisAssertWithInfo(c,NULL,rioWriteBulkCount(&cmd,'*',4));
redisAssertWithInfo(c,NULL,rioWriteBulkString(&cmd,"RESTORE",7));
redisAssertWithInfo(c,NULL,c->argv[3]->encoding == REDIS_ENCODING_RAW);
redisAssertWithInfo(c,NULL,rioWriteBulkString(&cmd,c->argv[3]->ptr,sdslen(c->argv[3]->ptr)));
redisAssertWithInfo(c,NULL,rioWriteBulkLongLong(&cmd,ttl));
/* Finally the last argument that is the serailized object payload
* in the DUMP format. */
createDumpPayload(&payload,o);
redisAssertWithInfo(c,NULL,rioWriteBulkString(&cmd,payload.io.buffer.ptr,
sdslen(payload.io.buffer.ptr)));
sdsfree(payload.io.buffer.ptr);
/* Tranfer the query to the other node in 64K chunks. */
{
sds buf = cmd.io.buffer.ptr;
size_t pos = 0, towrite;
int nwritten = 0;
while ((towrite = sdslen(buf)-pos) > 0) {
towrite = (towrite > (64*1024) ? (64*1024) : towrite);
nwritten = syncWrite(fd,buf+pos,towrite,timeout);
if (nwritten != (signed)towrite) goto socket_wr_err;
pos += nwritten;
}
}
/* Read back the reply. */
{
char buf1[1024];
char buf2[1024];
/* Read the two replies */
if (syncReadLine(fd, buf1, sizeof(buf1), timeout) <= 0)
goto socket_rd_err;
if (syncReadLine(fd, buf2, sizeof(buf2), timeout) <= 0)
goto socket_rd_err;
if (buf1[0] == '-' || buf2[0] == '-') {
addReplyErrorFormat(c,"Target instance replied with error: %s",
(buf1[0] == '-') ? buf1+1 : buf2+1);
} else {
robj *aux;
dbDelete(c->db,c->argv[3]);
signalModifiedKey(c->db,c->argv[3]);
addReply(c,shared.ok);
server.dirty++;
/* Translate MIGRATE as DEL for replication/AOF. */
aux = createStringObject("DEL",3);
rewriteClientCommandVector(c,2,aux,c->argv[3]);
decrRefCount(aux);
}
}
sdsfree(cmd.io.buffer.ptr);
close(fd);
return;
socket_wr_err:
addReplySds(c,sdsnew("-IOERR error or timeout writing to target instance\r\n"));
sdsfree(cmd.io.buffer.ptr);
close(fd);
return;
socket_rd_err:
addReplySds(c,sdsnew("-IOERR error or timeout reading from target node\r\n"));
sdsfree(cmd.io.buffer.ptr);
close(fd);
return;
}
src/networking.c
+30 -146
View File
@@ -41,17 +41,6 @@ size_t zmalloc_size_sds(sds s) {
return zmalloc_size(s-sizeof(struct sdshdr));
}
/* Return the amount of memory used by the sds string at object->ptr
* for a string object. */
size_t getStringObjectSdsUsedMemory(robj *o) {
redisAssertWithInfo(NULL,o,o->type == REDIS_STRING);
switch(o->encoding) {
case REDIS_ENCODING_RAW: return zmalloc_size_sds(o->ptr);
case REDIS_ENCODING_EMBSTR: return sdslen(o->ptr);
default: return 0; /* Just integer encoding for now. */
}
}
void *dupClientReplyValue(void *o) {
incrRefCount((robj*)o);
return o;
@@ -105,18 +94,15 @@ redisClient *createClient(int fd) {
c->repl_ack_off = 0;
c->repl_ack_time = 0;
c->slave_listening_port = 0;
c->slave_capa = SLAVE_CAPA_NONE;
c->reply = listCreate();
c->reply_bytes = 0;
c->obuf_soft_limit_reached_time = 0;
listSetFreeMethod(c->reply,decrRefCountVoid);
listSetDupMethod(c->reply,dupClientReplyValue);
c->btype = REDIS_BLOCKED_NONE;
c->bpop.timeout = 0;
c->bpop.keys = dictCreate(&setDictType,NULL);
c->bpop.timeout = 0;
c->bpop.target = NULL;
c->bpop.numreplicas = 0;
c->bpop.reploffset = 0;
c->woff = 0;
c->watched_keys = listCreate();
c->pubsub_channels = dictCreate(&setDictType,NULL);
c->pubsub_patterns = listCreate();
@@ -135,49 +121,23 @@ redisClient *createClient(int fd) {
* returns REDIS_OK, and make sure to install the write handler in our event
* loop so that when the socket is writable new data gets written.
*
* If the client should not receive new data, because it is a fake client
* (used to load AOF in memory), a master or because the setup of the write
* handler failed, the function returns REDIS_ERR.
*
* The function may return REDIS_OK without actually installing the write
* event handler in the following cases:
*
* 1) The event handler should already be installed since the output buffer
* already contained something.
* 2) The client is a slave but not yet online, so we want to just accumulate
* writes in the buffer but not actually sending them yet.
* If the client should not receive new data, because it is a fake client,
* a master, a slave not yet online, or because the setup of the write handler
* failed, the function returns REDIS_ERR.
*
* Typically gets called every time a reply is built, before adding more
* data to the clients output buffers. If the function returns REDIS_ERR no
* data should be appended to the output buffers. */
int prepareClientToWrite(redisClient *c) {
/* If it's the Lua client we always return ok without installing any
* handler since there is no socket at all. */
if (c->flags & REDIS_LUA_CLIENT) return REDIS_OK;
/* Masters don't receive replies, unless REDIS_MASTER_FORCE_REPLY flag
* is set. */
if ((c->flags & REDIS_MASTER) &&
!(c->flags & REDIS_MASTER_FORCE_REPLY)) return REDIS_ERR;
if (c->fd <= 0) return REDIS_ERR; /* Fake client for AOF loading. */
/* Only install the handler if not already installed and, in case of
* slaves, if the client can actually receive writes. */
if (c->fd <= 0) return REDIS_ERR; /* Fake client */
if (c->bufpos == 0 && listLength(c->reply) == 0 &&
(c->replstate == REDIS_REPL_NONE ||
(c->replstate == REDIS_REPL_ONLINE && !c->repl_put_online_on_ack)))
{
/* Try to install the write handler. */
if (aeCreateFileEvent(server.el, c->fd, AE_WRITABLE,
sendReplyToClient, c) == AE_ERR)
{
freeClientAsync(c);
return REDIS_ERR;
}
}
/* Authorize the caller to queue in the output buffer of this client. */
c->replstate == REDIS_REPL_ONLINE) && !c->repl_put_online_on_ack &&
aeCreateFileEvent(server.el, c->fd, AE_WRITABLE,
sendReplyToClient, c) == AE_ERR) return REDIS_ERR;
return REDIS_OK;
}
@@ -226,13 +186,12 @@ void _addReplyObjectToList(redisClient *c, robj *o) {
if (listLength(c->reply) == 0) {
incrRefCount(o);
listAddNodeTail(c->reply,o);
c->reply_bytes += getStringObjectSdsUsedMemory(o);
c->reply_bytes += zmalloc_size_sds(o->ptr);
} else {
tail = listNodeValue(listLast(c->reply));
/* Append to this object when possible. */
if (tail->ptr != NULL &&
tail->encoding == REDIS_ENCODING_RAW &&
sdslen(tail->ptr)+sdslen(o->ptr) <= REDIS_REPLY_CHUNK_BYTES)
{
c->reply_bytes -= zmalloc_size_sds(tail->ptr);
@@ -242,7 +201,7 @@ void _addReplyObjectToList(redisClient *c, robj *o) {
} else {
incrRefCount(o);
listAddNodeTail(c->reply,o);
c->reply_bytes += getStringObjectSdsUsedMemory(o);
c->reply_bytes += zmalloc_size_sds(o->ptr);
}
}
asyncCloseClientOnOutputBufferLimitReached(c);
@@ -265,7 +224,7 @@ void _addReplySdsToList(redisClient *c, sds s) {
tail = listNodeValue(listLast(c->reply));
/* Append to this object when possible. */
if (tail->ptr != NULL && tail->encoding == REDIS_ENCODING_RAW &&
if (tail->ptr != NULL &&
sdslen(tail->ptr)+sdslen(s) <= REDIS_REPLY_CHUNK_BYTES)
{
c->reply_bytes -= zmalloc_size_sds(tail->ptr);
@@ -290,12 +249,12 @@ void _addReplyStringToList(redisClient *c, char *s, size_t len) {
robj *o = createStringObject(s,len);
listAddNodeTail(c->reply,o);
c->reply_bytes += getStringObjectSdsUsedMemory(o);
c->reply_bytes += zmalloc_size_sds(o->ptr);
} else {
tail = listNodeValue(listLast(c->reply));
/* Append to this object when possible. */
if (tail->ptr != NULL && tail->encoding == REDIS_ENCODING_RAW &&
if (tail->ptr != NULL &&
sdslen(tail->ptr)+len <= REDIS_REPLY_CHUNK_BYTES)
{
c->reply_bytes -= zmalloc_size_sds(tail->ptr);
@@ -306,7 +265,7 @@ void _addReplyStringToList(redisClient *c, char *s, size_t len) {
robj *o = createStringObject(s,len);
listAddNodeTail(c->reply,o);
c->reply_bytes += getStringObjectSdsUsedMemory(o);
c->reply_bytes += zmalloc_size_sds(o->ptr);
}
}
asyncCloseClientOnOutputBufferLimitReached(c);
@@ -327,7 +286,7 @@ void addReply(redisClient *c, robj *obj) {
* If the encoding is RAW and there is room in the static buffer
* we'll be able to send the object to the client without
* messing with its page. */
if (sdsEncodedObject(obj)) {
if (obj->encoding == REDIS_ENCODING_RAW) {
if (_addReplyToBuffer(c,obj->ptr,sdslen(obj->ptr)) != REDIS_OK)
_addReplyObjectToList(c,obj);
} else if (obj->encoding == REDIS_ENCODING_INT) {
@@ -439,7 +398,6 @@ void setDeferredMultiBulkLength(redisClient *c, void *node, long length) {
len = listNodeValue(ln);
len->ptr = sdscatprintf(sdsempty(),"*%ld\r\n",length);
len->encoding = REDIS_ENCODING_RAW; /* in case it was an EMBSTR. */
c->reply_bytes += zmalloc_size_sds(len->ptr);
if (ln->next != NULL) {
next = listNodeValue(ln->next);
@@ -447,7 +405,7 @@ void setDeferredMultiBulkLength(redisClient *c, void *node, long length) {
/* Only glue when the next node is non-NULL (an sds in this case) */
if (next->ptr != NULL) {
c->reply_bytes -= zmalloc_size_sds(len->ptr);
c->reply_bytes -= getStringObjectSdsUsedMemory(next);
c->reply_bytes -= zmalloc_size_sds(next->ptr);
len->ptr = sdscatlen(len->ptr,next->ptr,sdslen(next->ptr));
c->reply_bytes += zmalloc_size_sds(len->ptr);
listDelNode(c->reply,ln->next);
@@ -515,7 +473,7 @@ void addReplyMultiBulkLen(redisClient *c, long length) {
void addReplyBulkLen(redisClient *c, robj *obj) {
size_t len;
if (sdsEncodedObject(obj)) {
if (obj->encoding == REDIS_ENCODING_RAW) {
len = sdslen(obj->ptr);
} else {
long n = (long)obj->ptr;
@@ -666,20 +624,6 @@ void disconnectSlaves(void) {
}
}
/* This function is called when the slave lose the connection with the
* master into an unexpected way. */
void replicationHandleMasterDisconnection(void) {
server.master = NULL;
server.repl_state = REDIS_REPL_CONNECT;
server.repl_down_since = server.unixtime;
/* We lost connection with our master, force our slaves to resync
* with us as well to load the new data set.
*
* If server.masterhost is NULL the user called SLAVEOF NO ONE so
* slave resync is not needed. */
if (server.masterhost != NULL) disconnectSlaves();
}
void freeClient(redisClient *c) {
listNode *ln;
@@ -714,7 +658,8 @@ void freeClient(redisClient *c) {
c->querybuf = NULL;
/* Deallocate structures used to block on blocking ops. */
if (c->flags & REDIS_BLOCKED) unblockClient(c);
if (c->flags & REDIS_BLOCKED)
unblockClientWaitingData(c);
dictRelease(c->bpop.keys);
/* UNWATCH all the keys */
@@ -837,11 +782,10 @@ void sendReplyToClient(aeEventLoop *el, int fd, void *privdata, int mask) {
} else {
o = listNodeValue(listFirst(c->reply));
objlen = sdslen(o->ptr);
objmem = getStringObjectSdsUsedMemory(o);
objmem = zmalloc_size_sds(o->ptr);
if (objlen == 0) {
listDelNode(c->reply,listFirst(c->reply));
c->reply_bytes -= objmem;
continue;
}
@@ -898,16 +842,12 @@ void sendReplyToClient(aeEventLoop *el, int fd, void *privdata, int mask) {
/* resetClient prepare the client to process the next command */
void resetClient(redisClient *c) {
redisCommandProc *prevcmd = c->cmd ? c->cmd->proc : NULL;
freeClientArgv(c);
c->reqtype = 0;
c->multibulklen = 0;
c->bulklen = -1;
/* We clear the ASKING flag as well if we are not inside a MULTI, and
* if what we just executed is not the ASKING command itself. */
if (!(c->flags & REDIS_MULTI) && prevcmd != askingCommand)
c->flags &= (~REDIS_ASKING);
/* We clear the ASKING flag as well if we are not inside a MULTI. */
if (!(c->flags & REDIS_MULTI)) c->flags &= (~REDIS_ASKING);
}
int processInlineBuffer(redisClient *c) {
@@ -974,7 +914,7 @@ int processInlineBuffer(redisClient *c) {
/* Helper function. Trims query buffer to make the function that processes
* multi bulk requests idempotent. */
static void setProtocolError(redisClient *c, int pos) {
if (server.verbosity >= REDIS_VERBOSE) {
if (server.verbosity <= REDIS_VERBOSE) {
sds client = catClientInfoString(sdsempty(),c);
redisLog(REDIS_VERBOSE,
"Protocol error from client: %s", client);
@@ -1125,9 +1065,6 @@ int processMultibulkBuffer(redisClient *c) {
void processInputBuffer(redisClient *c) {
/* Keep processing while there is something in the input buffer */
while(sdslen(c->querybuf)) {
/* Return if clients are paused. */
if (!(c->flags & REDIS_SLAVE) && clientsArePaused()) return;
/* Immediately abort if the client is in the middle of something. */
if (c->flags & REDIS_BLOCKED) return;
@@ -1320,7 +1257,6 @@ sds catClientInfoString(sds s, redisClient *client) {
if (client->flags & REDIS_UNBLOCKED) *p++ = 'u';
if (client->flags & REDIS_CLOSE_ASAP) *p++ = 'A';
if (client->flags & REDIS_UNIX_SOCKET) *p++ = 'U';
if (client->flags & REDIS_READONLY) *p++ = 'r';
if (p == flags) *p++ = 'N';
*p++ = '\0';
@@ -1498,13 +1434,6 @@ void clientCommand(redisClient *c) {
addReplyBulk(c,c->name);
else
addReply(c,shared.nullbulk);
} else if (!strcasecmp(c->argv[1]->ptr,"pause") && c->argc == 3) {
long long duration;
if (getTimeoutFromObjectOrReply(c,c->argv[2],&duration,UNIT_MILLISECONDS)
!= REDIS_OK) return;
pauseClients(duration);
addReply(c,shared.ok);
} else {
addReplyError(c, "Syntax error, try CLIENT (LIST | KILL ip:port | GETNAME | SETNAME connection-name)");
}
@@ -1679,6 +1608,12 @@ void flushSlavesOutputBuffers(void) {
redisClient *slave = listNodeValue(ln);
int events;
/* Note that the following will not flush output buffers of slaves
* in STATE_ONLINE but having put_online_on_ack set to true: in this
* case the writable event is never installed, since the purpose
* of put_online_on_ack is to postpone the moment it is installed.
* This is what we want since slaves in this state should not receive
* writes before the first ACK. */
events = aeGetFileEvents(server.el,slave->fd);
if (events & AE_WRITABLE &&
slave->replstate == REDIS_REPL_ONLINE &&
@@ -1689,57 +1624,6 @@ void flushSlavesOutputBuffers(void) {
}
}
/* Pause clients up to the specified unixtime (in ms). While clients
* are paused no command is processed from clients, so the data set can't
* change during that time.
*
* However while this function pauses normal and Pub/Sub clients, slaves are
* still served, so this function can be used on server upgrades where it is
* required that slaves process the latest bytes from the replication stream
* before being turned to masters.
*
* This function is also internally used by Redis Cluster for the manual
* failover procedure implemented by CLUSTER FAILOVER.
*
* The function always succeed, even if there is already a pause in progress.
* In such a case, the pause is extended if the duration is more than the
* time left for the previous duration. However if the duration is smaller
* than the time left for the previous pause, no change is made to the
* left duration. */
void pauseClients(mstime_t end) {
if (!server.clients_paused || end > server.clients_pause_end_time)
server.clients_pause_end_time = end;
server.clients_paused = 1;
}
/* Return non-zero if clients are currently paused. As a side effect the
* function checks if the pause time was reached and clear it. */
int clientsArePaused(void) {
if (server.clients_paused &&
server.clients_pause_end_time < server.mstime)
{
listNode *ln;
listIter li;
redisClient *c;
server.clients_paused = 0;
/* Put all the clients in the unblocked clients queue in order to
* force the re-processing of the input buffer if any. */
listRewind(server.clients,&li);
while ((ln = listNext(&li)) != NULL) {
c = listNodeValue(ln);
/* Don't touch slaves and blocked clients. The latter pending
* requests be processed when unblocked. */
if (c->flags & (REDIS_SLAVE|REDIS_BLOCKED)) continue;
c->flags |= REDIS_UNBLOCKED;
listAddNodeTail(server.unblocked_clients,c);
}
}
return server.clients_paused;
}
/* This function is called by Redis in order to process a few events from
* time to time while blocked into some not interruptible operation.
* This allows to reply to clients with the -LOADING error while loading the
src/object.c
+66 -152
View File
@@ -44,52 +44,12 @@ robj *createObject(int type, void *ptr) {
o->refcount = 1;
/* Set the LRU to the current lruclock (minutes resolution). */
o->lru = LRU_CLOCK();
o->lru = server.lruclock;
return o;
}
/* Create a string object with encoding REDIS_ENCODING_RAW, that is a plain
* string object where o->ptr points to a proper sds string. */
robj *createRawStringObject(char *ptr, size_t len) {
return createObject(REDIS_STRING,sdsnewlen(ptr,len));
}
/* Create a string object with encoding REDIS_ENCODING_EMBSTR, that is
* an object where the sds string is actually an unmodifiable string
* allocated in the same chunk as the object itself. */
robj *createEmbeddedStringObject(char *ptr, size_t len) {
robj *o = zmalloc(sizeof(robj)+sizeof(struct sdshdr)+len+1);
struct sdshdr *sh = (void*)(o+1);
o->type = REDIS_STRING;
o->encoding = REDIS_ENCODING_EMBSTR;
o->ptr = sh+1;
o->refcount = 1;
o->lru = LRU_CLOCK();
sh->len = len;
sh->free = 0;
if (ptr) {
memcpy(sh->buf,ptr,len);
sh->buf[len] = '\0';
} else {
memset(sh->buf,0,len+1);
}
return o;
}
/* Create a string object with EMBSTR encoding if it is smaller than
* REIDS_ENCODING_EMBSTR_SIZE_LIMIT, otherwise the RAW encoding is
* used.
*
* The current limit of 39 is chosen so that the biggest string object
* we allocate as EMBSTR will still fit into the 64 byte arena of jemalloc. */
#define REDIS_ENCODING_EMBSTR_SIZE_LIMIT 39
robj *createStringObject(char *ptr, size_t len) {
if (len <= REDIS_ENCODING_EMBSTR_SIZE_LIMIT)
return createEmbeddedStringObject(ptr,len);
else
return createRawStringObject(ptr,len);
return createObject(REDIS_STRING,sdsnewlen(ptr,len));
}
robj *createStringObjectFromLongLong(long long value) {
@@ -151,33 +111,9 @@ robj *createStringObjectFromLongDouble(long double value, int humanfriendly) {
return createStringObject(buf,len);
}
/* Duplicate a string object, with the guarantee that the returned object
* has the same encoding as the original one.
*
* This function also guarantees that duplicating a small integere object
* (or a string object that contains a representation of a small integer)
* will always result in a fresh object that is unshared (refcount == 1).
*
* The resulting object always has refcount set to 1. */
robj *dupStringObject(robj *o) {
robj *d;
redisAssert(o->type == REDIS_STRING);
switch(o->encoding) {
case REDIS_ENCODING_RAW:
return createRawStringObject(o->ptr,sdslen(o->ptr));
case REDIS_ENCODING_EMBSTR:
return createEmbeddedStringObject(o->ptr,sdslen(o->ptr));
case REDIS_ENCODING_INT:
d = createObject(REDIS_STRING, NULL);
d->encoding = REDIS_ENCODING_INT;
d->ptr = o->ptr;
return d;
default:
redisPanic("Wrong encoding.");
break;
}
redisAssertWithInfo(NULL,o,o->encoding == REDIS_ENCODING_RAW);
return createStringObject(o->ptr,sdslen(o->ptr));
}
robj *createListObject(void) {
@@ -366,78 +302,61 @@ robj *tryObjectEncoding(robj *o) {
sds s = o->ptr;
size_t len;
/* Make sure this is a string object, the only type we encode
* in this function. Other types use encoded memory efficient
* representations but are handled by the commands implementing
* the type. */
redisAssertWithInfo(NULL,o,o->type == REDIS_STRING);
/* We try some specialized encoding only for objects that are
* RAW or EMBSTR encoded, in other words objects that are still
* in represented by an actually array of chars. */
if (!sdsEncodedObject(o)) return o;
if (o->encoding != REDIS_ENCODING_RAW)
return o; /* Already encoded */
/* It's not safe to encode shared objects: shared objects can be shared
* everywhere in the "object space" of Redis and may end in places where
* they are not handled. We handle them only as values in the keyspace. */
* everywhere in the "object space" of Redis. Encoded objects can only
* appear as "values" (and not, for instance, as keys) */
if (o->refcount > 1) return o;
/* Check if we can represent this string as a long integer.
* Note that we are sure that a string larger than 21 chars is not
* representable as a 32 nor 64 bit integer. */
/* Currently we try to encode only strings */
redisAssertWithInfo(NULL,o,o->type == REDIS_STRING);
/* Check if we can represent this string as a long integer */
len = sdslen(s);
if (len <= 21 && string2l(s,len,&value)) {
/* This object is encodable as a long. Try to use a shared object.
* Note that we avoid using shared integers when maxmemory is used
* because every object needs to have a private LRU field for the LRU
* algorithm to work well. */
if ((server.maxmemory == 0 ||
(server.maxmemory_policy != REDIS_MAXMEMORY_VOLATILE_LRU &&
server.maxmemory_policy != REDIS_MAXMEMORY_ALLKEYS_LRU)) &&
value >= 0 &&
value < REDIS_SHARED_INTEGERS)
if (len > 21 || !string2l(s,len,&value)) {
/* We can't encode the object...
*
* Do the last try, and at least optimize the SDS string inside
* the string object to require little space, in case there
* is more than 10% of free space at the end of the SDS string.
*
* We do that for larger strings, using the arbitrary value
* of 32 bytes. This code was backported from the unstable branch
* where this is performed when the object is too large to be
* encoded as EMBSTR. */
if (len > 32 &&
o->encoding == REDIS_ENCODING_RAW &&
sdsavail(s) > len/10)
{
decrRefCount(o);
incrRefCount(shared.integers[value]);
return shared.integers[value];
} else {
if (o->encoding == REDIS_ENCODING_RAW) sdsfree(o->ptr);
o->encoding = REDIS_ENCODING_INT;
o->ptr = (void*) value;
return o;
o->ptr = sdsRemoveFreeSpace(o->ptr);
}
/* Return the original object. */
return o;
}
/* If the string is small and is still RAW encoded,
* try the EMBSTR encoding which is more efficient.
* In this representation the object and the SDS string are allocated
* in the same chunk of memory to save space and cache misses. */
if (len <= REDIS_ENCODING_EMBSTR_SIZE_LIMIT) {
robj *emb;
if (o->encoding == REDIS_ENCODING_EMBSTR) return o;
emb = createEmbeddedStringObject(s,sdslen(s));
decrRefCount(o);
return emb;
}
/* We can't encode the object...
/* Ok, this object can be encoded...
*
* Do the last try, and at least optimize the SDS string inside
* the string object to require little space, in case there
* is more than 10% of free space at the end of the SDS string.
* Can I use a shared object? Only if the object is inside a given range
*
* We do that only for relatively large strings as this branch
* is only entered if the length of the string is greater than
* REDIS_ENCODING_EMBSTR_SIZE_LIMIT. */
if (o->encoding == REDIS_ENCODING_RAW &&
sdsavail(s) > len/10)
* Note that we also avoid using shared integers when maxmemory is used
* because every object needs to have a private LRU field for the LRU
* algorithm to work well. */
if ((server.maxmemory == 0 ||
(server.maxmemory_policy != REDIS_MAXMEMORY_VOLATILE_LRU &&
server.maxmemory_policy != REDIS_MAXMEMORY_ALLKEYS_LRU)) &&
value >= 0 && value < REDIS_SHARED_INTEGERS)
{
o->ptr = sdsRemoveFreeSpace(o->ptr);
decrRefCount(o);
incrRefCount(shared.integers[value]);
return shared.integers[value];
} else {
o->encoding = REDIS_ENCODING_INT;
sdsfree(o->ptr);
o->ptr = (void*) value;
return o;
}
/* Return the original object. */
return o;
}
/* Get a decoded version of an encoded object (returned as a new object).
@@ -445,7 +364,7 @@ robj *tryObjectEncoding(robj *o) {
robj *getDecodedObject(robj *o) {
robj *dec;
if (sdsEncodedObject(o)) {
if (o->encoding == REDIS_ENCODING_RAW) {
incrRefCount(o);
return o;
}
@@ -477,19 +396,19 @@ int compareStringObjectsWithFlags(robj *a, robj *b, int flags) {
size_t alen, blen, minlen;
if (a == b) return 0;
if (sdsEncodedObject(a)) {
astr = a->ptr;
alen = sdslen(astr);
} else {
if (a->encoding != REDIS_ENCODING_RAW) {
alen = ll2string(bufa,sizeof(bufa),(long) a->ptr);
astr = bufa;
}
if (sdsEncodedObject(b)) {
bstr = b->ptr;
blen = sdslen(bstr);
} else {
astr = a->ptr;
alen = sdslen(astr);
}
if (b->encoding != REDIS_ENCODING_RAW) {
blen = ll2string(bufb,sizeof(bufb),(long) b->ptr);
bstr = bufb;
} else {
bstr = b->ptr;
blen = sdslen(bstr);
}
if (flags & REDIS_COMPARE_COLL) {
return strcoll(astr,bstr);
@@ -518,10 +437,7 @@ int collateStringObjects(robj *a, robj *b) {
* this function is faster then checking for (compareStringObject(a,b) == 0)
* because it can perform some more optimization. */
int equalStringObjects(robj *a, robj *b) {
if (a->encoding == REDIS_ENCODING_INT &&
b->encoding == REDIS_ENCODING_INT){
/* If both strings are integer encoded just check if the stored
* long is the same. */
if (a->encoding != REDIS_ENCODING_RAW && b->encoding != REDIS_ENCODING_RAW){
return a->ptr == b->ptr;
} else {
return compareStringObjects(a,b) == 0;
@@ -530,7 +446,7 @@ int equalStringObjects(robj *a, robj *b) {
size_t stringObjectLen(robj *o) {
redisAssertWithInfo(NULL,o,o->type == REDIS_STRING);
if (sdsEncodedObject(o)) {
if (o->encoding == REDIS_ENCODING_RAW) {
return sdslen(o->ptr);
} else {
char buf[32];
@@ -547,7 +463,7 @@ int getDoubleFromObject(robj *o, double *target) {
value = 0;
} else {
redisAssertWithInfo(NULL,o,o->type == REDIS_STRING);
if (sdsEncodedObject(o)) {
if (o->encoding == REDIS_ENCODING_RAW) {
errno = 0;
value = strtod(o->ptr, &eptr);
if (isspace(((char*)o->ptr)[0]) ||
@@ -589,7 +505,7 @@ int getLongDoubleFromObject(robj *o, long double *target) {
value = 0;
} else {
redisAssertWithInfo(NULL,o,o->type == REDIS_STRING);
if (sdsEncodedObject(o)) {
if (o->encoding == REDIS_ENCODING_RAW) {
errno = 0;
value = strtold(o->ptr, &eptr);
if (isspace(((char*)o->ptr)[0]) || eptr[0] != '\0' ||
@@ -627,7 +543,7 @@ int getLongLongFromObject(robj *o, long long *target) {
value = 0;
} else {
redisAssertWithInfo(NULL,o,o->type == REDIS_STRING);
if (sdsEncodedObject(o)) {
if (o->encoding == REDIS_ENCODING_RAW) {
errno = 0;
value = strtoll(o->ptr, &eptr, 10);
if (isspace(((char*)o->ptr)[0]) || eptr[0] != '\0' ||
@@ -682,19 +598,17 @@ char *strEncoding(int encoding) {
case REDIS_ENCODING_ZIPLIST: return "ziplist";
case REDIS_ENCODING_INTSET: return "intset";
case REDIS_ENCODING_SKIPLIST: return "skiplist";
case REDIS_ENCODING_EMBSTR: return "embstr";
default: return "unknown";
}
}
/* Given an object returns the min number of milliseconds the object was never
/* Given an object returns the min number of seconds the object was never
* requested, using an approximated LRU algorithm. */
unsigned long long estimateObjectIdleTime(robj *o) {
unsigned long long lruclock = LRU_CLOCK();
if (lruclock >= o->lru) {
return (lruclock - o->lru) * REDIS_LRU_CLOCK_RESOLUTION;
unsigned long estimateObjectIdleTime(robj *o) {
if (server.lruclock >= o->lru) {
return (server.lruclock - o->lru) * REDIS_LRU_CLOCK_RESOLUTION;
} else {
return (lruclock + (REDIS_LRU_CLOCK_MAX - o->lru)) *
return ((REDIS_LRU_CLOCK_MAX - o->lru) + server.lruclock) *
REDIS_LRU_CLOCK_RESOLUTION;
}
}
@@ -731,7 +645,7 @@ void objectCommand(redisClient *c) {
} else if (!strcasecmp(c->argv[1]->ptr,"idletime") && c->argc == 3) {
if ((o = objectCommandLookupOrReply(c,c->argv[2],shared.nullbulk))
== NULL) return;
addReplyLongLong(c,estimateObjectIdleTime(o)/1000);
addReplyLongLong(c,estimateObjectIdleTime(o));
} else {
addReplyError(c,"Syntax error. Try OBJECT (refcount|encoding|idletime)");
}
src/pubsub.c
+4 -7
View File
@@ -56,7 +56,7 @@ int clientSubscriptionsCount(redisClient *c) {
/* Subscribe a client to a channel. Returns 1 if the operation succeeded, or
* 0 if the client was already subscribed to that channel. */
int pubsubSubscribeChannel(redisClient *c, robj *channel) {
dictEntry *de;
struct dictEntry *de;
list *clients = NULL;
int retval = 0;
@@ -86,7 +86,7 @@ int pubsubSubscribeChannel(redisClient *c, robj *channel) {
/* Unsubscribe a client from a channel. Returns 1 if the operation succeeded, or
* 0 if the client was not subscribed to the specified channel. */
int pubsubUnsubscribeChannel(redisClient *c, robj *channel, int notify) {
dictEntry *de;
struct dictEntry *de;
list *clients;
listNode *ln;
int retval = 0;
@@ -224,7 +224,7 @@ int pubsubUnsubscribeAllPatterns(redisClient *c, int notify) {
/* Publish a message */
int pubsubPublishMessage(robj *channel, robj *message) {
int receivers = 0;
dictEntry *de;
struct dictEntry *de;
listNode *ln;
listIter li;
@@ -316,10 +316,7 @@ void punsubscribeCommand(redisClient *c) {
void publishCommand(redisClient *c) {
int receivers = pubsubPublishMessage(c->argv[1],c->argv[2]);
if (server.cluster_enabled)
clusterPropagatePublish(c->argv[1],c->argv[2]);
else
forceCommandPropagation(c,REDIS_PROPAGATE_REPL);
forceCommandPropagation(c,REDIS_PROPAGATE_REPL);
addReplyLongLong(c,receivers);
}
src/rdb.c
+41 -25
View File
@@ -325,7 +325,7 @@ int rdbSaveStringObject(rio *rdb, robj *obj) {
if (obj->encoding == REDIS_ENCODING_INT) {
return rdbSaveLongLongAsStringObject(rdb,(long)obj->ptr);
} else {
redisAssertWithInfo(NULL,obj,sdsEncodedObject(obj));
redisAssertWithInfo(NULL,obj,obj->encoding == REDIS_ENCODING_RAW);
return rdbSaveRawString(rdb,obj->ptr,sdslen(obj->ptr));
}
}
@@ -333,7 +333,7 @@ int rdbSaveStringObject(rio *rdb, robj *obj) {
robj *rdbGenericLoadStringObject(rio *rdb, int encode) {
int isencoded;
uint32_t len;
robj *o;
sds val;
len = rdbLoadLen(rdb,&isencoded);
if (isencoded) {
@@ -350,13 +350,12 @@ robj *rdbGenericLoadStringObject(rio *rdb, int encode) {
}
if (len == REDIS_RDB_LENERR) return NULL;
o = encode ? createStringObject(NULL,len) :
createRawStringObject(NULL,len);
if (len && rioRead(rdb,o->ptr,len) == 0) {
decrRefCount(o);
val = sdsnewlen(NULL,len);
if (len && rioRead(rdb,val,len) == 0) {
sdsfree(val);
return NULL;
}
return o;
return createObject(REDIS_STRING,val);
}
robj *rdbLoadStringObject(rio *rdb) {
@@ -755,9 +754,9 @@ int rdbSave(char *filename) {
return REDIS_OK;
werr:
redisLog(REDIS_WARNING,"Write error saving DB on disk: %s", strerror(errno));
fclose(fp);
unlink(tmpfile);
redisLog(REDIS_WARNING,"Write error saving DB on disk: %s", strerror(errno));
return REDIS_ERR;
}
@@ -845,7 +844,7 @@ robj *rdbLoadObject(int rdbtype, rio *rdb) {
/* If we are using a ziplist and the value is too big, convert
* the object to a real list. */
if (o->encoding == REDIS_ENCODING_ZIPLIST &&
sdsEncodedObject(ele) &&
ele->encoding == REDIS_ENCODING_RAW &&
sdslen(ele->ptr) > server.list_max_ziplist_value)
listTypeConvert(o,REDIS_ENCODING_LINKEDLIST);
@@ -891,7 +890,7 @@ robj *rdbLoadObject(int rdbtype, rio *rdb) {
}
/* This will also be called when the set was just converted
* to a regular hash table encoded set */
* to regular hash table encoded set */
if (o->encoding == REDIS_ENCODING_HT) {
dictAdd((dict*)o->ptr,ele,NULL);
} else {
@@ -919,8 +918,9 @@ robj *rdbLoadObject(int rdbtype, rio *rdb) {
if (rdbLoadDoubleValue(rdb,&score) == -1) return NULL;
/* Don't care about integer-encoded strings. */
if (sdsEncodedObject(ele) && sdslen(ele->ptr) > maxelelen)
maxelelen = sdslen(ele->ptr);
if (ele->encoding == REDIS_ENCODING_RAW &&
sdslen(ele->ptr) > maxelelen)
maxelelen = sdslen(ele->ptr);
znode = zslInsert(zs->zsl,score,ele);
dictAdd(zs->dict,ele,&znode->score);
@@ -952,10 +952,10 @@ robj *rdbLoadObject(int rdbtype, rio *rdb) {
/* Load raw strings */
field = rdbLoadStringObject(rdb);
if (field == NULL) return NULL;
redisAssert(sdsEncodedObject(field));
redisAssert(field->encoding == REDIS_ENCODING_RAW);
value = rdbLoadStringObject(rdb);
if (value == NULL) return NULL;
redisAssert(sdsEncodedObject(value));
redisAssert(field->encoding == REDIS_ENCODING_RAW);
/* Add pair to ziplist */
o->ptr = ziplistPush(o->ptr, field->ptr, sdslen(field->ptr), ZIPLIST_TAIL);
@@ -1420,7 +1420,7 @@ int rdbSaveToSlavesSockets(void) {
if (slave->replstate == REDIS_REPL_WAIT_BGSAVE_START) {
clientids[numfds] = slave->id;
fds[numfds++] = slave->fd;
slave->replstate = REDIS_REPL_WAIT_BGSAVE_END;
replicationSetupSlaveForFullResync(slave,getPsyncInitialOffset());
/* Put the socket in non-blocking mode to simplify RDB transfer.
* We'll restore it when the children returns (since duped socket
* will share the O_NONBLOCK attribute with the parent). */
@@ -1498,27 +1498,43 @@ int rdbSaveToSlavesSockets(void) {
exitFromChild((retval == REDIS_OK) ? 0 : 1);
} else {
/* Parent */
zfree(clientids); /* Not used by parent. Free ASAP. */
server.stat_fork_time = ustime()-start;
server.stat_fork_rate = (double) zmalloc_used_memory() * 1000000 / server.stat_fork_time / (1024*1024*1024); /* GB per second. */
latencyAddSampleIfNeeded("fork",server.stat_fork_time/1000);
if (childpid == -1) {
redisLog(REDIS_WARNING,"Can't save in background: fork: %s",
strerror(errno));
zfree(fds);
/* Undo the state change. The caller will perform cleanup on
* all the slaves in BGSAVE_START state, but an early call to
* replicationSetupSlaveForFullResync() turned it into BGSAVE_END */
listRewind(server.slaves,&li);
while((ln = listNext(&li))) {
redisClient *slave = ln->value;
int j;
for (j = 0; j < numfds; j++) {
if (slave->id == clientids[j]) {
slave->replstate = REDIS_REPL_WAIT_BGSAVE_START;
break;
}
}
}
close(pipefds[0]);
close(pipefds[1]);
return REDIS_ERR;
} else {
redisLog(REDIS_NOTICE,"Background RDB transfer started by pid %d",
childpid);
server.rdb_save_time_start = time(NULL);
server.rdb_child_pid = childpid;
server.rdb_child_type = REDIS_RDB_CHILD_TYPE_SOCKET;
updateDictResizePolicy();
}
redisLog(REDIS_NOTICE,"Background RDB transfer started by pid %d",childpid);
server.rdb_save_time_start = time(NULL);
server.rdb_child_pid = childpid;
server.rdb_child_type = REDIS_RDB_CHILD_TYPE_SOCKET;
updateDictResizePolicy();
zfree(clientids);
zfree(fds);
return REDIS_OK;
return (childpid == -1) ? REDIS_ERR : REDIS_OK;
}
return REDIS_OK; /* unreached */
return REDIS_OK; /* Unreached. */
}
void saveCommand(redisClient *c) {
src/redis-cli.c
+33 -56
View File
@@ -64,17 +64,6 @@
#define REDIS_CLI_HISTFILE_ENV "REDISCLI_HISTFILE"
#define REDIS_CLI_HISTFILE_DEFAULT ".rediscli_history"
/* --latency-dist palettes. */
int spectrum_palette_color_size = 19;
int spectrum_palette_color[] = {0,233,234,235,237,239,241,243,245,247,144,143,142,184,226,214,208,202,196};
int spectrum_palette_mono_size = 13;
int spectrum_palette_mono[] = {0,233,234,235,237,239,241,243,245,247,249,251,253};
/* The actual palette in use. */
int *spectrum_palette;
int spectrum_palette_size;
static redisContext *context;
static struct config {
char *hostip;
@@ -343,7 +332,7 @@ static void completionCallback(const char *buf, linenoiseCompletions *lc) {
*--------------------------------------------------------------------------- */
/* Send AUTH command to the server */
static int cliAuth(void) {
static int cliAuth() {
redisReply *reply;
if (config.auth == NULL) return REDIS_OK;
@@ -356,7 +345,7 @@ static int cliAuth(void) {
}
/* Send SELECT dbnum to the server */
static int cliSelect(void) {
static int cliSelect() {
redisReply *reply;
if (config.dbnum == 0) return REDIS_OK;
@@ -607,7 +596,6 @@ static int cliReadReply(int output_raw_strings) {
printf("-> Redirected to slot [%d] located at %s:%d\n",
slot, config.hostip, config.hostport);
config.cluster_reissue_command = 1;
cliRefreshPrompt();
}
if (output) {
@@ -789,9 +777,6 @@ static int parseOptions(int argc, char **argv) {
config.latency_mode = 1;
} else if (!strcmp(argv[i],"--latency-dist")) {
config.latency_dist_mode = 1;
} else if (!strcmp(argv[i],"--mono")) {
spectrum_palette = spectrum_palette_mono;
spectrum_palette_size = spectrum_palette_mono_size;
} else if (!strcmp(argv[i],"--latency-history")) {
config.latency_mode = 1;
config.latency_history = 1;
@@ -934,33 +919,6 @@ static char **convertToSds(int count, char** args) {
return sds;
}
static int issueCommandRepeat(int argc, char **argv, long repeat) {
while (1) {
config.cluster_reissue_command = 0;
if (cliSendCommand(argc,argv,repeat) != REDIS_OK) {
cliConnect(1);
/* If we still cannot send the command print error.
* We'll try to reconnect the next time. */
if (cliSendCommand(argc,argv,repeat) != REDIS_OK) {
cliPrintContextError();
return REDIS_ERR;
}
}
/* Issue the command again if we got redirected in cluster mode */
if (config.cluster_mode && config.cluster_reissue_command) {
cliConnect(1);
} else {
break;
}
}
return REDIS_OK;
}
static int issueCommand(int argc, char **argv) {
return issueCommandRepeat(argc, argv, config.repeat);
}
static void repl(void) {
sds historyfile = NULL;
int history = 0;
@@ -1016,8 +974,26 @@ static void repl(void) {
repeat = 1;
}
issueCommandRepeat(argc-skipargs, argv+skipargs, repeat);
while (1) {
config.cluster_reissue_command = 0;
if (cliSendCommand(argc-skipargs,argv+skipargs,repeat)
!= REDIS_OK)
{
cliConnect(1);
/* If we still cannot send the command print error.
* We'll try to reconnect the next time. */
if (cliSendCommand(argc-skipargs,argv+skipargs,repeat)
!= REDIS_OK)
cliPrintContextError();
}
/* Issue the command again if we got redirected in cluster mode */
if (config.cluster_mode && config.cluster_reissue_command) {
cliConnect(1);
} else {
break;
}
}
elapsed = mstime()-start_time;
if (elapsed >= 500) {
printf("(%.2fs)\n",(double)elapsed/1000);
@@ -1038,9 +1014,10 @@ static int noninteractive(int argc, char **argv) {
if (config.stdinarg) {
argv = zrealloc(argv, (argc+1)*sizeof(char*));
argv[argc] = readArgFromStdin();
retval = issueCommand(argc+1, argv);
retval = cliSendCommand(argc+1, argv, config.repeat);
} else {
retval = issueCommand(argc, argv);
/* stdin is probably a tty, can be tested with S_ISCHR(s.st_mode) */
retval = cliSendCommand(argc, argv, config.repeat);
}
return retval;
}
@@ -1084,7 +1061,7 @@ static int evalMode(int argc, char **argv) {
argv2[2] = sdscatprintf(sdsempty(),"%d",keys);
/* Call it */
return issueCommand(argc+3-got_comma, argv2);
return cliSendCommand(argc+3-got_comma, argv2, config.repeat);
}
/*------------------------------------------------------------------------------
@@ -1140,6 +1117,13 @@ static void latencyMode(void) {
*--------------------------------------------------------------------------- */
#define LATENCY_DIST_DEFAULT_INTERVAL 1000 /* milliseconds. */
#define LATENCY_DIST_MIN_GRAY 233 /* Less than that is too hard to see gray. */
#define LATENCY_DIST_MAX_GRAY 255
#define LATENCY_DIST_GRAYS (LATENCY_DIST_MAX_GRAY-LATENCY_DIST_MIN_GRAY+1)
/* Gray palette. */
int spectrum_palette_size = 24;
int spectrum_palette[] = {0, 233,234,235,236,237,238,239,240,241,242,243,244,245,246,247,248,249,250,251,252,253,254,255};
/* Structure to store samples distribution. */
struct distsamples {
@@ -1582,7 +1566,7 @@ static redisReply *sendScan(unsigned long long *it) {
assert(reply->element[1]->type == REDIS_REPLY_ARRAY);
/* Update iterator */
*it = atoi(reply->element[0]->str);
*it = strtoull(reply->element[0]->str, NULL, 10);
return reply;
}
@@ -1950,7 +1934,6 @@ static void statMode(void) {
/* Children */
aux = getLongInfoField(reply->str,"bgsave_in_progress");
aux |= getLongInfoField(reply->str,"aof_rewrite_in_progress") << 1;
aux |= getLongInfoField(reply->str,"loading") << 2;
switch(aux) {
case 0: break;
case 1:
@@ -1962,9 +1945,6 @@ static void statMode(void) {
case 3:
printf("SAVE+AOF");
break;
case 4:
printf("LOAD");
break;
}
printf("\n");
@@ -2205,9 +2185,6 @@ int main(int argc, char **argv) {
config.eval = NULL;
config.last_cmd_type = -1;
spectrum_palette = spectrum_palette_color;
spectrum_palette_size = spectrum_palette_color_size;
if (!isatty(fileno(stdout)) && (getenv("FAKETTY") == NULL))
config.output = OUTPUT_RAW;
else
src/redis-trib.rb
-1373
View File
File diff suppressed because it is too large Load Diff
src/redis.c
+77 -379
View File
@@ -28,7 +28,6 @@
*/
#include "redis.h"
#include "cluster.h"
#include "slowlog.h"
#include "bio.h"
#include "latency.h"
@@ -113,8 +112,6 @@ struct redisServer server; /* server global state */
* server this data. Normally no command is accepted in this condition
* but just a few.
* M: Do not automatically propagate the command on MONITOR.
* k: Perform an implicit ASKING for this command, so the command will be
* accepted in cluster mode if the slot is marked as 'importing'.
* F: Fast command: O(1) or O(log(N)) command that should never delay
* its execution as long as the kernel scheduler is giving us time.
* Note that commands that may trigger a DEL as a side effect (like SET)
@@ -240,7 +237,7 @@ struct redisCommand redisCommandTable[] = {
{"replconf",replconfCommand,-1,"arslt",0,NULL,0,0,0,0,0},
{"flushdb",flushdbCommand,1,"w",0,NULL,0,0,0,0,0},
{"flushall",flushallCommand,1,"w",0,NULL,0,0,0,0,0},
{"sort",sortCommand,-2,"wm",0,sortGetKeys,1,1,1,0,0},
{"sort",sortCommand,-2,"wm",0,NULL,1,1,1,0,0},
{"info",infoCommand,-1,"rlt",0,NULL,0,0,0,0,0},
{"monitor",monitorCommand,1,"ars",0,NULL,0,0,0,0,0},
{"ttl",ttlCommand,2,"rF",0,NULL,1,1,1,0,0},
@@ -258,36 +255,28 @@ struct redisCommand redisCommandTable[] = {
{"pubsub",pubsubCommand,-2,"pltrR",0,NULL,0,0,0,0,0},
{"watch",watchCommand,-2,"rsF",0,NULL,1,-1,1,0,0},
{"unwatch",unwatchCommand,1,"rsF",0,NULL,0,0,0,0,0},
{"cluster",clusterCommand,-2,"ar",0,NULL,0,0,0,0,0},
{"restore",restoreCommand,-4,"wm",0,NULL,1,1,1,0,0},
{"restore-asking",restoreCommand,-4,"wmk",0,NULL,1,1,1,0,0},
{"migrate",migrateCommand,-6,"w",0,NULL,0,0,0,0,0},
{"asking",askingCommand,1,"r",0,NULL,0,0,0,0,0},
{"readonly",readonlyCommand,1,"rF",0,NULL,0,0,0,0,0},
{"readwrite",readwriteCommand,1,"rF",0,NULL,0,0,0,0,0},
{"restore",restoreCommand,4,"wm",0,NULL,1,1,1,0,0},
{"migrate",migrateCommand,6,"w",0,NULL,0,0,0,0,0},
{"dump",dumpCommand,2,"r",0,NULL,1,1,1,0,0},
{"object",objectCommand,3,"r",0,NULL,2,2,2,0,0},
{"client",clientCommand,-2,"rs",0,NULL,0,0,0,0,0},
{"eval",evalCommand,-3,"s",0,evalGetKeys,0,0,0,0,0},
{"evalsha",evalShaCommand,-3,"s",0,evalGetKeys,0,0,0,0,0},
{"eval",evalCommand,-3,"s",0,zunionInterGetKeys,0,0,0,0,0},
{"evalsha",evalShaCommand,-3,"s",0,zunionInterGetKeys,0,0,0,0,0},
{"slowlog",slowlogCommand,-2,"r",0,NULL,0,0,0,0,0},
{"script",scriptCommand,-2,"rs",0,NULL,0,0,0,0,0},
{"time",timeCommand,1,"rRF",0,NULL,0,0,0,0,0},
{"bitop",bitopCommand,-4,"wm",0,NULL,2,-1,1,0,0},
{"bitcount",bitcountCommand,-2,"r",0,NULL,1,1,1,0,0},
{"bitpos",bitposCommand,-3,"r",0,NULL,1,1,1,0,0},
{"wait",waitCommand,3,"rs",0,NULL,0,0,0,0,0},
{"command",commandCommand,0,"rlt",0,NULL,0,0,0,0,0},
{"pfselftest",pfselftestCommand,1,"r",0,NULL,0,0,0,0,0},
{"pfadd",pfaddCommand,-2,"wmF",0,NULL,1,1,1,0,0},
{"pfcount",pfcountCommand,-2,"r",0,NULL,1,1,1,0,0},
{"pfcount",pfcountCommand,-2,"r",0,NULL,1,-1,1,0,0},
{"pfmerge",pfmergeCommand,-2,"wm",0,NULL,1,-1,1,0,0},
{"pfdebug",pfdebugCommand,-3,"w",0,NULL,0,0,0,0,0},
{"latency",latencyCommand,-2,"arslt",0,NULL,0,0,0,0,0}
};
struct evictionPoolEntry *evictionPoolAlloc(void);
/*============================ Utility functions ============================ */
/* Low level logging. To use only for very big messages, otherwise
@@ -311,21 +300,11 @@ void redisLogRaw(int level, const char *msg) {
} else {
int off;
struct timeval tv;
int role_char;
pid_t pid = getpid();
gettimeofday(&tv,NULL);
off = strftime(buf,sizeof(buf),"%d %b %H:%M:%S.",localtime(&tv.tv_sec));
snprintf(buf+off,sizeof(buf)-off,"%03d",(int)tv.tv_usec/1000);
if (server.sentinel_mode) {
role_char = 'X'; /* Sentinel. */
} else if (pid != server.pid) {
role_char = 'C'; /* RDB / AOF writing child. */
} else {
role_char = (server.masterhost ? 'S':'M'); /* Slave or Master. */
}
fprintf(fp,"%d:%c %s %c %s\n",
(int)getpid(),role_char, buf,c[level],msg);
fprintf(fp,"[%d] %s %c %s\n",(int)getpid(),buf,c[level],msg);
}
fflush(fp);
@@ -366,8 +345,9 @@ void redisLogFromHandler(int level, const char *msg) {
open(server.logfile, O_APPEND|O_CREAT|O_WRONLY, 0644);
if (fd == -1) return;
ll2string(buf,sizeof(buf),getpid());
if (write(fd,"[",1) == -1) goto err;
if (write(fd,buf,strlen(buf)) == -1) goto err;
if (write(fd,":signal-handler (",17) == -1) goto err;
if (write(fd," | signal handler] (",20) == -1) goto err;
ll2string(buf,sizeof(buf),time(NULL));
if (write(fd,buf,strlen(buf)) == -1) goto err;
if (write(fd,") ",2) == -1) goto err;
@@ -501,7 +481,7 @@ int dictEncObjKeyCompare(void *privdata, const void *key1,
unsigned int dictEncObjHash(const void *key) {
robj *o = (robj*) key;
if (sdsEncodedObject(o)) {
if (o->encoding == REDIS_ENCODING_RAW) {
return dictGenHashFunction(o->ptr, sdslen((sds)o->ptr));
} else {
if (o->encoding == REDIS_ENCODING_INT) {
@@ -603,39 +583,6 @@ dictType keylistDictType = {
dictListDestructor /* val destructor */
};
/* Cluster nodes hash table, mapping nodes addresses 1.2.3.4:6379 to
* clusterNode structures. */
dictType clusterNodesDictType = {
dictSdsHash, /* hash function */
NULL, /* key dup */
NULL, /* val dup */
dictSdsKeyCompare, /* key compare */
dictSdsDestructor, /* key destructor */
NULL /* val destructor */
};
/* Cluster re-addition blacklist. This maps node IDs to the time
* we can re-add this node. The goal is to avoid readding a removed
* node for some time. */
dictType clusterNodesBlackListDictType = {
dictSdsCaseHash, /* hash function */
NULL, /* key dup */
NULL, /* val dup */
dictSdsKeyCaseCompare, /* key compare */
dictSdsDestructor, /* key destructor */
NULL /* val destructor */
};
/* Migrate cache dict type. */
dictType migrateCacheDictType = {
dictSdsHash, /* hash function */
NULL, /* key dup */
NULL, /* val dup */
dictSdsKeyCompare, /* key compare */
dictSdsDestructor, /* key destructor */
NULL /* val destructor */
};
/* Replication cached script dict (server.repl_scriptcache_dict).
* Keys are sds SHA1 strings, while values are not used at all in the current
* implementation. */
@@ -713,7 +660,7 @@ void updateDictResizePolicy(void) {
*
* The parameter 'now' is the current time in milliseconds as is passed
* to the function to avoid too many gettimeofday() syscalls. */
int activeExpireCycleTryExpire(redisDb *db, dictEntry *de, long long now) {
int activeExpireCycleTryExpire(redisDb *db, struct dictEntry *de, long long now) {
long long t = dictGetSignedIntegerVal(de);
if (now > t) {
sds key = dictGetKey(de);
@@ -871,10 +818,12 @@ void activeExpireCycle(int type) {
}
}
unsigned int getLRUClock(void) {
return (mstime()/REDIS_LRU_CLOCK_RESOLUTION) & REDIS_LRU_CLOCK_MAX;
void updateLRUClock(void) {
server.lruclock = (server.unixtime/REDIS_LRU_CLOCK_RESOLUTION) &
REDIS_LRU_CLOCK_MAX;
}
/* Add a sample to the operations per second array of samples. */
void trackInstantaneousMetric(int metric, long long current_reading) {
long long t = mstime() - server.inst_metric[metric].last_sample_time;
@@ -917,20 +866,9 @@ int clientsCronHandleTimeout(redisClient *c) {
freeClient(c);
return 1;
} else if (c->flags & REDIS_BLOCKED) {
/* Blocked OPS timeout is handled with milliseconds resolution.
* However note that the actual resolution is limited by
* server.hz. */
mstime_t now_ms = mstime();
if (c->bpop.timeout != 0 && c->bpop.timeout < now_ms) {
/* Handle blocking operation specific timeout. */
replyToBlockedClientTimedOut(c);
unblockClient(c);
} else if (server.cluster_enabled) {
/* Cluster: handle unblock & redirect of clients blocked
* into keys no longer served by this server. */
if (clusterRedirectBlockedClientIfNeeded(c))
unblockClient(c);
if (c->bpop.timeout != 0 && c->bpop.timeout < now) {
addReply(c,shared.nullmultibulk);
unblockClientWaitingData(c);
}
}
return 0;
@@ -1085,18 +1023,19 @@ int serverCron(struct aeEventLoop *eventLoop, long long id, void *clientData) {
server.stat_net_output_bytes);
}
/* We have just REDIS_LRU_BITS bits per object for LRU information.
* So we use an (eventually wrapping) LRU clock.
/* We have just 22 bits per object for LRU information.
* So we use an (eventually wrapping) LRU clock with 10 seconds resolution.
* 2^22 bits with 10 seconds resolution is more or less 1.5 years.
*
* Note that even if the counter wraps it's not a big problem,
* everything will still work but some object will appear younger
* to Redis. However for this to happen a given object should never be
* touched for all the time needed to the counter to wrap, which is
* not likely.
* Note that even if this will wrap after 1.5 years it's not a problem,
* everything will still work but just some object will appear younger
* to Redis. But for this to happen a given object should never be touched
* for 1.5 years.
*
* Note that you can change the resolution altering the
* REDIS_LRU_CLOCK_RESOLUTION define. */
server.lruclock = getLRUClock();
* REDIS_LRU_CLOCK_RESOLUTION define.
*/
updateLRUClock();
/* Record the max memory used since the server was started. */
if (zmalloc_used_memory() > server.stat_peak_memory)
@@ -1231,28 +1170,15 @@ int serverCron(struct aeEventLoop *eventLoop, long long id, void *clientData) {
/* Close clients that need to be closed asynchronous */
freeClientsInAsyncFreeQueue();
/* Clear the paused clients flag if needed. */
clientsArePaused(); /* Don't check return value, just use the side effect. */
/* Replication cron function -- used to reconnect to master and
* to detect transfer failures. */
run_with_period(1000) replicationCron();
/* Run the Redis Cluster cron. */
run_with_period(100) {
if (server.cluster_enabled) clusterCron();
}
/* Run the Sentinel timer if we are in sentinel mode. */
/* Run the sentinel timer if we are in sentinel mode. */
run_with_period(100) {
if (server.sentinel_mode) sentinelTimer();
}
/* Cleanup expired MIGRATE cached sockets. */
run_with_period(1000) {
migrateCloseTimedoutSockets();
}
server.cronloops++;
return 1000/server.hz;
}
@@ -1262,41 +1188,29 @@ int serverCron(struct aeEventLoop *eventLoop, long long id, void *clientData) {
* for ready file descriptors. */
void beforeSleep(struct aeEventLoop *eventLoop) {
REDIS_NOTUSED(eventLoop);
/* Call the Redis Cluster before sleep function. Note that this function
* may change the state of Redis Cluster (from ok to fail or vice versa),
* so it's a good idea to call it before serving the unblocked clients
* later in this function. */
if (server.cluster_enabled) clusterBeforeSleep();
listNode *ln;
redisClient *c;
/* Run a fast expire cycle (the called function will return
* ASAP if a fast cycle is not needed). */
if (server.active_expire_enabled && server.masterhost == NULL)
activeExpireCycle(ACTIVE_EXPIRE_CYCLE_FAST);
/* Send all the slaves an ACK request if at least one client blocked
* during the previous event loop iteration. */
if (server.get_ack_from_slaves) {
robj *argv[3];
argv[0] = createStringObject("REPLCONF",8);
argv[1] = createStringObject("GETACK",6);
argv[2] = createStringObject("*",1); /* Not used argument. */
replicationFeedSlaves(server.slaves, server.slaveseldb, argv, 3);
decrRefCount(argv[0]);
decrRefCount(argv[1]);
decrRefCount(argv[2]);
server.get_ack_from_slaves = 0;
}
/* Unblock all the clients blocked for synchronous replication
* in WAIT. */
if (listLength(server.clients_waiting_acks))
processClientsWaitingReplicas();
/* Try to process pending commands for clients that were just unblocked. */
if (listLength(server.unblocked_clients))
processUnblockedClients();
while (listLength(server.unblocked_clients)) {
ln = listFirst(server.unblocked_clients);
redisAssert(ln != NULL);
c = ln->value;
listDelNode(server.unblocked_clients,ln);
c->flags &= ~REDIS_UNBLOCKED;
/* Process remaining data in the input buffer. */
if (c->querybuf && sdslen(c->querybuf) > 0) {
server.current_client = c;
processInputBuffer(c);
server.current_client = NULL;
}
}
/* Write the AOF buffer on disk */
flushAppendOnlyFile(0);
@@ -1350,8 +1264,6 @@ void createSharedObjects(void) {
"-EXECABORT Transaction discarded because of previous errors.\r\n"));
shared.noreplicaserr = createObject(REDIS_STRING,sdsnew(
"-NOREPLICAS Not enough good slaves to write.\r\n"));
shared.busykeyerr = createObject(REDIS_STRING,sdsnew(
"-BUSYKEY Target key name already exists.\r\n"));
shared.space = createObject(REDIS_STRING,sdsnew(" "));
shared.colon = createObject(REDIS_STRING,sdsnew(":"));
shared.plus = createObject(REDIS_STRING,sdsnew("+"));
@@ -1466,21 +1378,14 @@ void initServerConfig(void) {
server.repl_timeout = REDIS_REPL_TIMEOUT;
server.repl_min_slaves_to_write = REDIS_DEFAULT_MIN_SLAVES_TO_WRITE;
server.repl_min_slaves_max_lag = REDIS_DEFAULT_MIN_SLAVES_MAX_LAG;
server.cluster_enabled = 0;
server.cluster_node_timeout = REDIS_CLUSTER_DEFAULT_NODE_TIMEOUT;
server.cluster_migration_barrier = REDIS_CLUSTER_DEFAULT_MIGRATION_BARRIER;
server.cluster_slave_validity_factor = REDIS_CLUSTER_DEFAULT_SLAVE_VALIDITY;
server.cluster_require_full_coverage = REDIS_CLUSTER_DEFAULT_REQUIRE_FULL_COVERAGE;
server.cluster_configfile = zstrdup(REDIS_DEFAULT_CLUSTER_CONFIG_FILE);
server.lua_caller = NULL;
server.lua_time_limit = REDIS_LUA_TIME_LIMIT;
server.lua_client = NULL;
server.lua_timedout = 0;
server.migrate_cached_sockets = dictCreate(&migrateCacheDictType,NULL);
server.next_client_id = 1; /* Client IDs, start from 1 .*/
server.loading_process_events_interval_bytes = (1024*1024*2);
server.lruclock = getLRUClock();
updateLRUClock();
resetServerSaveParams();
appendServerSaveParams(60*60,1); /* save after 1 hour and 1 change */
@@ -1704,7 +1609,7 @@ int listenToPort(int port, int *fds, int *count) {
redisLog(REDIS_WARNING,
"Creating Server TCP listening socket %s:%d: %s",
server.bindaddr[j] ? server.bindaddr[j] : "*",
port, server.neterr);
server.port, server.neterr);
return REDIS_ERR;
}
anetNonBlock(NULL,fds[*count]);
@@ -1740,6 +1645,7 @@ void resetServerStats(void) {
}
server.stat_net_input_bytes = 0;
server.stat_net_output_bytes = 0;
server.aof_delayed_fsync = 0;
}
void initServer(void) {
@@ -1754,7 +1660,6 @@ void initServer(void) {
server.syslog_facility);
}
server.pid = getpid();
server.current_client = NULL;
server.clients = listCreate();
server.clients_to_close = listCreate();
@@ -1763,9 +1668,6 @@ void initServer(void) {
server.slaveseldb = -1; /* Force to emit the first SELECT command. */
server.unblocked_clients = listCreate();
server.ready_keys = listCreate();
server.clients_waiting_acks = listCreate();
server.get_ack_from_slaves = 0;
server.clients_paused = 0;
createSharedObjects();
adjustOpenFilesLimit();
@@ -1783,7 +1685,7 @@ void initServer(void) {
server.sofd = anetUnixServer(server.neterr,server.unixsocket,
server.unixsocketperm, server.tcp_backlog);
if (server.sofd == ANET_ERR) {
redisLog(REDIS_WARNING, "Opening Unix socket: %s", server.neterr);
redisLog(REDIS_WARNING, "Opening socket: %s", server.neterr);
exit(1);
}
anetNonBlock(NULL,server.sofd);
@@ -1802,7 +1704,6 @@ void initServer(void) {
server.db[j].blocking_keys = dictCreate(&keylistDictType,NULL);
server.db[j].ready_keys = dictCreate(&setDictType,NULL);
server.db[j].watched_keys = dictCreate(&keylistDictType,NULL);
server.db[j].eviction_pool = evictionPoolAlloc();
server.db[j].id = j;
server.db[j].avg_ttl = 0;
}
@@ -1873,7 +1774,6 @@ void initServer(void) {
server.maxmemory_policy = REDIS_MAXMEMORY_NO_EVICTION;
}
if (server.cluster_enabled) clusterInit();
replicationScriptCacheInit();
scriptingInit();
slowlogInit();
@@ -1905,7 +1805,6 @@ void populateCommandTable(void) {
case 'l': c->flags |= REDIS_CMD_LOADING; break;
case 't': c->flags |= REDIS_CMD_STALE; break;
case 'M': c->flags |= REDIS_CMD_SKIP_MONITOR; break;
case 'k': c->flags |= REDIS_CMD_ASKING; break;
case 'F': c->flags |= REDIS_CMD_FAST; break;
default: redisPanic("Unsupported command flag"); break;
}
@@ -2158,33 +2057,6 @@ int processCommand(redisClient *c) {
return REDIS_OK;
}
/* If cluster is enabled perform the cluster redirection here.
* However we don't perform the redirection if:
* 1) The sender of this command is our master.
* 2) The command has no key arguments. */
if (server.cluster_enabled &&
!(c->flags & REDIS_MASTER) &&
!(c->flags & REDIS_LUA_CLIENT &&
server.lua_caller->flags & REDIS_MASTER) &&
!(c->cmd->getkeys_proc == NULL && c->cmd->firstkey == 0))
{
int hashslot;
if (server.cluster->state != REDIS_CLUSTER_OK) {
flagTransaction(c);
clusterRedirectClient(c,NULL,0,REDIS_CLUSTER_REDIR_DOWN_STATE);
return REDIS_OK;
} else {
int error_code;
clusterNode *n = getNodeByQuery(c,c->cmd,c->argv,c->argc,&hashslot,&error_code);
if (n == NULL || n != server.cluster->myself) {
flagTransaction(c);
clusterRedirectClient(c,n,hashslot,error_code);
return REDIS_OK;
}
}
}
/* Handle the maxmemory directive.
*
* First we try to free some memory if possible (if there are volatile
@@ -2297,7 +2169,6 @@ int processCommand(redisClient *c) {
addReply(c,shared.queued);
} else {
call(c,REDIS_CALL_FULL);
c->woff = server.master_repl_offset;
if (listLength(server.ready_keys))
handleClientsBlockedOnLists();
}
@@ -2313,8 +2184,6 @@ void closeListeningSockets(int unlink_unix_socket) {
for (j = 0; j < server.ipfd_count; j++) close(server.ipfd[j]);
if (server.sofd != -1) close(server.sofd);
if (server.cluster_enabled)
for (j = 0; j < server.cfd_count; j++) close(server.cfd[j]);
if (unlink_unix_socket && server.unixsocket) {
redisLog(REDIS_NOTICE,"Removing the unix socket file.");
unlink(server.unixsocket); /* don't care if this fails */
@@ -2505,7 +2374,6 @@ void addReplyCommand(redisClient *c, struct redisCommand *cmd) {
flagcount += addReplyCommandFlag(c,cmd,REDIS_CMD_LOADING, "loading");
flagcount += addReplyCommandFlag(c,cmd,REDIS_CMD_STALE, "stale");
flagcount += addReplyCommandFlag(c,cmd,REDIS_CMD_SKIP_MONITOR, "skip_monitor");
flagcount += addReplyCommandFlag(c,cmd,REDIS_CMD_ASKING, "asking");
flagcount += addReplyCommandFlag(c,cmd,REDIS_CMD_FAST, "fast");
if (cmd->getkeys_proc) {
addReplyStatus(c, "movablekeys");
@@ -2539,24 +2407,6 @@ void commandCommand(redisClient *c) {
}
} else if (!strcasecmp(c->argv[1]->ptr, "count") && c->argc == 2) {
addReplyLongLong(c, dictSize(server.commands));
} else if (!strcasecmp(c->argv[1]->ptr,"getkeys") && c->argc >= 3) {
struct redisCommand *cmd = lookupCommand(c->argv[2]->ptr);
int *keys, numkeys, j;
if (!cmd) {
addReplyErrorFormat(c,"Invalid command specified");
return;
} else if ((cmd->arity > 0 && cmd->arity != c->argc-2) ||
((c->argc-2) < -cmd->arity))
{
addReplyError(c,"Invalid number of arguments specified for command");
return;
}
keys = getKeysFromCommand(cmd,c->argv+2,c->argc-2,&numkeys);
addReplyMultiBulkLen(c,numkeys);
for (j = 0; j < numkeys; j++) addReplyBulk(c,c->argv[keys[j]+2]);
getKeysFreeResult(keys);
} else {
addReplyError(c, "Unknown subcommand or wrong number of arguments.");
return;
@@ -2619,8 +2469,7 @@ sds genRedisInfoString(char *section) {
static struct utsname name;
char *mode;
if (server.cluster_enabled) mode = "cluster";
else if (server.sentinel_mode) mode = "sentinel";
if (server.sentinel_mode) mode = "sentinel";
else mode = "standalone";
if (sections++) info = sdscat(info,"\r\n");
@@ -2832,8 +2681,7 @@ sds genRedisInfoString(char *section) {
"keyspace_misses:%lld\r\n"
"pubsub_channels:%ld\r\n"
"pubsub_patterns:%lu\r\n"
"latest_fork_usec:%lld\r\n"
"migrate_cached_sockets:%ld\r\n",
"latest_fork_usec:%lld\r\n",
server.stat_numconnections,
server.stat_numcommands,
getInstantaneousMetric(REDIS_METRIC_COMMAND),
@@ -2851,8 +2699,7 @@ sds genRedisInfoString(char *section) {
server.stat_keyspace_misses,
dictSize(server.pubsub_channels),
listLength(server.pubsub_patterns),
server.stat_fork_time,
dictSize(server.migrate_cached_sockets));
server.stat_fork_time);
}
/* Replication */
@@ -3004,15 +2851,6 @@ sds genRedisInfoString(char *section) {
}
}
/* Cluster */
if (allsections || defsections || !strcasecmp(section,"cluster")) {
if (sections++) info = sdscat(info,"\r\n");
info = sdscatprintf(info,
"# Cluster\r\n"
"cluster_enabled:%d\r\n",
server.cluster_enabled);
}
/* Key space */
if (allsections || defsections || !strcasecmp(section,"keyspace")) {
if (sections++) info = sdscat(info,"\r\n");
@@ -3057,9 +2895,8 @@ void monitorCommand(redisClient *c) {
/* ============================ Maxmemory directive ======================== */
/* freeMemoryIfNeeded() gets called when 'maxmemory' is set on the config
* file to limit the max memory used by the server, before processing a
* command.
/* This function gets called when 'maxmemory' is set on the config file to limit
* the max memory used by the server, before processing a command.
*
* The goal of the function is to free enough memory to keep Redis under the
* configured memory limit.
@@ -3072,121 +2909,11 @@ void monitorCommand(redisClient *c) {
* function returns REDIS_OK, otherwise REDIS_ERR is returned, and the caller
* should block the execution of commands that will result in more memory
* used by the server.
*
* ------------------------------------------------------------------------
*
* LRU approximation algorithm
*
* Redis uses an approximation of the LRU algorithm that runs in constant
* memory. Every time there is a key to expire, we sample N keys (with
* N very small, usually in around 5) to populate a pool of best keys to
* evict of M keys (the pool size is defined by REDIS_EVICTION_POOL_SIZE).
*
* The N keys sampled are added in the pool of good keys to expire (the one
* with an old access time) if they are better than one of the current keys
* in the pool.
*
* After the pool is populated, the best key we have in the pool is expired.
* However note that we don't remove keys from the pool when they are deleted
* so the pool may contain keys that no longer exist.
*
* When we try to evict a key, and all the entries in the pool don't exist
* we populate it again. This time we'll be sure that the pool has at least
* one key that can be evicted, if there is at least one key that can be
* evicted in the whole database. */
/* Create a new eviction pool. */
struct evictionPoolEntry *evictionPoolAlloc(void) {
struct evictionPoolEntry *ep;
int j;
ep = zmalloc(sizeof(*ep)*REDIS_EVICTION_POOL_SIZE);
for (j = 0; j < REDIS_EVICTION_POOL_SIZE; j++) {
ep[j].idle = 0;
ep[j].key = NULL;
}
return ep;
}
/* This is an helper function for freeMemoryIfNeeded(), it is used in order
* to populate the evictionPool with a few entries every time we want to
* expire a key. Keys with idle time smaller than one of the current
* keys are added. Keys are always added if there are free entries.
*
* We insert keys on place in ascending order, so keys with the smaller
* idle time are on the left, and keys with the higher idle time on the
* right. */
#define EVICTION_SAMPLES_ARRAY_SIZE 16
void evictionPoolPopulate(dict *sampledict, dict *keydict, struct evictionPoolEntry *pool) {
int j, k, count;
dictEntry *_samples[EVICTION_SAMPLES_ARRAY_SIZE];
dictEntry **samples;
/* Try to use a static buffer: this function is a big hit...
* Note: it was actually measured that this helps. */
if (server.maxmemory_samples <= EVICTION_SAMPLES_ARRAY_SIZE) {
samples = _samples;
} else {
samples = zmalloc(sizeof(samples[0])*server.maxmemory_samples);
}
count = dictGetSomeKeys(sampledict,samples,server.maxmemory_samples);
for (j = 0; j < count; j++) {
unsigned long long idle;
sds key;
robj *o;
dictEntry *de;
de = samples[j];
key = dictGetKey(de);
/* If the dictionary we are sampling from is not the main
* dictionary (but the expires one) we need to lookup the key
* again in the key dictionary to obtain the value object. */
if (sampledict != keydict) de = dictFind(keydict, key);
o = dictGetVal(de);
idle = estimateObjectIdleTime(o);
/* Insert the element inside the pool.
* First, find the first empty bucket or the first populated
* bucket that has an idle time smaller than our idle time. */
k = 0;
while (k < REDIS_EVICTION_POOL_SIZE &&
pool[k].key &&
pool[k].idle < idle) k++;
if (k == 0 && pool[REDIS_EVICTION_POOL_SIZE-1].key != NULL) {
/* Can't insert if the element is < the worst element we have
* and there are no empty buckets. */
continue;
} else if (k < REDIS_EVICTION_POOL_SIZE && pool[k].key == NULL) {
/* Inserting into empty position. No setup needed before insert. */
} else {
/* Inserting in the middle. Now k points to the first element
* greater than the element to insert. */
if (pool[REDIS_EVICTION_POOL_SIZE-1].key == NULL) {
/* Free space on the right? Insert at k shifting
* all the elements from k to end to the right. */
memmove(pool+k+1,pool+k,
sizeof(pool[0])*(REDIS_EVICTION_POOL_SIZE-k-1));
} else {
/* No free space on right? Insert at k-1 */
k--;
/* Shift all elements on the left of k (included) to the
* left, so we discard the element with smaller idle time. */
sdsfree(pool[0].key);
memmove(pool,pool+1,sizeof(pool[0])*k);
}
}
pool[k].key = sdsdup(key);
pool[k].idle = idle;
}
if (samples != _samples) zfree(samples);
}
*/
int freeMemoryIfNeeded(void) {
size_t mem_used, mem_tofree, mem_freed;
int slaves = listLength(server.slaves);
mstime_t latency, eviction_latency;
mstime_t latency;
/* Remove the size of slaves output buffers and AOF buffer from the
* count of used memory. */
@@ -3226,7 +2953,7 @@ int freeMemoryIfNeeded(void) {
for (j = 0; j < server.dbnum; j++) {
long bestval = 0; /* just to prevent warning */
sds bestkey = NULL;
dictEntry *de;
struct dictEntry *de;
redisDb *db = server.db+j;
dict *dict;
@@ -3251,34 +2978,24 @@ int freeMemoryIfNeeded(void) {
else if (server.maxmemory_policy == REDIS_MAXMEMORY_ALLKEYS_LRU ||
server.maxmemory_policy == REDIS_MAXMEMORY_VOLATILE_LRU)
{
struct evictionPoolEntry *pool = db->eviction_pool;
for (k = 0; k < server.maxmemory_samples; k++) {
sds thiskey;
long thisval;
robj *o;
while(bestkey == NULL) {
evictionPoolPopulate(dict, db->dict, db->eviction_pool);
/* Go backward from best to worst element to evict. */
for (k = REDIS_EVICTION_POOL_SIZE-1; k >= 0; k--) {
if (pool[k].key == NULL) continue;
de = dictFind(dict,pool[k].key);
de = dictGetRandomKey(dict);
thiskey = dictGetKey(de);
/* When policy is volatile-lru we need an additional lookup
* to locate the real key, as dict is set to db->expires. */
if (server.maxmemory_policy == REDIS_MAXMEMORY_VOLATILE_LRU)
de = dictFind(db->dict, thiskey);
o = dictGetVal(de);
thisval = estimateObjectIdleTime(o);
/* Remove the entry from the pool. */
sdsfree(pool[k].key);
/* Shift all elements on its right to left. */
memmove(pool+k,pool+k+1,
sizeof(pool[0])*(REDIS_EVICTION_POOL_SIZE-k-1));
/* Clear the element on the right which is empty
* since we shifted one position to the left. */
pool[REDIS_EVICTION_POOL_SIZE-1].key = NULL;
pool[REDIS_EVICTION_POOL_SIZE-1].idle = 0;
/* If the key exists, is our pick. Otherwise it is
* a ghost and we need to try the next element. */
if (de) {
bestkey = dictGetKey(de);
break;
} else {
/* Ghost... */
continue;
}
/* Higher idle time is better candidate for deletion */
if (bestkey == NULL || thisval > bestval) {
bestkey = thiskey;
bestval = thisval;
}
}
}
@@ -3317,11 +3034,7 @@ int freeMemoryIfNeeded(void) {
* AOF and Output buffer memory will be freed eventually so
* we only care about memory used by the key space. */
delta = (long long) zmalloc_used_memory();
latencyStartMonitor(eviction_latency);
dbDelete(db,keyobj);
latencyEndMonitor(eviction_latency);
latencyAddSampleIfNeeded("eviction-del",eviction_latency);
latencyRemoveNestedEvent(latency,eviction_latency);
delta -= (long long) zmalloc_used_memory();
mem_freed += delta;
server.stat_evictedkeys++;
@@ -3432,11 +3145,9 @@ void usage(void) {
void redisAsciiArt(void) {
#include "asciilogo.h"
char *buf = zmalloc(1024*16);
char *mode;
char *mode = "stand alone";
if (server.cluster_enabled) mode = "cluster";
else if (server.sentinel_mode) mode = "sentinel";
else mode = "standalone";
if (server.sentinel_mode) mode = "sentinel";
if (server.syslog_enabled) {
redisLog(REDIS_NOTICE,
@@ -3553,15 +3264,10 @@ void redisOutOfMemoryHandler(size_t allocation_size) {
void redisSetProcTitle(char *title) {
#ifdef USE_SETPROCTITLE
char *server_mode = "";
if (server.cluster_enabled) server_mode = " [cluster]";
else if (server.sentinel_mode) server_mode = " [sentinel]";
setproctitle("%s %s:%d%s",
setproctitle("%s %s:%d",
title,
server.bindaddr_count ? server.bindaddr[0] : "*",
server.port,
server_mode);
server.port);
#else
REDIS_NOTUSED(title);
#endif
@@ -3660,14 +3366,6 @@ int main(int argc, char **argv) {
#endif
checkTcpBacklogSettings();
loadDataFromDisk();
if (server.cluster_enabled) {
if (verifyClusterConfigWithData() == REDIS_ERR) {
redisLog(REDIS_WARNING,
"You can't have keys in a DB different than DB 0 when in "
"Cluster mode. Exiting.");
exit(1);
}
}
if (server.ipfd_count > 0)
redisLog(REDIS_NOTICE,"The server is now ready to accept connections on port %d", server.port);
if (server.sofd > 0)
src/redis.h
+48 -139
View File
@@ -116,7 +116,7 @@ typedef long long mstime_t; /* millisecond time type. */
#define REDIS_DEFAULT_SLAVE_READ_ONLY 1
#define REDIS_DEFAULT_REPL_DISABLE_TCP_NODELAY 0
#define REDIS_DEFAULT_MAXMEMORY 0
#define REDIS_DEFAULT_MAXMEMORY_SAMPLES 5
#define REDIS_DEFAULT_MAXMEMORY_SAMPLES 3
#define REDIS_DEFAULT_AOF_FILENAME "appendonly.aof"
#define REDIS_DEFAULT_AOF_NO_FSYNC_ON_REWRITE 0
#define REDIS_DEFAULT_AOF_LOAD_TRUNCATED 1
@@ -124,7 +124,7 @@ typedef long long mstime_t; /* millisecond time type. */
#define REDIS_DEFAULT_AOF_REWRITE_INCREMENTAL_FSYNC 1
#define REDIS_DEFAULT_MIN_SLAVES_TO_WRITE 0
#define REDIS_DEFAULT_MIN_SLAVES_MAX_LAG 10
#define REDIS_IP_STR_LEN 46 /* INET6_ADDRSTRLEN is 46, but we need to be sure */
#define REDIS_IP_STR_LEN INET6_ADDRSTRLEN
#define REDIS_PEER_ID_LEN (REDIS_IP_STR_LEN+32) /* Must be enough for ip:port */
#define REDIS_BINDADDR_MAX 16
#define REDIS_MIN_RESERVED_FDS 32
@@ -194,7 +194,6 @@ typedef long long mstime_t; /* millisecond time type. */
#define REDIS_ENCODING_ZIPLIST 5 /* Encoded as ziplist */
#define REDIS_ENCODING_INTSET 6 /* Encoded as intset */
#define REDIS_ENCODING_SKIPLIST 7 /* Encoded as skiplist */
#define REDIS_ENCODING_EMBSTR 8 /* Embedded sds string encoding */
/* Defines related to the dump file format. To store 32 bits lengths for short
* keys requires a lot of space, so we check the most significant 2 bits of
@@ -250,12 +249,6 @@ typedef long long mstime_t; /* millisecond time type. */
#define REDIS_READONLY (1<<17) /* Cluster client is in read-only state. */
#define REDIS_PUBSUB (1<<18) /* Client is in Pub/Sub mode. */
/* Client block type (btype field in client structure)
* if REDIS_BLOCKED flag is set. */
#define REDIS_BLOCKED_NONE 0 /* Not blocked, no REDIS_BLOCKED flag set. */
#define REDIS_BLOCKED_LIST 1 /* BLPOP & co. */
#define REDIS_BLOCKED_WAIT 2 /* WAIT for synchronous replication. */
/* Client request types */
#define REDIS_REQ_INLINE 1
#define REDIS_REQ_MULTIBULK 2
@@ -267,22 +260,35 @@ typedef long long mstime_t; /* millisecond time type. */
#define REDIS_CLIENT_TYPE_PUBSUB 2 /* Clients subscribed to PubSub channels. */
#define REDIS_CLIENT_TYPE_COUNT 3
/* Slave replication state - from the point of view of the slave. */
/* Slave replication state. Used in server.repl_state for slaves to remember
* what to do next. */
#define REDIS_REPL_NONE 0 /* No active replication */
#define REDIS_REPL_CONNECT 1 /* Must connect to master */
#define REDIS_REPL_CONNECTING 2 /* Connecting to master */
#define REDIS_REPL_RECEIVE_PONG 3 /* Wait for PING reply */
#define REDIS_REPL_TRANSFER 4 /* Receiving .rdb from master */
#define REDIS_REPL_CONNECTED 5 /* Connected to master */
#define REDIS_REPL_SEND_AUTH 4 /* Send AUTH to master */
#define REDIS_REPL_RECEIVE_AUTH 5 /* Wait for AUTH reply */
#define REDIS_REPL_SEND_PORT 6 /* Send REPLCONF listening-port */
#define REDIS_REPL_RECEIVE_PORT 7 /* Wait for REPLCONF reply */
#define REDIS_REPL_SEND_CAPA 8 /* Send REPLCONF capa */
#define REDIS_REPL_RECEIVE_CAPA 9 /* Wait for REPLCONF reply */
#define REDIS_REPL_SEND_PSYNC 10 /* Send PSYNC */
#define REDIS_REPL_RECEIVE_PSYNC 11 /* Wait for PSYNC reply */
#define REDIS_REPL_TRANSFER 12 /* Receiving .rdb from master */
#define REDIS_REPL_CONNECTED 13 /* Connected to master */
/* Slave replication state - from the point of view of the master.
/* State of slaves from the POV of the master. Used in client->replstate.
* In SEND_BULK and ONLINE state the slave receives new updates
* in its output queue. In the WAIT_BGSAVE state instead the server is waiting
* to start the next background saving in order to send updates to it. */
#define REDIS_REPL_WAIT_BGSAVE_START 6 /* We need to produce a new RDB file. */
#define REDIS_REPL_WAIT_BGSAVE_END 7 /* Waiting RDB file creation to finish. */
#define REDIS_REPL_SEND_BULK 8 /* Sending RDB file to slave. */
#define REDIS_REPL_ONLINE 9 /* RDB file transmitted, sending just updates. */
#define REDIS_REPL_WAIT_BGSAVE_START 14 /* We need to produce a new RDB file. */
#define REDIS_REPL_WAIT_BGSAVE_END 15 /* Waiting RDB file creation to finish. */
#define REDIS_REPL_SEND_BULK 16 /* Sending RDB file to slave. */
#define REDIS_REPL_ONLINE 17 /* RDB file transmitted, sending just updates. */
/* Slave capabilities. */
#define SLAVE_CAPA_NONE 0
#define SLAVE_CAPA_EOF (1<<0) /* Can parse the RDB EOF streaming format. */
/* Synchronous read timeout - slave side */
#define REDIS_REPL_SYNCIO_TIMEOUT 5
@@ -341,7 +347,7 @@ typedef long long mstime_t; /* millisecond time type. */
#define REDIS_MAXMEMORY_ALLKEYS_LRU 3
#define REDIS_MAXMEMORY_ALLKEYS_RANDOM 4
#define REDIS_MAXMEMORY_NO_EVICTION 5
#define REDIS_DEFAULT_MAXMEMORY_POLICY REDIS_MAXMEMORY_NO_EVICTION
#define REDIS_DEFAULT_MAXMEMORY_POLICY REDIS_MAXMEMORY_VOLATILE_LRU
/* Scripting */
#define REDIS_LUA_TIME_LIMIT 5000 /* milliseconds */
@@ -408,7 +414,7 @@ typedef long long mstime_t; /* millisecond time type. */
/* The actual Redis Object */
#define REDIS_LRU_BITS 24
#define REDIS_LRU_CLOCK_MAX ((1<<REDIS_LRU_BITS)-1) /* Max value of obj->lru */
#define REDIS_LRU_CLOCK_RESOLUTION 1000 /* LRU clock resolution in ms */
#define REDIS_LRU_CLOCK_RESOLUTION 1 /* LRU clock resolution in seconds */
typedef struct redisObject {
unsigned type:4;
unsigned encoding:4;
@@ -417,12 +423,6 @@ typedef struct redisObject {
void *ptr;
} robj;
/* Macro used to obtain the current LRU clock.
* If the current resolution is lower than the frequency we refresh the
* LRU clock (as it should be in production servers) we return the
* precomputed value, otherwise we need to resort to a function call. */
#define LRU_CLOCK() ((1000/server.hz <= REDIS_LRU_CLOCK_RESOLUTION) ? server.lruclock : getLRUClock())
/* Macro used to initialize a Redis object allocated on the stack.
* Note that this macro is taken near the structure definition to make sure
* we'll update it when the structure is changed, to avoid bugs like
@@ -434,30 +434,13 @@ typedef struct redisObject {
_var.ptr = _ptr; \
} while(0);
/* To improve the quality of the LRU approximation we take a set of keys
* that are good candidate for eviction across freeMemoryIfNeeded() calls.
*
* Entries inside the eviciton pool are taken ordered by idle time, putting
* greater idle times to the right (ascending order).
*
* Empty entries have the key pointer set to NULL. */
#define REDIS_EVICTION_POOL_SIZE 16
struct evictionPoolEntry {
unsigned long long idle; /* Object idle time. */
sds key; /* Key name. */
};
/* Redis database representation. There are multiple databases identified
* by integers from 0 (the default database) up to the max configured
* database. The database number is the 'id' field in the structure. */
typedef struct redisDb {
dict *dict; /* The keyspace for this DB */
dict *expires; /* Timeout of keys with a timeout set */
dict *blocking_keys; /* Keys with clients waiting for data (BLPOP) */
dict *ready_keys; /* Blocked keys that received a PUSH */
dict *watched_keys; /* WATCHED keys for MULTI/EXEC CAS */
struct evictionPoolEntry *eviction_pool; /* Eviction pool of keys */
int id; /* Database ID */
int id;
long long avg_ttl; /* Average TTL, just for stats */
} redisDb;
@@ -475,22 +458,13 @@ typedef struct multiState {
time_t minreplicas_timeout; /* MINREPLICAS timeout as unixtime. */
} multiState;
/* This structure holds the blocking operation state for a client.
* The fields used depend on client->btype. */
typedef struct blockingState {
/* Generic fields. */
mstime_t timeout; /* Blocking operation timeout. If UNIX current time
* is > timeout then the operation timed out. */
/* REDIS_BLOCK_LIST */
dict *keys; /* The keys we are waiting to terminate a blocking
* operation such as BLPOP. Otherwise NULL. */
time_t timeout; /* Blocking operation timeout. If UNIX current time
* is > timeout then the operation timed out. */
robj *target; /* The key that should receive the element,
* for BRPOPLPUSH. */
/* REDIS_BLOCK_WAIT */
int numreplicas; /* Number of replicas we are waiting for ACK. */
long long reploffset; /* Replication offset to reach. */
} blockingState;
/* The following structure represents a node in the server.ready_keys list,
@@ -543,12 +517,14 @@ typedef struct redisClient {
long long reploff; /* replication offset if this is our master */
long long repl_ack_off; /* replication ack offset, if this is a slave */
long long repl_ack_time;/* replication ack time, if this is a slave */
long long psync_initial_offset; /* FULLRESYNC reply offset other slaves
copying this slave output buffer
should use. */
char replrunid[REDIS_RUN_ID_SIZE+1]; /* master run id if this is a master */
int slave_listening_port; /* As configured with: SLAVECONF listening-port */
int slave_capa; /* Slave capabilities: SLAVE_CAPA_* bitwise OR. */
multiState mstate; /* MULTI/EXEC state */
int btype; /* Type of blocking op if REDIS_BLOCKED. */
blockingState bpop; /* blocking state */
long long woff; /* Last write global replication offset. */
blockingState bpop; /* blocking state */
list *watched_keys; /* Keys WATCHED for MULTI/EXEC CAS */
dict *pubsub_channels; /* channels a client is interested in (SUBSCRIBE) */
list *pubsub_patterns; /* patterns a client is interested in (SUBSCRIBE) */
@@ -570,7 +546,7 @@ struct sharedObjectsStruct {
*emptymultibulk, *wrongtypeerr, *nokeyerr, *syntaxerr, *sameobjecterr,
*outofrangeerr, *noscripterr, *loadingerr, *slowscripterr, *bgsaveerr,
*masterdownerr, *roslaveerr, *execaborterr, *noautherr, *noreplicaserr,
*busykeyerr, *oomerr, *plus, *messagebulk, *pmessagebulk, *subscribebulk,
*oomerr, *plus, *messagebulk, *pmessagebulk, *subscribebulk,
*unsubscribebulk, *psubscribebulk, *punsubscribebulk, *del, *rpop, *lpop,
*lpush, *emptyscan, *minstring, *maxstring,
*select[REDIS_SHARED_SELECT_CMDS],
@@ -637,8 +613,6 @@ typedef struct redisOpArray {
* Global server state
*----------------------------------------------------------------------------*/
struct clusterState;
/* AIX defines hz to __hz, we don't use this define and in order to allow
* Redis build on AIX we need to undef it. */
#ifdef _AIX
@@ -647,7 +621,6 @@ struct clusterState;
struct redisServer {
/* General */
pid_t pid; /* Main process pid. */
char *configfile; /* Absolute config file path, or NULL */
int hz; /* serverCron() calls frequency in hertz */
redisDb *db;
@@ -673,16 +646,11 @@ struct redisServer {
int ipfd[REDIS_BINDADDR_MAX]; /* TCP socket file descriptors */
int ipfd_count; /* Used slots in ipfd[] */
int sofd; /* Unix socket file descriptor */
int cfd[REDIS_BINDADDR_MAX];/* Cluster bus listening socket */
int cfd_count; /* Used slots in cfd[] */
list *clients; /* List of active clients */
list *clients_to_close; /* Clients to close asynchronously */
list *slaves, *monitors; /* List of slaves and MONITORs */
redisClient *current_client; /* Current client, only used on crash report */
int clients_paused; /* True if clients are currently paused */
mstime_t clients_pause_end_time; /* Time when we undo clients_paused */
char neterr[ANET_ERR_LEN]; /* Error buffer for anet.c */
dict *migrate_cached_sockets;/* MIGRATE cached sockets */
uint64_t next_client_id; /* Next client unique ID. Incremental. */
/* RDB / AOF loading information */
int loading; /* We are loading data from disk if true */
@@ -757,16 +725,6 @@ struct redisServer {
int aof_last_write_status; /* REDIS_OK or REDIS_ERR */
int aof_last_write_errno; /* Valid if aof_last_write_status is ERR */
int aof_load_truncated; /* Don't stop on unexpected AOF EOF. */
/* AOF pipes used to communicate between parent and child during rewrite. */
int aof_pipe_write_data_to_child;
int aof_pipe_read_data_from_parent;
int aof_pipe_write_ack_to_parent;
int aof_pipe_read_ack_from_child;
int aof_pipe_write_ack_to_child;
int aof_pipe_read_ack_from_parent;
int aof_stop_sending_diff; /* If true stop sending accumulated diffs
to child process. */
sds aof_child_diff; /* AOF diff accumulator child side. */
/* RDB persistence */
long long dirty; /* Changes to DB from the last save */
long long dirty_before_bgsave; /* Used to restore dirty on failed BGSAVE */
@@ -838,9 +796,6 @@ struct redisServer {
dict *repl_scriptcache_dict; /* SHA1 all slaves are aware of. */
list *repl_scriptcache_fifo; /* First in, first out LRU eviction. */
unsigned int repl_scriptcache_size; /* Max number of elements. */
/* Synchronous replication. */
list *clients_waiting_acks; /* Clients waiting in WAIT command. */
int get_ack_from_slaves; /* If true we send REPLCONF GETACK. */
/* Limits */
unsigned int maxclients; /* Max number of simultaneous clients */
unsigned long long maxmemory; /* Max number of memory bytes to use */
@@ -872,15 +827,6 @@ struct redisServer {
list *pubsub_patterns; /* A list of pubsub_patterns */
int notify_keyspace_events; /* Events to propagate via Pub/Sub. This is an
xor of REDIS_NOTIFY... flags. */
/* Cluster */
int cluster_enabled; /* Is cluster enabled? */
mstime_t cluster_node_timeout; /* Cluster node timeout. */
char *cluster_configfile; /* Cluster auto-generated config file name. */
struct clusterState *cluster; /* State of the cluster */
int cluster_migration_barrier; /* Cluster replicas migration barrier. */
int cluster_slave_validity_factor; /* Slave max data age for failover. */
int cluster_require_full_coverage; /* If true, put the cluster down if
there is at least an uncovered slot. */
/* Scripting */
lua_State *lua; /* The Lua interpreter. We use just one for all clients */
redisClient *lua_client; /* The "fake client" to query Redis from Lua */
@@ -912,15 +858,14 @@ typedef struct pubsubPattern {
} pubsubPattern;
typedef void redisCommandProc(redisClient *c);
typedef int *redisGetKeysProc(struct redisCommand *cmd, robj **argv, int argc, int *numkeys);
typedef int *redisGetKeysProc(struct redisCommand *cmd, robj **argv, int argc, int *numkeys, int flags);
struct redisCommand {
char *name;
redisCommandProc *proc;
int arity;
char *sflags; /* Flags as string representation, one char per flag. */
int flags; /* The actual flags, obtained from the 'sflags' field. */
/* Use a function to determine keys arguments in a command line.
* Used for Redis Cluster redirect. */
/* Use a function to determine keys arguments in a command line. */
redisGetKeysProc *getkeys_proc;
/* What keys should be loaded in background when calling this command? */
int firstkey; /* The first argument that's a key (0 = no keys) */
@@ -996,8 +941,6 @@ extern struct redisServer server;
extern struct sharedObjectsStruct shared;
extern dictType setDictType;
extern dictType zsetDictType;
extern dictType clusterNodesDictType;
extern dictType clusterNodesBlackListDictType;
extern dictType dbDictType;
extern dictType shaScriptObjectDictType;
extern double R_Zero, R_PosInf, R_NegInf, R_Nan;
@@ -1060,9 +1003,6 @@ int getClientTypeByName(char *name);
char *getClientTypeName(int class);
void flushSlavesOutputBuffers(void);
void disconnectSlaves(void);
int listenToPort(int port, int *fds, int *count);
void pauseClients(mstime_t duration);
int clientsArePaused(void);
int processEventsWhileBlocked(void);
#ifdef __GNUC__
@@ -1115,8 +1055,6 @@ void freeZsetObject(robj *o);
void freeHashObject(robj *o);
robj *createObject(int type, void *ptr);
robj *createStringObject(char *ptr, size_t len);
robj *createRawStringObject(char *ptr, size_t len);
robj *createEmbeddedStringObject(char *ptr, size_t len);
robj *dupStringObject(robj *o);
int isObjectRepresentableAsLongLong(robj *o, long long *llongval);
robj *tryObjectEncoding(robj *o);
@@ -1142,8 +1080,7 @@ char *strEncoding(int encoding);
int compareStringObjects(robj *a, robj *b);
int collateStringObjects(robj *a, robj *b);
int equalStringObjects(robj *a, robj *b);
unsigned long long estimateObjectIdleTime(robj *o);
#define sdsEncodedObject(objptr) (objptr->encoding == REDIS_ENCODING_RAW || objptr->encoding == REDIS_ENCODING_EMBSTR)
unsigned long estimateObjectIdleTime(robj *o);
/* Synchronous I/O with timeout */
ssize_t syncWrite(int fd, char *ptr, ssize_t size, long long timeout);
@@ -1158,19 +1095,17 @@ void replicationCron(void);
void replicationHandleMasterDisconnection(void);
void replicationCacheMaster(redisClient *c);
void resizeReplicationBacklog(long long newsize);
void replicationSetMaster(char *ip, int port);
void replicationUnsetMaster(void);
void refreshGoodSlavesCount(void);
void replicationScriptCacheInit(void);
void replicationScriptCacheFlush(void);
void replicationScriptCacheAdd(sds sha1);
int replicationScriptCacheExists(sds sha1);
void processClientsWaitingReplicas(void);
void unblockClientWaitingReplicas(redisClient *c);
int replicationCountAcksByOffset(long long offset);
void replicationSetMaster(char *ip, int port);
void replicationUnsetMaster(void);
void replicationSendNewlineToMaster(void);
long long replicationGetSlaveOffset(void);
char *replicationGetSlaveName(redisClient *c);
long long getPsyncInitialOffset(void);
int replicationSetupSlaveForFullResync(redisClient *slave, long long offset);
/* Generic persistence functions */
void startLoading(FILE *fp);
@@ -1212,13 +1147,11 @@ zskiplistNode *zslInsert(zskiplist *zsl, double score, robj *obj);
unsigned char *zzlInsert(unsigned char *zl, robj *ele, double score);
int zslDelete(zskiplist *zsl, double score, robj *obj);
zskiplistNode *zslFirstInRange(zskiplist *zsl, zrangespec *range);
zskiplistNode *zslLastInRange(zskiplist *zsl, zrangespec *range);
double zzlGetScore(unsigned char *sptr);
void zzlNext(unsigned char *zl, unsigned char **eptr, unsigned char **sptr);
void zzlPrev(unsigned char *zl, unsigned char **eptr, unsigned char **sptr);
unsigned int zsetLength(robj *zobj);
void zsetConvert(robj *zobj, int encoding);
unsigned long zslGetRank(zskiplist *zsl, double score, robj *o);
/* Core functions */
int freeMemoryIfNeeded(void);
@@ -1250,7 +1183,6 @@ void adjustOpenFilesLimit(void);
void closeListeningSockets(int unlink_unix_socket);
void updateCachedTime(void);
void resetServerStats(void);
unsigned int getLRUClock(void);
/* Set data type */
robj *setTypeCreate(robj *value);
@@ -1327,28 +1259,18 @@ long long emptyDb(void(callback)(void*));
int selectDb(redisClient *c, int id);
void signalModifiedKey(redisDb *db, robj *key);
void signalFlushedDb(int dbid);
unsigned int getKeysInSlot(unsigned int hashslot, robj **keys, unsigned int count);
unsigned int countKeysInSlot(unsigned int hashslot);
unsigned int delKeysInSlot(unsigned int hashslot);
int verifyClusterConfigWithData(void);
unsigned int GetKeysInSlot(unsigned int hashslot, robj **keys, unsigned int count);
void scanGenericCommand(redisClient *c, robj *o, unsigned long cursor);
int parseScanCursorOrReply(redisClient *c, robj *o, unsigned long *cursor);
/* API to get key arguments from commands */
int *getKeysFromCommand(struct redisCommand *cmd, robj **argv, int argc, int *numkeys);
#define REDIS_GETKEYS_ALL 0
#define REDIS_GETKEYS_PRELOAD 1
int *getKeysFromCommand(struct redisCommand *cmd, robj **argv, int argc, int *numkeys, int flags);
void getKeysFreeResult(int *result);
int *zunionInterGetKeys(struct redisCommand *cmd,robj **argv, int argc, int *numkeys);
int *evalGetKeys(struct redisCommand *cmd, robj **argv, int argc, int *numkeys);
int *sortGetKeys(struct redisCommand *cmd, robj **argv, int argc, int *numkeys);
/* Cluster */
void clusterInit(void);
unsigned short crc16(const char *buf, int len);
unsigned int keyHashSlot(char *key, int keylen);
void clusterCron(void);
void clusterPropagatePublish(robj *channel, robj *message);
void migrateCloseTimedoutSockets(void);
void clusterBeforeSleep(void);
int *noPreloadGetKeys(struct redisCommand *cmd,robj **argv, int argc, int *numkeys, int flags);
int *renameGetKeys(struct redisCommand *cmd,robj **argv, int argc, int *numkeys, int flags);
int *zunionInterGetKeys(struct redisCommand *cmd,robj **argv, int argc, int *numkeys, int flags);
/* Sentinel */
void initSentinelConfig(void);
@@ -1360,14 +1282,6 @@ void sentinelIsRunning(void);
/* Scripting */
void scriptingInit(void);
/* Blocked clients */
void processUnblockedClients(void);
void blockClient(redisClient *c, int btype);
void unblockClient(redisClient *c);
void replyToBlockedClientTimedOut(redisClient *c);
int getTimeoutFromObjectOrReply(redisClient *c, robj *object, mstime_t *timeout, int unit);
void disconnectAllBlockedClients(void);
/* Git SHA1 */
char *redisGitSHA1(void);
char *redisGitDirty(void);
@@ -1508,12 +1422,8 @@ void publishCommand(redisClient *c);
void pubsubCommand(redisClient *c);
void watchCommand(redisClient *c);
void unwatchCommand(redisClient *c);
void clusterCommand(redisClient *c);
void restoreCommand(redisClient *c);
void migrateCommand(redisClient *c);
void askingCommand(redisClient *c);
void readonlyCommand(redisClient *c);
void readwriteCommand(redisClient *c);
void dumpCommand(redisClient *c);
void objectCommand(redisClient *c);
void clientCommand(redisClient *c);
@@ -1525,7 +1435,6 @@ void bitopCommand(redisClient *c);
void bitcountCommand(redisClient *c);
void bitposCommand(redisClient *c);
void replconfCommand(redisClient *c);
void waitCommand(redisClient *c);
void pfselftestCommand(redisClient *c);
void pfaddCommand(redisClient *c);
void pfcountCommand(redisClient *c);
src/replication.c
+439 -338
View File
File diff suppressed because it is too large Load Diff
src/scripting.c
+11 -27
View File
@@ -30,7 +30,6 @@
#include "redis.h"
#include "sha1.h"
#include "rand.h"
#include "cluster.h"
#include <lua.h>
#include <lauxlib.h>
@@ -309,7 +308,6 @@ int luaRedisGenericCommand(lua_State *lua, int raise_error) {
luaPushError(lua,"Unknown Redis command called from Lua script");
goto cleanup;
}
c->cmd = cmd;
/* There are commands that are not allowed inside scripts. */
if (cmd->flags & REDIS_CMD_NOSCRIPT) {
@@ -356,24 +354,8 @@ int luaRedisGenericCommand(lua_State *lua, int raise_error) {
if (cmd->flags & REDIS_CMD_RANDOM) server.lua_random_dirty = 1;
if (cmd->flags & REDIS_CMD_WRITE) server.lua_write_dirty = 1;
/* If this is a Redis Cluster node, we need to make sure Lua is not
* trying to access non-local keys, with the exception of commands
* received from our master. */
if (server.cluster_enabled && !(server.lua_caller->flags & REDIS_MASTER)) {
/* Duplicate relevant flags in the lua client. */
c->flags &= ~(REDIS_READONLY|REDIS_ASKING);
c->flags |= server.lua_caller->flags & (REDIS_READONLY|REDIS_ASKING);
if (getNodeByQuery(c,c->cmd,c->argv,c->argc,NULL,NULL) !=
server.cluster->myself)
{
luaPushError(lua,
"Lua script attempted to access a non local key in a "
"cluster node");
goto cleanup;
}
}
/* Run the command */
c->cmd = cmd;
call(c,REDIS_CALL_SLOWLOG | REDIS_CALL_STATS);
/* Convert the result of the Redis command into a suitable Lua type.
@@ -418,8 +400,7 @@ cleanup:
* (we must be the only owner) for us to cache it. */
if (j < LUA_CMD_OBJCACHE_SIZE &&
o->refcount == 1 &&
(o->encoding == REDIS_ENCODING_RAW ||
o->encoding == REDIS_ENCODING_EMBSTR) &&
o->encoding == REDIS_ENCODING_RAW &&
sdslen(o->ptr) <= LUA_CMD_OBJCACHE_MAX_LEN)
{
struct sdshdr *sh = (void*)(((char*)(o->ptr))-(sizeof(struct sdshdr)));
@@ -612,11 +593,12 @@ void scriptingEnableGlobalsProtection(lua_State *lua) {
/* strict.lua from: http://metalua.luaforge.net/src/lib/strict.lua.html.
* Modified to be adapted to Redis. */
s[j++]="local dbg=debug\n";
s[j++]="local mt = {}\n";
s[j++]="setmetatable(_G, mt)\n";
s[j++]="mt.__newindex = function (t, n, v)\n";
s[j++]=" if debug.getinfo(2) then\n";
s[j++]=" local w = debug.getinfo(2, \"S\").what\n";
s[j++]=" if dbg.getinfo(2) then\n";
s[j++]=" local w = dbg.getinfo(2, \"S\").what\n";
s[j++]=" if w ~= \"main\" and w ~= \"C\" then\n";
s[j++]=" error(\"Script attempted to create global variable '\"..tostring(n)..\"'\", 2)\n";
s[j++]=" end\n";
@@ -624,11 +606,12 @@ void scriptingEnableGlobalsProtection(lua_State *lua) {
s[j++]=" rawset(t, n, v)\n";
s[j++]="end\n";
s[j++]="mt.__index = function (t, n)\n";
s[j++]=" if debug.getinfo(2) and debug.getinfo(2, \"S\").what ~= \"C\" then\n";
s[j++]=" if dbg.getinfo(2) and dbg.getinfo(2, \"S\").what ~= \"C\" then\n";
s[j++]=" error(\"Script attempted to access unexisting global variable '\"..tostring(n)..\"'\", 2)\n";
s[j++]=" end\n";
s[j++]=" return rawget(t, n)\n";
s[j++]="end\n";
s[j++]="debug = nil\n";
s[j++]=NULL;
for (j = 0; s[j] != NULL; j++) code = sdscatlen(code,s[j],strlen(s[j]));
@@ -732,10 +715,11 @@ void scriptingInit(void) {
* information about the caller, that's what makes sense from the point
* of view of the user debugging a script. */
{
char *errh_func = "function __redis__err__handler(err)\n"
" local i = debug.getinfo(2,'nSl')\n"
char *errh_func = "local dbg = debug\n"
"function __redis__err__handler(err)\n"
" local i = dbg.getinfo(2,'nSl')\n"
" if i and i.what == 'C' then\n"
" i = debug.getinfo(3,'nSl')\n"
" i = dbg.getinfo(3,'nSl')\n"
" end\n"
" if i then\n"
" return i.source .. ':' .. i.currentline .. ': ' .. err\n"
src/sentinel.c
+63 -4
View File
@@ -397,7 +397,6 @@ struct redisCommand sentinelcmds[] = {
{"publish",sentinelPublishCommand,3,"",0,NULL,0,0,0,0,0},
{"info",sentinelInfoCommand,-1,"",0,NULL,0,0,0,0,0},
{"role",sentinelRoleCommand,1,"l",0,NULL,0,0,0,0,0},
{"client",clientCommand,-2,"rs",0,NULL,0,0,0,0,0},
{"shutdown",shutdownCommand,-1,"",0,NULL,0,0,0,0,0}
};
@@ -1157,7 +1156,9 @@ void sentinelDelFlagsToDictOfRedisInstances(dict *instances, int flags) {
* 1) Remove all slaves.
* 2) Remove all sentinels.
* 3) Remove most of the flags resulting from runtime operations.
* 4) Reset timers to their default value.
* 4) Reset timers to their default value. For example after a reset it will be
* possible to failover again the same master ASAP, without waiting the
* failover timeout delay.
* 5) In the process of doing this undo the failover if in progress.
* 6) Disconnect the connections with the master (will reconnect automatically).
*/
@@ -1180,7 +1181,7 @@ void sentinelResetMaster(sentinelRedisInstance *ri, int flags) {
}
ri->failover_state = SENTINEL_FAILOVER_STATE_NONE;
ri->failover_state_change_time = 0;
ri->failover_start_time = 0;
ri->failover_start_time = 0; /* We can failover again ASAP. */
ri->promoted_slave = NULL;
sdsfree(ri->runid);
sdsfree(ri->slave_master_host);
@@ -1544,7 +1545,7 @@ void rewriteConfigSentinelOption(struct rewriteConfigState *state) {
slave_addr = master->addr;
line = sdscatprintf(sdsempty(),
"sentinel known-slave %s %s %d",
master->name, ri->addr->ip, ri->addr->port);
master->name, slave_addr->ip, slave_addr->port);
rewriteConfigRewriteLine(state,"sentinel",line,1);
}
dictReleaseIterator(di2);
@@ -2614,6 +2615,31 @@ sentinelRedisInstance *sentinelGetMasterByNameOrReplyError(redisClient *c,
return ri;
}
#define SENTINEL_ISQR_OK 0
#define SENTINEL_ISQR_NOQUORUM (1<<0)
#define SENTINEL_ISQR_NOAUTH (1<<1)
int sentinelIsQuorumReachable(sentinelRedisInstance *master, int *usableptr) {
dictIterator *di;
dictEntry *de;
int usable = 1; /* Number of usable Sentinels. Init to 1 to count myself. */
int result = SENTINEL_ISQR_OK;
int voters = dictSize(master->sentinels)+1; /* Known Sentinels + myself. */
di = dictGetIterator(master->sentinels);
while((de = dictNext(di)) != NULL) {
sentinelRedisInstance *ri = dictGetVal(de);
if (ri->flags & (SRI_S_DOWN|SRI_O_DOWN)) continue;
usable++;
}
dictReleaseIterator(di);
if (usable < (int)master->quorum) result |= SENTINEL_ISQR_NOQUORUM;
if (usable < voters/2+1) result |= SENTINEL_ISQR_NOAUTH;
if (usableptr) *usableptr = usable;
return result;
}
void sentinelCommand(redisClient *c) {
if (!strcasecmp(c->argv[1]->ptr,"masters")) {
/* SENTINEL MASTERS */
@@ -2764,16 +2790,49 @@ void sentinelCommand(redisClient *c) {
sentinelEvent(REDIS_WARNING,"+monitor",ri,"%@ quorum %d",ri->quorum);
addReply(c,shared.ok);
}
} else if (!strcasecmp(c->argv[1]->ptr,"flushconfig")) {
if (c->argc != 2) goto numargserr;
sentinelFlushConfig();
addReply(c,shared.ok);
return;
} else if (!strcasecmp(c->argv[1]->ptr,"remove")) {
/* SENTINEL REMOVE <name> */
sentinelRedisInstance *ri;
if (c->argc != 3) goto numargserr;
if ((ri = sentinelGetMasterByNameOrReplyError(c,c->argv[2]))
== NULL) return;
sentinelEvent(REDIS_WARNING,"-monitor",ri,"%@");
dictDelete(sentinel.masters,c->argv[2]->ptr);
sentinelFlushConfig();
addReply(c,shared.ok);
} else if (!strcasecmp(c->argv[1]->ptr,"ckquorum")) {
/* SENTINEL CKQUORUM <name> */
sentinelRedisInstance *ri;
int usable;
if (c->argc != 3) goto numargserr;
if ((ri = sentinelGetMasterByNameOrReplyError(c,c->argv[2]))
== NULL) return;
int result = sentinelIsQuorumReachable(ri,&usable);
if (result == SENTINEL_ISQR_OK) {
addReplySds(c, sdscatfmt(sdsempty(),
"+OK %i usable Sentinels. Quorum and failover authorization "
"can be reached\r\n",usable));
} else {
sds e = sdscatfmt(sdsempty(),
"-NOQUORUM %i usable Sentinels. ",usable);
if (result & SENTINEL_ISQR_NOQUORUM)
e = sdscat(e,"Not enough available Sentinels to reach the"
" specified quorum for this master");
if (result & SENTINEL_ISQR_NOAUTH) {
if (result & SENTINEL_ISQR_NOQUORUM) e = sdscat(e,". ");
e = sdscat(e, "Not enough available Sentinels to reach the"
" majority and authorize a failover");
}
e = sdscat(e,"\r\n");
addReplySds(c,e);
}
} else if (!strcasecmp(c->argv[1]->ptr,"set")) {
if (c->argc < 3 || c->argc % 2 == 0) goto numargserr;
sentinelSetCommand(c);
src/slowlog.c
+1 -1
View File
@@ -63,7 +63,7 @@ slowlogEntry *slowlogCreateEntry(robj **argv, int argc, long long duration) {
} else {
/* Trim too long strings as well... */
if (argv[j]->type == REDIS_STRING &&
sdsEncodedObject(argv[j]) &&
argv[j]->encoding == REDIS_ENCODING_RAW &&
sdslen(argv[j]->ptr) > SLOWLOG_ENTRY_MAX_STRING)
{
sds s = sdsnewlen(argv[j]->ptr, SLOWLOG_ENTRY_MAX_STRING);
src/sort.c
+8 -29
View File
@@ -194,7 +194,6 @@ void sortCommand(redisClient *c) {
int j, dontsort = 0, vectorlen;
int getop = 0; /* GET operation counter */
int int_convertion_error = 0;
int syntax_error = 0;
robj *sortval, *sortby = NULL, *storekey = NULL;
redisSortObject *vector; /* Resulting vector to sort */
@@ -237,8 +236,9 @@ void sortCommand(redisClient *c) {
(getLongFromObjectOrReply(c, c->argv[j+2], &limit_count, NULL)
!= REDIS_OK))
{
syntax_error++;
break;
decrRefCount(sortval);
listRelease(operations);
return;
}
j+=2;
} else if (!strcasecmp(c->argv[j]->ptr,"store") && leftargs >= 1) {
@@ -248,43 +248,22 @@ void sortCommand(redisClient *c) {
sortby = c->argv[j+1];
/* If the BY pattern does not contain '*', i.e. it is constant,
* we don't need to sort nor to lookup the weight keys. */
if (strchr(c->argv[j+1]->ptr,'*') == NULL) {
dontsort = 1;
} else {
/* If BY is specified with a real patter, we can't accept
* it in cluster mode. */
if (server.cluster_enabled) {
addReplyError(c,"BY option of SORT denied in Cluster mode.");
syntax_error++;
break;
}
}
if (strchr(c->argv[j+1]->ptr,'*') == NULL) dontsort = 1;
j++;
} else if (!strcasecmp(c->argv[j]->ptr,"get") && leftargs >= 1) {
if (server.cluster_enabled) {
addReplyError(c,"GET option of SORT denied in Cluster mode.");
syntax_error++;
break;
}
listAddNodeTail(operations,createSortOperation(
REDIS_SORT_GET,c->argv[j+1]));
getop++;
j++;
} else {
decrRefCount(sortval);
listRelease(operations);
addReply(c,shared.syntaxerr);
syntax_error++;
break;
return;
}
j++;
}
/* Handle syntax errors set during options parsing. */
if (syntax_error) {
decrRefCount(sortval);
listRelease(operations);
return;
}
/* When sorting a set with no sort specified, we must sort the output
* so the result is consistent across scripting and replication.
*
@@ -438,7 +417,7 @@ void sortCommand(redisClient *c) {
if (alpha) {
if (sortby) vector[j].u.cmpobj = getDecodedObject(byval);
} else {
if (sdsEncodedObject(byval)) {
if (byval->encoding == REDIS_ENCODING_RAW) {
char *eptr;
vector[j].u.score = strtod(byval->ptr,&eptr);
src/t_hash.c
+4 -1
View File
@@ -43,7 +43,7 @@ void hashTypeTryConversion(robj *o, robj **argv, int start, int end) {
if (o->encoding != REDIS_ENCODING_ZIPLIST) return;
for (i = start; i <= end; i++) {
if (sdsEncodedObject(argv[i]) &&
if (argv[i]->encoding == REDIS_ENCODING_RAW &&
sdslen(argv[i]->ptr) > server.hash_max_ziplist_value)
{
hashTypeConvert(o, REDIS_ENCODING_HT);
@@ -384,12 +384,15 @@ robj *hashTypeCurrentObject(hashTypeIterator *hi, int what) {
} else {
dst = createStringObjectFromLongLong(vll);
}
} else if (hi->encoding == REDIS_ENCODING_HT) {
hashTypeCurrentFromHashTable(hi, what, &dst);
incrRefCount(dst);
} else {
redisPanic("Unknown hash encoding");
}
return dst;
}
src/t_list.c
+38 -14
View File
@@ -38,7 +38,7 @@
* objects are never too long. */
void listTypeTryConversion(robj *subject, robj *value) {
if (subject->encoding != REDIS_ENCODING_ZIPLIST) return;
if (sdsEncodedObject(value) &&
if (value->encoding == REDIS_ENCODING_RAW &&
sdslen(value->ptr) > server.list_max_ziplist_value)
listTypeConvert(subject,REDIS_ENCODING_LINKEDLIST);
}
@@ -232,7 +232,7 @@ void listTypeInsert(listTypeEntry *entry, robj *value, int where) {
int listTypeEqual(listTypeEntry *entry, robj *o) {
listTypeIterator *li = entry->li;
if (li->encoding == REDIS_ENCODING_ZIPLIST) {
redisAssertWithInfo(NULL,o,sdsEncodedObject(o));
redisAssertWithInfo(NULL,o,o->encoding == REDIS_ENCODING_RAW);
return ziplistCompare(entry->zi,o->ptr,sdslen(o->ptr));
} else if (li->encoding == REDIS_ENCODING_LINKEDLIST) {
return equalStringObjects(o,listNodeValue(entry->ln));
@@ -772,7 +772,7 @@ void rpoplpushCommand(redisClient *c) {
/* Set a client in blocking mode for the specified key, with the specified
* timeout */
void blockForKeys(redisClient *c, robj **keys, int numkeys, mstime_t timeout, robj *target) {
void blockForKeys(redisClient *c, robj **keys, int numkeys, time_t timeout, robj *target) {
dictEntry *de;
list *l;
int j;
@@ -802,11 +802,13 @@ void blockForKeys(redisClient *c, robj **keys, int numkeys, mstime_t timeout, ro
}
listAddNodeTail(l,c);
}
blockClient(c,REDIS_BLOCKED_LIST);
/* Mark the client as a blocked client */
c->flags |= REDIS_BLOCKED;
server.bpop_blocked_clients++;
}
/* Unblock a client that's waiting in a blocking operation such as BLPOP.
* You should never call this function directly, but unblockClient() instead. */
/* Unblock a client that's waiting in a blocking operation such as BLPOP */
void unblockClientWaitingData(redisClient *c) {
dictEntry *de;
dictIterator *di;
@@ -834,6 +836,10 @@ void unblockClientWaitingData(redisClient *c) {
decrRefCount(c->bpop.target);
c->bpop.target = NULL;
}
c->flags &= ~REDIS_BLOCKED;
c->flags |= REDIS_UNBLOCKED;
server.bpop_blocked_clients--;
listAddNodeTail(server.unblocked_clients,c);
}
/* If the specified key has clients blocked waiting for list pushes, this
@@ -988,10 +994,10 @@ void handleClientsBlockedOnLists(void) {
if (value) {
/* Protect receiver->bpop.target, that will be
* freed by the next unblockClient()
* freed by the next unblockClientWaitingData()
* call. */
if (dstkey) incrRefCount(dstkey);
unblockClient(receiver);
unblockClientWaitingData(receiver);
if (serveClientBlockedOnList(receiver,
rl->key,dstkey,rl->db,value,
@@ -1024,14 +1030,32 @@ void handleClientsBlockedOnLists(void) {
}
}
int getTimeoutFromObjectOrReply(redisClient *c, robj *object, time_t *timeout) {
long tval;
if (getLongFromObjectOrReply(c,object,&tval,
"timeout is not an integer or out of range") != REDIS_OK)
return REDIS_ERR;
if (tval < 0) {
addReplyError(c,"timeout is negative");
return REDIS_ERR;
}
if (tval > 0) tval += server.unixtime;
*timeout = tval;
return REDIS_OK;
}
/* Blocking RPOP/LPOP */
void blockingPopGenericCommand(redisClient *c, int where) {
robj *o;
mstime_t timeout;
time_t timeout;
int j;
if (getTimeoutFromObjectOrReply(c,c->argv[c->argc-1],&timeout,UNIT_SECONDS)
!= REDIS_OK) return;
if (getTimeoutFromObjectOrReply(c,c->argv[c->argc-1],&timeout) != REDIS_OK)
return;
for (j = 1; j < c->argc-1; j++) {
o = lookupKeyWrite(c->db,c->argv[j]);
@@ -1090,10 +1114,10 @@ void brpopCommand(redisClient *c) {
}
void brpoplpushCommand(redisClient *c) {
mstime_t timeout;
time_t timeout;
if (getTimeoutFromObjectOrReply(c,c->argv[3],&timeout,UNIT_SECONDS)
!= REDIS_OK) return;
if (getTimeoutFromObjectOrReply(c,c->argv[3],&timeout) != REDIS_OK)
return;
robj *key = lookupKeyWrite(c->db, c->argv[1]);
src/t_set.c
+8 -3
View File
@@ -321,7 +321,7 @@ void smoveCommand(redisClient *c) {
/* If srcset and dstset are equal, SMOVE is a no-op */
if (srcset == dstset) {
addReply(c,shared.cone);
addReply(c,setTypeIsMember(srcset,ele) ? shared.cone : shared.czero);
return;
}
@@ -495,8 +495,11 @@ void srandmemberWithCountCommand(redisClient *c) {
if (encoding == REDIS_ENCODING_INTSET) {
retval = dictAdd(d,createStringObjectFromLongLong(llele),NULL);
} else {
} else if (ele->encoding == REDIS_ENCODING_RAW) {
retval = dictAdd(d,dupStringObject(ele),NULL);
} else if (ele->encoding == REDIS_ENCODING_INT) {
retval = dictAdd(d,
createStringObjectFromLongLong((long)ele->ptr),NULL);
}
redisAssert(retval == DICT_OK);
}
@@ -524,8 +527,10 @@ void srandmemberWithCountCommand(redisClient *c) {
encoding = setTypeRandomElement(set,&ele,&llele);
if (encoding == REDIS_ENCODING_INTSET) {
ele = createStringObjectFromLongLong(llele);
} else {
} else if (ele->encoding == REDIS_ENCODING_RAW) {
ele = dupStringObject(ele);
} else if (ele->encoding == REDIS_ENCODING_INT) {
ele = createStringObjectFromLongLong((long)ele->ptr);
}
/* Try to add the object to the dictionary. If it already exists
* free it, otherwise increment the number of objects we have
src/t_string.c
+5 -15
View File
@@ -337,21 +337,11 @@ void incrDecrCommand(redisClient *c, long long incr) {
return;
}
value += incr;
if (o && o->refcount == 1 && o->encoding == REDIS_ENCODING_INT &&
(value < 0 || value >= REDIS_SHARED_INTEGERS) &&
value >= LONG_MIN && value <= LONG_MAX)
{
new = o;
o->ptr = (void*)((long)value);
} else {
new = createStringObjectFromLongLong(value);
if (o) {
dbOverwrite(c->db,c->argv[1],new);
} else {
dbAdd(c->db,c->argv[1],new);
}
}
new = createStringObjectFromLongLong(value);
if (o)
dbOverwrite(c->db,c->argv[1],new);
else
dbAdd(c->db,c->argv[1],new);
signalModifiedKey(c->db,c->argv[1]);
notifyKeyspaceEvent(REDIS_NOTIFY_STRING,"incrby",c->argv[1],c->db->id);
server.dirty++;
src/t_zset.c
+6 -8
View File
@@ -952,7 +952,7 @@ unsigned char *zzlInsertAt(unsigned char *zl, unsigned char *eptr, robj *ele, do
int scorelen;
size_t offset;
redisAssertWithInfo(NULL,ele,sdsEncodedObject(ele));
redisAssertWithInfo(NULL,ele,ele->encoding == REDIS_ENCODING_RAW);
scorelen = d2string(scorebuf,sizeof(scorebuf),score);
if (eptr == NULL) {
zl = ziplistPush(zl,ele->ptr,sdslen(ele->ptr),ZIPLIST_TAIL);
@@ -1710,7 +1710,7 @@ int zuiLongLongFromValue(zsetopval *val) {
if (val->ele->encoding == REDIS_ENCODING_INT) {
val->ell = (long)val->ele->ptr;
val->flags |= OPVAL_VALID_LL;
} else if (sdsEncodedObject(val->ele)) {
} else if (val->ele->encoding == REDIS_ENCODING_RAW) {
if (string2ll(val->ele->ptr,sdslen(val->ele->ptr),&val->ell))
val->flags |= OPVAL_VALID_LL;
} else {
@@ -1745,7 +1745,7 @@ int zuiBufferFromValue(zsetopval *val) {
if (val->ele->encoding == REDIS_ENCODING_INT) {
val->elen = ll2string((char*)val->_buf,sizeof(val->_buf),(long)val->ele->ptr);
val->estr = val->_buf;
} else if (sdsEncodedObject(val->ele)) {
} else if (val->ele->encoding == REDIS_ENCODING_RAW) {
val->elen = sdslen(val->ele->ptr);
val->estr = val->ele->ptr;
} else {
@@ -1970,10 +1970,9 @@ void zunionInterGenericCommand(redisClient *c, robj *dstkey, int op) {
dictAdd(dstzset->dict,tmp,&znode->score);
incrRefCount(tmp); /* added to dictionary */
if (sdsEncodedObject(tmp)) {
if (tmp->encoding == REDIS_ENCODING_RAW)
if (sdslen(tmp->ptr) > maxelelen)
maxelelen = sdslen(tmp->ptr);
}
}
}
zuiClearIterator(&src[0]);
@@ -2009,7 +2008,7 @@ void zunionInterGenericCommand(redisClient *c, robj *dstkey, int op) {
/* Remember the longest single element encountered,
* to understand if it's possible to convert to ziplist
* at the end. */
if (sdsEncodedObject(tmp)) {
if (tmp->encoding == REDIS_ENCODING_RAW) {
if (sdslen(tmp->ptr) > maxelelen)
maxelelen = sdslen(tmp->ptr);
}
@@ -2786,8 +2785,7 @@ void zrankGenericCommand(redisClient *c, int reverse) {
checkType(c,zobj,REDIS_ZSET)) return;
llen = zsetLength(zobj);
redisAssertWithInfo(c,ele,sdsEncodedObject(ele));
redisAssertWithInfo(c,ele,ele->encoding == REDIS_ENCODING_RAW);
if (zobj->encoding == REDIS_ENCODING_ZIPLIST) {
unsigned char *zl = zobj->ptr;
unsigned char *eptr, *sptr;
src/util.c
+13 -58
View File
@@ -38,10 +38,8 @@
#include <sys/time.h>
#include <float.h>
#include <stdint.h>
#include <errno.h>
#include "util.h"
#include "sha1.h"
/* Glob-style pattern matching. */
int stringmatchlen(const char *pattern, int patternLen,
@@ -171,12 +169,11 @@ int stringmatch(const char *pattern, const char *string, int nocase) {
}
/* Convert a string representing an amount of memory into the number of
* bytes, so for instance memtoll("1Gb") will return 1073741824 that is
* bytes, so for instance memtoll("1Gi") will return 1073741824 that is
* (1024*1024*1024).
*
* On parsing error, if *err is not NULL, it's set to 1, otherwise it's
* set to 0. On error the function return value is 0, regardless of the
* fact 'err' is NULL or not. */
* set to 0 */
long long memtoll(const char *p, int *err) {
const char *u;
char buf[128];
@@ -185,7 +182,6 @@ long long memtoll(const char *p, int *err) {
unsigned int digits;
if (err) *err = 0;
/* Search the first non digit character. */
u = p;
if (*u == '-') u++;
@@ -206,26 +202,16 @@ long long memtoll(const char *p, int *err) {
mul = 1024L*1024*1024;
} else {
if (err) *err = 1;
return 0;
mul = 1;
}
/* Copy the digits into a buffer, we'll use strtoll() to convert
* the digit (without the unit) into a number. */
digits = u-p;
if (digits >= sizeof(buf)) {
if (err) *err = 1;
return 0;
return LLONG_MAX;
}
memcpy(buf,p,digits);
buf[digits] = '\0';
char *endptr;
errno = 0;
val = strtoll(buf,&endptr,10);
if ((val == 0 && errno == EINVAL) || *endptr != '\0') {
if (err) *err = 1;
return 0;
}
val = strtoll(buf,NULL,10);
return val*mul;
}
@@ -442,44 +428,11 @@ int d2string(char *buf, size_t len, double value) {
* having run_id == A, and you reconnect and it has run_id == B, you can be
* sure that it is either a different instance or it was restarted. */
void getRandomHexChars(char *p, unsigned int len) {
FILE *fp = fopen("/dev/urandom","r");
char *charset = "0123456789abcdef";
unsigned int j;
/* Global state. */
static int seed_initialized = 0;
static unsigned char seed[20]; /* The SHA1 seed, from /dev/urandom. */
static uint64_t counter = 0; /* The counter we hash with the seed. */
if (!seed_initialized) {
/* Initialize a seed and use SHA1 in counter mode, where we hash
* the same seed with a progressive counter. For the goals of this
* function we just need non-colliding strings, there are no
* cryptographic security needs. */
FILE *fp = fopen("/dev/urandom","r");
if (fp && fread(seed,sizeof(seed),1,fp) == 1)
seed_initialized = 1;
if (fp) fclose(fp);
}
if (seed_initialized) {
while(len) {
unsigned char digest[20];
SHA1_CTX ctx;
unsigned int copylen = len > 20 ? 20 : len;
SHA1Init(&ctx);
SHA1Update(&ctx, seed, sizeof(seed));
SHA1Update(&ctx, (unsigned char*)&counter,sizeof(counter));
SHA1Final(digest, &ctx);
counter++;
memcpy(p,digest,copylen);
/* Convert to hex digits. */
for (j = 0; j < copylen; j++) p[j] = charset[p[j] & 0x0F];
len -= copylen;
p += copylen;
}
} else {
if (fp == NULL || fread(p,len,1,fp) == 0) {
/* If we can't read from /dev/urandom, do some reasonable effort
* in order to create some entropy, since this function is used to
* generate run_id and cluster instance IDs */
@@ -506,12 +459,14 @@ void getRandomHexChars(char *p, unsigned int len) {
x += sizeof(pid);
}
/* Finally xor it with rand() output, that was already seeded with
* time() at startup, and convert to hex digits. */
for (j = 0; j < len; j++) {
* time() at startup. */
for (j = 0; j < len; j++)
p[j] ^= rand();
p[j] = charset[p[j] & 0x0F];
}
}
/* Turn it into hex digits taking just 4 bits out of 8 for every byte. */
for (j = 0; j < len; j++)
p[j] = charset[p[j] & 0x0F];
if (fp) fclose(fp);
}
/* Given the filename, return the absolute path as an SDS string, or NULL
src/version.h
+1 -1
View File
@@ -1 +1 @@
#define REDIS_VERSION "3.0.1"
#define REDIS_VERSION "2.8.22"
tests/cluster/cluster.tcl
-130
View File
@@ -1,130 +0,0 @@
# Cluster-specific test functions.
#
# Copyright (C) 2014 Salvatore Sanfilippo antirez@gmail.com
# This software is released under the BSD License. See the COPYING file for
# more information.
# Returns a parsed CLUSTER NODES output as a list of dictionaries.
proc get_cluster_nodes id {
set lines [split [R $id cluster nodes] "\r\n"]
set nodes {}
foreach l $lines {
set l [string trim $l]
if {$l eq {}} continue
set args [split $l]
set node [dict create \
id [lindex $args 0] \
addr [lindex $args 1] \
flags [split [lindex $args 2] ,] \
slaveof [lindex $args 3] \
ping_sent [lindex $args 4] \
pong_recv [lindex $args 5] \
config_epoch [lindex $args 6] \
linkstate [lindex $args 7] \
slots [lrange $args 8 -1] \
]
lappend nodes $node
}
return $nodes
}
# Test node for flag.
proc has_flag {node flag} {
expr {[lsearch -exact [dict get $node flags] $flag] != -1}
}
# Returns the parsed myself node entry as a dictionary.
proc get_myself id {
set nodes [get_cluster_nodes $id]
foreach n $nodes {
if {[has_flag $n myself]} {return $n}
}
return {}
}
# Return the value of the specified CLUSTER INFO field.
proc CI {n field} {
get_info_field [R $n cluster info] $field
}
# Assuming nodes are reest, this function performs slots allocation.
# Only the first 'n' nodes are used.
proc cluster_allocate_slots {n} {
set slot 16383
while {$slot >= 0} {
# Allocate successive slots to random nodes.
set node [randomInt $n]
lappend slots_$node $slot
incr slot -1
}
for {set j 0} {$j < $n} {incr j} {
R $j cluster addslots {*}[set slots_${j}]
}
}
# Check that cluster nodes agree about "state", or raise an error.
proc assert_cluster_state {state} {
foreach_redis_id id {
if {[instance_is_killed redis $id]} continue
wait_for_condition 1000 50 {
[CI $id cluster_state] eq $state
} else {
fail "Cluster node $id cluster_state:[CI $id cluster_state]"
}
}
}
# Search the first node starting from ID $first that is not
# already configured as a slave.
proc cluster_find_available_slave {first} {
foreach_redis_id id {
if {$id < $first} continue
if {[instance_is_killed redis $id]} continue
set me [get_myself $id]
if {[dict get $me slaveof] eq {-}} {return $id}
}
fail "No available slaves"
}
# Add 'slaves' slaves to a cluster composed of 'masters' masters.
# It assumes that masters are allocated sequentially from instance ID 0
# to N-1.
proc cluster_allocate_slaves {masters slaves} {
for {set j 0} {$j < $slaves} {incr j} {
set master_id [expr {$j % $masters}]
set slave_id [cluster_find_available_slave $masters]
set master_myself [get_myself $master_id]
R $slave_id cluster replicate [dict get $master_myself id]
}
}
# Create a cluster composed of the specified number of masters and slaves.
proc create_cluster {masters slaves} {
cluster_allocate_slots $masters
if {$slaves} {
cluster_allocate_slaves $masters $slaves
}
assert_cluster_state ok
}
# Set the cluster node-timeout to all the reachalbe nodes.
proc set_cluster_node_timeout {to} {
foreach_redis_id id {
catch {R $id CONFIG SET cluster-node-timeout $to}
}
}
# Check if the cluster is writable and readable. Use node "id"
# as a starting point to talk with the cluster.
proc cluster_write_test {id} {
set prefix [randstring 20 20 alpha]
set port [get_instance_attrib redis $id port]
set cluster [redis_cluster 127.0.0.1:$port]
for {set j 0} {$j < 100} {incr j} {
$cluster set key.$j $prefix.$j
}
for {set j 0} {$j < 100} {incr j} {
assert {[$cluster get key.$j] eq "$prefix.$j"}
}
$cluster close
}
tests/cluster/run.tcl
-28
View File
@@ -1,28 +0,0 @@
# Cluster test suite. Copyright (C) 2014 Salvatore Sanfilippo antirez@gmail.com
# This software is released under the BSD License. See the COPYING file for
# more information.
cd tests/cluster
source cluster.tcl
source ../instances.tcl
source ../../support/cluster.tcl ; # Redis Cluster client.
set ::instances_count 20 ; # How many instances we use at max.
proc main {} {
parse_options
spawn_instance redis $::redis_base_port $::instances_count {
"cluster-enabled yes"
"appendonly yes"
}
run_tests
cleanup
end_tests
}
if {[catch main e]} {
puts $::errorInfo
if {$::pause_on_error} pause_on_error
cleanup
exit 1
}
tests/cluster/tests/00-base.tcl
-59
View File
@@ -1,59 +0,0 @@
# Check the basic monitoring and failover capabilities.
source "../tests/includes/init-tests.tcl"
if {$::simulate_error} {
test "This test will fail" {
fail "Simulated error"
}
}
test "Different nodes have different IDs" {
set ids {}
set numnodes 0
foreach_redis_id id {
incr numnodes
# Every node should just know itself.
set nodeid [dict get [get_myself $id] id]
assert {$nodeid ne {}}
lappend ids $nodeid
}
set numids [llength [lsort -unique $ids]]
assert {$numids == $numnodes}
}
test "It is possible to perform slot allocation" {
cluster_allocate_slots 5
}
test "After the join, every node gets a different config epoch" {
set trynum 60
while {[incr trynum -1] != 0} {
# We check that this condition is true for *all* the nodes.
set ok 1 ; # Will be set to 0 every time a node is not ok.
foreach_redis_id id {
set epochs {}
foreach n [get_cluster_nodes $id] {
lappend epochs [dict get $n config_epoch]
}
if {[lsort $epochs] != [lsort -unique $epochs]} {
set ok 0 ; # At least one collision!
}
}
if {$ok} break
after 1000
puts -nonewline .
flush stdout
}
if {$trynum == 0} {
fail "Config epoch conflict resolution is not working."
}
}
test "Nodes should report cluster_state is ok now" {
assert_cluster_state ok
}
test "It is possible to write and read from the cluster" {
cluster_write_test 0
}
tests/cluster/tests/01-faildet.tcl
-38
View File
@@ -1,38 +0,0 @@
# Check the basic monitoring and failover capabilities.
source "../tests/includes/init-tests.tcl"
test "Create a 5 nodes cluster" {
create_cluster 5 5
}
test "Cluster should start ok" {
assert_cluster_state ok
}
test "Killing two slave nodes" {
kill_instance redis 5
kill_instance redis 6
}
test "Cluster should be still up" {
assert_cluster_state ok
}
test "Killing one master node" {
kill_instance redis 0
}
# Note: the only slave of instance 0 is already down so no
# failover is possible, that would change the state back to ok.
test "Cluster should be down now" {
assert_cluster_state fail
}
test "Restarting master node" {
restart_instance redis 0
}
test "Cluster should be up again" {
assert_cluster_state ok
}
tests/cluster/tests/02-failover.tcl
-65
View File
@@ -1,65 +0,0 @@
# Check the basic monitoring and failover capabilities.
source "../tests/includes/init-tests.tcl"
test "Create a 5 nodes cluster" {
create_cluster 5 5
}
test "Cluster is up" {
assert_cluster_state ok
}
test "Cluster is writable" {
cluster_write_test 0
}
test "Instance #5 is a slave" {
assert {[RI 5 role] eq {slave}}
}
test "Instance #5 synced with the master" {
wait_for_condition 1000 50 {
[RI 5 master_link_status] eq {up}
} else {
fail "Instance #5 master link status is not up"
}
}
set current_epoch [CI 1 cluster_current_epoch]
test "Killing one master node" {
kill_instance redis 0
}
test "Wait for failover" {
wait_for_condition 1000 50 {
[CI 1 cluster_current_epoch] > $current_epoch
} else {
fail "No failover detected"
}
}
test "Cluster should eventually be up again" {
assert_cluster_state ok
}
test "Cluster is writable" {
cluster_write_test 1
}
test "Instance #5 is now a master" {
assert {[RI 5 role] eq {master}}
}
test "Restarting the previously killed master node" {
restart_instance redis 0
}
test "Instance #0 gets converted into a slave" {
wait_for_condition 1000 50 {
[RI 0 role] eq {slave}
} else {
fail "Old master was not converted into slave"
}
}
tests/cluster/tests/03-failover-loop.tcl
-115
View File
@@ -1,115 +0,0 @@
# Failover stress test.
# In this test a different node is killed in a loop for N
# iterations. The test checks that certain properties
# are preseved across iterations.
source "../tests/includes/init-tests.tcl"
test "Create a 5 nodes cluster" {
create_cluster 5 5
}
test "Cluster is up" {
assert_cluster_state ok
}
set iterations 20
set cluster [redis_cluster 127.0.0.1:[get_instance_attrib redis 0 port]]
while {[incr iterations -1]} {
set tokill [randomInt 10]
set other [expr {($tokill+1)%10}] ; # Some other instance.
set key [randstring 20 20 alpha]
set val [randstring 20 20 alpha]
set role [RI $tokill role]
if {$role eq {master}} {
set slave {}
set myid [dict get [get_myself $tokill] id]
foreach_redis_id id {
if {$id == $tokill} continue
if {[dict get [get_myself $id] slaveof] eq $myid} {
set slave $id
}
}
if {$slave eq {}} {
fail "Unable to retrieve slave's ID for master #$tokill"
}
}
puts "--- Iteration $iterations ---"
if {$role eq {master}} {
test "Wait for slave of #$tokill to sync" {
wait_for_condition 1000 50 {
[string match {*state=online*} [RI $tokill slave0]]
} else {
fail "Slave of node #$tokill is not ok"
}
}
set slave_config_epoch [CI $slave cluster_my_epoch]
}
test "Cluster is writable before failover" {
for {set i 0} {$i < 100} {incr i} {
catch {$cluster set $key:$i $val:$i} err
assert {$err eq {OK}}
}
# Wait for the write to propagate to the slave if we
# are going to kill a master.
if {$role eq {master}} {
R $tokill wait 1 20000
}
}
test "Killing node #$tokill" {
kill_instance redis $tokill
}
if {$role eq {master}} {
test "Wait failover by #$slave with old epoch $slave_config_epoch" {
wait_for_condition 1000 50 {
[CI $slave cluster_my_epoch] > $slave_config_epoch
} else {
fail "No failover detected, epoch is still [CI $slave cluster_my_epoch]"
}
}
}
test "Cluster should eventually be up again" {
assert_cluster_state ok
}
test "Cluster is writable again" {
for {set i 0} {$i < 100} {incr i} {
catch {$cluster set $key:$i:2 $val:$i:2} err
assert {$err eq {OK}}
}
}
test "Restarting node #$tokill" {
restart_instance redis $tokill
}
test "Instance #$tokill is now a slave" {
wait_for_condition 1000 50 {
[RI $tokill role] eq {slave}
} else {
fail "Restarted instance is not a slave"
}
}
test "We can read back the value we set before" {
for {set i 0} {$i < 100} {incr i} {
catch {$cluster get $key:$i} err
assert {$err eq "$val:$i"}
catch {$cluster get $key:$i:2} err
assert {$err eq "$val:$i:2"}
}
}
}
test "Post condition: current_epoch >= my_epoch everywhere" {
foreach_redis_id id {
assert {[CI $id cluster_current_epoch] >= [CI $id cluster_my_epoch]}
}
}
tests/cluster/tests/04-resharding.tcl
-102
View File
@@ -1,102 +0,0 @@
# Failover stress test.
# In this test a different node is killed in a loop for N
# iterations. The test checks that certain properties
# are preseved across iterations.
source "../tests/includes/init-tests.tcl"
test "Create a 5 nodes cluster" {
create_cluster 5 5
}
test "Cluster is up" {
assert_cluster_state ok
}
# Return nno-zero if the specified PID is about a process still in execution,
# otherwise 0 is returned.
proc process_is_running {pid} {
# PS should return with an error if PID is non existing,
# and catch will return non-zero. We want to return non-zero if
# the PID exists, so we invert the return value with expr not operator.
expr {![catch {exec ps -p $pid}]}
}
# Our resharding test performs the following actions:
#
# - N commands are sent to the cluster in the course of the test.
# - Every command selects a random key from key:0 to key:MAX-1.
# - The operation RPUSH key <randomvalue> is perforemd.
# - Tcl remembers into an array all the values pushed to each list.
# - After N/2 commands, the resharding process is started in background.
# - The test continues while the resharding is in progress.
# - At the end of the test, we wait for the resharding process to stop.
# - Finally the keys are checked to see if they contain the value they should.
set numkeys 50000
set numops 200000
set cluster [redis_cluster 127.0.0.1:[get_instance_attrib redis 0 port]]
catch {unset content}
array set content {}
set tribpid {}
test "Cluster consistency during live resharding" {
for {set j 0} {$j < $numops} {incr j} {
# Trigger the resharding once we execute half the ops.
if {$tribpid ne {} &&
($j % 10000) == 0 &&
![process_is_running $tribpid]} {
set tribpid {}
}
if {$j >= $numops/2 && $tribpid eq {}} {
puts -nonewline "...Starting resharding..."
flush stdout
set target [dict get [get_myself [randomInt 5]] id]
set tribpid [lindex [exec \
../../../src/redis-trib.rb reshard \
--from all \
--to $target \
--slots 100 \
--yes \
127.0.0.1:[get_instance_attrib redis 0 port] \
| [info nameofexecutable] \
../tests/helpers/onlydots.tcl \
&] 0]
}
# Write random data to random list.
set listid [randomInt $numkeys]
set key "key:$listid"
set ele [randomValue]
# We write both with Lua scripts and with plain commands.
# This way we are able to stress Lua -> Redis command invocation
# as well, that has tests to prevent Lua to write into wrong
# hash slots.
if {$listid % 2} {
$cluster rpush $key $ele
} else {
$cluster eval {redis.call("rpush",KEYS[1],ARGV[1])} 1 $key $ele
}
lappend content($key) $ele
if {($j % 1000) == 0} {
puts -nonewline W; flush stdout
}
}
# Wait for the resharding process to end
wait_for_condition 1000 500 {
[process_is_running $tribpid] == 0
} else {
fail "Resharding is not terminating after some time."
}
}
test "Verify $numkeys keys for consistency with logical content" {
# Check that the Redis Cluster content matches our logical content.
foreach {key value} [array get content] {
assert {[$cluster lrange $key 0 -1] eq $value}
}
}
tests/cluster/tests/05-slave-selection.tcl
-94
View File
@@ -1,94 +0,0 @@
# Slave selection test
# Check the algorithm trying to pick the slave with the most complete history.
source "../tests/includes/init-tests.tcl"
# Create a cluster with 5 master and 10 slaves, so that we have 2
# slaves for each master.
test "Create a 5 nodes cluster" {
create_cluster 5 10
}
test "Cluster is up" {
assert_cluster_state ok
}
test "The first master has actually two slaves" {
assert {[llength [lindex [R 0 role] 2]] == 2}
}
test {Slaves of #0 are instance #5 and #10 as expected} {
set port0 [get_instance_attrib redis 0 port]
assert {[lindex [R 5 role] 2] == $port0}
assert {[lindex [R 10 role] 2] == $port0}
}
test "Instance #5 and #10 synced with the master" {
wait_for_condition 1000 50 {
[RI 5 master_link_status] eq {up} &&
[RI 10 master_link_status] eq {up}
} else {
fail "Instance #5 or #10 master link status is not up"
}
}
set cluster [redis_cluster 127.0.0.1:[get_instance_attrib redis 0 port]]
test "Slaves are both able to receive and acknowledge writes" {
for {set j 0} {$j < 100} {incr j} {
$cluster set $j $j
}
assert {[R 0 wait 2 60000] == 2}
}
test "Write data while slave #10 is paused and can't receive it" {
# Stop the slave with a multi/exec transaction so that the master will
# be killed as soon as it can accept writes again.
R 10 multi
R 10 debug sleep 10
R 10 client kill 127.0.0.1:$port0
R 10 deferred 1
R 10 exec
# Write some data the slave can't receive.
for {set j 0} {$j < 100} {incr j} {
$cluster set $j $j
}
# Prevent the master from accepting new slaves.
# Use a large pause value since we'll kill it anyway.
R 0 CLIENT PAUSE 60000
# Wait for the slave to return available again
R 10 deferred 0
assert {[R 10 read] eq {OK OK}}
# Kill the master so that a reconnection will not be possible.
kill_instance redis 0
}
test "Wait for instance #5 (and not #10) to turn into a master" {
wait_for_condition 1000 50 {
[RI 5 role] eq {master}
} else {
fail "No failover detected"
}
}
test "Wait for the node #10 to return alive before ending the test" {
R 10 ping
}
test "Cluster should eventually be up again" {
assert_cluster_state ok
}
test "Node #10 should eventually replicate node #5" {
set port5 [get_instance_attrib redis 5 port]
wait_for_condition 1000 50 {
([lindex [R 10 role] 2] == $port5) &&
([lindex [R 10 role] 3] eq {connected})
} else {
fail "#10 didn't became slave of #5"
}
}
tests/cluster/tests/06-slave-stop-cond.tcl
-73
View File
@@ -1,73 +0,0 @@
# Slave stop condition test
# Check that if there is a disconnection time limit, the slave will not try
# to failover its master.
source "../tests/includes/init-tests.tcl"
# Create a cluster with 5 master and 5 slaves.
test "Create a 5 nodes cluster" {
create_cluster 5 5
}
test "Cluster is up" {
assert_cluster_state ok
}
test "The first master has actually one slave" {
assert {[llength [lindex [R 0 role] 2]] == 1}
}
test {Slaves of #0 is instance #5 as expected} {
set port0 [get_instance_attrib redis 0 port]
assert {[lindex [R 5 role] 2] == $port0}
}
test "Instance #5 synced with the master" {
wait_for_condition 1000 50 {
[RI 5 master_link_status] eq {up}
} else {
fail "Instance #5 master link status is not up"
}
}
test "Lower the slave validity factor of #5 to the value of 2" {
assert {[R 5 config set cluster-slave-validity-factor 2] eq {OK}}
}
test "Break master-slave link and prevent further reconnections" {
# Stop the slave with a multi/exec transaction so that the master will
# be killed as soon as it can accept writes again.
R 5 multi
R 5 client kill 127.0.0.1:$port0
# here we should sleep 6 or more seconds (node_timeout * slave_validity)
# but the actual validity time is actually incremented by the
# repl-ping-slave-period value which is 10 seconds by default. So we
# need to wait more than 16 seconds.
R 5 debug sleep 20
R 5 deferred 1
R 5 exec
# Prevent the master from accepting new slaves.
# Use a large pause value since we'll kill it anyway.
R 0 CLIENT PAUSE 60000
# Wait for the slave to return available again
R 5 deferred 0
assert {[R 5 read] eq {OK OK}}
# Kill the master so that a reconnection will not be possible.
kill_instance redis 0
}
test "Slave #5 is reachable and alive" {
assert {[R 5 ping] eq {PONG}}
}
test "Slave #5 should not be able to failover" {
after 10000
assert {[RI 5 role] eq {slave}}
}
test "Cluster should be down" {
assert_cluster_state fail
}
tests/cluster/tests/07-replica-migration.tcl
-47
View File
@@ -1,47 +0,0 @@
# Replica migration test.
# Check that orphaned masters are joined by replicas of masters having
# multiple replicas attached, according to the migration barrier settings.
source "../tests/includes/init-tests.tcl"
# Create a cluster with 5 master and 10 slaves, so that we have 2
# slaves for each master.
test "Create a 5 nodes cluster" {
create_cluster 5 10
}
test "Cluster is up" {
assert_cluster_state ok
}
test "Each master should have two replicas attached" {
foreach_redis_id id {
if {$id < 5} {
wait_for_condition 1000 50 {
[llength [lindex [R 0 role] 2]] == 2
} else {
fail "Master #$id does not have 2 slaves as expected"
}
}
}
}
test "Killing all the slaves of master #0 and #1" {
kill_instance redis 5
kill_instance redis 10
kill_instance redis 6
kill_instance redis 11
after 4000
}
foreach_redis_id id {
if {$id < 5} {
test "Master #$id should have at least one replica" {
wait_for_condition 1000 50 {
[llength [lindex [R $id role] 2]] >= 1
} else {
fail "Master #$id has no replicas"
}
}
}
}
tests/cluster/tests/08-update-msg.tcl
-90
View File
@@ -1,90 +0,0 @@
# Test UPDATE messages sent by other nodes when the currently authorirative
# master is unavaialble. The test is performed in the following steps:
#
# 1) Master goes down.
# 2) Slave failover and becomes new master.
# 3) New master is partitoned away.
# 4) Old master returns.
# 5) At this point we expect the old master to turn into a slave ASAP because
# of the UPDATE messages it will receive from the other nodes when its
# configuration will be found to be outdated.
source "../tests/includes/init-tests.tcl"
test "Create a 5 nodes cluster" {
create_cluster 5 5
}
test "Cluster is up" {
assert_cluster_state ok
}
test "Cluster is writable" {
cluster_write_test 0
}
test "Instance #5 is a slave" {
assert {[RI 5 role] eq {slave}}
}
test "Instance #5 synced with the master" {
wait_for_condition 1000 50 {
[RI 5 master_link_status] eq {up}
} else {
fail "Instance #5 master link status is not up"
}
}
set current_epoch [CI 1 cluster_current_epoch]
test "Killing one master node" {
kill_instance redis 0
}
test "Wait for failover" {
wait_for_condition 1000 50 {
[CI 1 cluster_current_epoch] > $current_epoch
} else {
fail "No failover detected"
}
}
test "Cluster should eventually be up again" {
assert_cluster_state ok
}
test "Cluster is writable" {
cluster_write_test 1
}
test "Instance #5 is now a master" {
assert {[RI 5 role] eq {master}}
}
test "Killing the new master #5" {
kill_instance redis 5
}
test "Cluster should be down now" {
assert_cluster_state fail
}
test "Restarting the old master node" {
restart_instance redis 0
}
test "Instance #0 gets converted into a slave" {
wait_for_condition 1000 50 {
[RI 0 role] eq {slave}
} else {
fail "Old master was not converted into slave"
}
}
test "Restarting the new master node" {
restart_instance redis 5
}
test "Cluster is up again" {
assert_cluster_state ok
}
tests/cluster/tests/09-pubsub.tcl
-40
View File
@@ -1,40 +0,0 @@
# Test PUBLISH propagation across the cluster.
source "../tests/includes/init-tests.tcl"
test "Create a 5 nodes cluster" {
create_cluster 5 5
}
proc test_cluster_publish {instance instances} {
# Subscribe all the instances but the one we use to send.
for {set j 0} {$j < $instances} {incr j} {
if {$j != $instance} {
R $j deferred 1
R $j subscribe testchannel
R $j read; # Read the subscribe reply
}
}
set data [randomValue]
R $instance PUBLISH testchannel $data
# Read the message back from all the nodes.
for {set j 0} {$j < $instances} {incr j} {
if {$j != $instance} {
set msg [R $j read]
assert {$data eq [lindex $msg 2]}
R $j unsubscribe testchannel
R $j read; # Read the unsubscribe reply
R $j deferred 0
}
}
}
test "Test publishing to master" {
test_cluster_publish 0 10
}
test "Test publishing to slave" {
test_cluster_publish 5 10
}
tests/cluster/tests/10-manual-failover.tcl
-192
View File
@@ -1,192 +0,0 @@
# Check the manual failover
source "../tests/includes/init-tests.tcl"
test "Create a 5 nodes cluster" {
create_cluster 5 5
}
test "Cluster is up" {
assert_cluster_state ok
}
test "Cluster is writable" {
cluster_write_test 0
}
test "Instance #5 is a slave" {
assert {[RI 5 role] eq {slave}}
}
test "Instance #5 synced with the master" {
wait_for_condition 1000 50 {
[RI 5 master_link_status] eq {up}
} else {
fail "Instance #5 master link status is not up"
}
}
set current_epoch [CI 1 cluster_current_epoch]
set numkeys 50000
set numops 10000
set cluster [redis_cluster 127.0.0.1:[get_instance_attrib redis 0 port]]
catch {unset content}
array set content {}
test "Send CLUSTER FAILOVER to #5, during load" {
for {set j 0} {$j < $numops} {incr j} {
# Write random data to random list.
set listid [randomInt $numkeys]
set key "key:$listid"
set ele [randomValue]
# We write both with Lua scripts and with plain commands.
# This way we are able to stress Lua -> Redis command invocation
# as well, that has tests to prevent Lua to write into wrong
# hash slots.
if {$listid % 2} {
$cluster rpush $key $ele
} else {
$cluster eval {redis.call("rpush",KEYS[1],ARGV[1])} 1 $key $ele
}
lappend content($key) $ele
if {($j % 1000) == 0} {
puts -nonewline W; flush stdout
}
if {$j == $numops/2} {R 5 cluster failover}
}
}
test "Wait for failover" {
wait_for_condition 1000 50 {
[CI 1 cluster_current_epoch] > $current_epoch
} else {
fail "No failover detected"
}
}
test "Cluster should eventually be up again" {
assert_cluster_state ok
}
test "Cluster is writable" {
cluster_write_test 1
}
test "Instance #5 is now a master" {
assert {[RI 5 role] eq {master}}
}
test "Verify $numkeys keys for consistency with logical content" {
# Check that the Redis Cluster content matches our logical content.
foreach {key value} [array get content] {
assert {[$cluster lrange $key 0 -1] eq $value}
}
}
test "Instance #0 gets converted into a slave" {
wait_for_condition 1000 50 {
[RI 0 role] eq {slave}
} else {
fail "Old master was not converted into slave"
}
}
## Check that manual failover does not happen if we can't talk with the master.
source "../tests/includes/init-tests.tcl"
test "Create a 5 nodes cluster" {
create_cluster 5 5
}
test "Cluster is up" {
assert_cluster_state ok
}
test "Cluster is writable" {
cluster_write_test 0
}
test "Instance #5 is a slave" {
assert {[RI 5 role] eq {slave}}
}
test "Instance #5 synced with the master" {
wait_for_condition 1000 50 {
[RI 5 master_link_status] eq {up}
} else {
fail "Instance #5 master link status is not up"
}
}
test "Make instance #0 unreachable without killing it" {
R 0 deferred 1
R 0 DEBUG SLEEP 10
}
test "Send CLUSTER FAILOVER to instance #5" {
R 5 cluster failover
}
test "Instance #5 is still a slave after some time (no failover)" {
after 5000
assert {[RI 5 role] eq {master}}
}
test "Wait for instance #0 to return back alive" {
R 0 deferred 0
assert {[R 0 read] eq {OK}}
}
## Check with "force" failover happens anyway.
source "../tests/includes/init-tests.tcl"
test "Create a 5 nodes cluster" {
create_cluster 5 5
}
test "Cluster is up" {
assert_cluster_state ok
}
test "Cluster is writable" {
cluster_write_test 0
}
test "Instance #5 is a slave" {
assert {[RI 5 role] eq {slave}}
}
test "Instance #5 synced with the master" {
wait_for_condition 1000 50 {
[RI 5 master_link_status] eq {up}
} else {
fail "Instance #5 master link status is not up"
}
}
test "Make instance #0 unreachable without killing it" {
R 0 deferred 1
R 0 DEBUG SLEEP 10
}
test "Send CLUSTER FAILOVER to instance #5" {
R 5 cluster failover force
}
test "Instance #5 is a master after some time" {
wait_for_condition 1000 50 {
[RI 5 role] eq {master}
} else {
fail "Instance #5 is not a master after some time regardless of FORCE"
}
}
test "Wait for instance #0 to return back alive" {
R 0 deferred 0
assert {[R 0 read] eq {OK}}
}
tests/cluster/tests/11-manual-takeover.tcl
-59
View File
@@ -1,59 +0,0 @@
# Manual takeover test
source "../tests/includes/init-tests.tcl"
test "Create a 5 nodes cluster" {
create_cluster 5 5
}
test "Cluster is up" {
assert_cluster_state ok
}
test "Cluster is writable" {
cluster_write_test 0
}
test "Killing majority of master nodes" {
kill_instance redis 0
kill_instance redis 1
kill_instance redis 2
}
test "Cluster should eventually be down" {
assert_cluster_state fail
}
test "Use takeover to bring slaves back" {
R 5 cluster failover takeover
R 6 cluster failover takeover
R 7 cluster failover takeover
}
test "Cluster should eventually be up again" {
assert_cluster_state ok
}
test "Cluster is writable" {
cluster_write_test 4
}
test "Instance #5, #6, #7 are now masters" {
assert {[RI 5 role] eq {master}}
assert {[RI 6 role] eq {master}}
assert {[RI 7 role] eq {master}}
}
test "Restarting the previously killed master nodes" {
restart_instance redis 0
restart_instance redis 1
restart_instance redis 2
}
test "Instance #0, #1, #2 gets converted into a slaves" {
wait_for_condition 1000 50 {
[RI 0 role] eq {slave} && [RI 1 role] eq {slave} && [RI 2 role] eq {slave}
} else {
fail "Old masters not converted into slaves"
}
}
tests/cluster/tests/helpers/onlydots.tcl
-16
View File
@@ -1,16 +0,0 @@
# Read the standard input and only shows dots in the output, filtering out
# all the other characters. Designed to avoid bufferization so that when
# we get the output of redis-trib and want to show just the dots, we'll see
# the dots as soon as redis-trib will output them.
fconfigure stdin -buffering none
while 1 {
set c [read stdin 1]
if {$c eq {}} {
exit 0; # EOF
} elseif {$c eq {.}} {
puts -nonewline .
flush stdout
}
}
tests/cluster/tests/includes/init-tests.tcl
-70
View File
@@ -1,70 +0,0 @@
# Initialization tests -- most units will start including this.
test "(init) Restart killed instances" {
foreach type {redis} {
foreach_${type}_id id {
if {[get_instance_attrib $type $id pid] == -1} {
puts -nonewline "$type/$id "
flush stdout
restart_instance $type $id
}
}
}
}
test "Cluster nodes are reachable" {
foreach_redis_id id {
# Every node should be reachable.
wait_for_condition 1000 50 {
([catch {R $id ping} ping_reply] == 0) &&
($ping_reply eq {PONG})
} else {
catch {R $id ping} err
fail "Node #$id keeps replying '$err' to PING."
}
}
}
test "Cluster nodes hard reset" {
foreach_redis_id id {
if {$::valgrind} {
set node_timeout 10000
} else {
set node_timeout 3000
}
catch {R $id flushall} ; # May fail for readonly slaves.
R $id MULTI
R $id cluster reset hard
R $id cluster set-config-epoch [expr {$id+1}]
R $id EXEC
R $id config set cluster-node-timeout $node_timeout
R $id config set cluster-slave-validity-factor 10
R $id config rewrite
}
}
test "Cluster Join and auto-discovery test" {
# Join node 0 with 1, 1 with 2, ... and so forth.
# If auto-discovery works all nodes will know every other node
# eventually.
set ids {}
foreach_redis_id id {lappend ids $id}
for {set j 0} {$j < [expr [llength $ids]-1]} {incr j} {
set a [lindex $ids $j]
set b [lindex $ids [expr $j+1]]
set b_port [get_instance_attrib redis $b port]
R $a cluster meet 127.0.0.1 $b_port
}
foreach_redis_id id {
wait_for_condition 1000 50 {
[llength [get_cluster_nodes $id]] == [llength $ids]
} else {
fail "Cluster failed to join into a full mesh."
}
}
}
test "Before slots allocation, all nodes report cluster failure" {
assert_cluster_state fail
}
tests/cluster/tmp/.gitignore
-2
View File
@@ -1,2 +0,0 @@
redis_*
sentinel_*
tests/instances.tcl
+14 -55
View File
@@ -16,10 +16,8 @@ source ../support/server.tcl
source ../support/test.tcl
set ::verbose 0
set ::valgrind 0
set ::pause_on_error 0
set ::simulate_error 0
set ::failed 0
set ::sentinel_instances {}
set ::redis_instances {}
set ::sentinel_base_port 20000
@@ -34,25 +32,6 @@ if {[catch {cd tmp}]} {
exit 1
}
# Execute the specified instance of the server specified by 'type', using
# the provided configuration file. Returns the PID of the process.
proc exec_instance {type cfgfile} {
if {$type eq "redis"} {
set prgname redis-server
} elseif {$type eq "sentinel"} {
set prgname redis-sentinel
} else {
error "Unknown instance type."
}
if {$::valgrind} {
set pid [exec valgrind --track-origins=yes --suppressions=../../../src/valgrind.sup --show-reachable=no --show-possibly-lost=no --leak-check=full ../../../src/${prgname} $cfgfile &]
} else {
set pid [exec ../../../src/${prgname} $cfgfile &]
}
return $pid
}
# Spawn a redis or sentinel instance, depending on 'type'.
proc spawn_instance {type base_port count {conf {}}} {
for {set j 0} {$j < $count} {incr j} {
@@ -79,7 +58,14 @@ proc spawn_instance {type base_port count {conf {}}} {
close $cfg
# Finally exec it and remember the pid for later cleanup.
set pid [exec_instance $type $cfgfile]
if {$type eq "redis"} {
set prgname redis-server
} elseif {$type eq "sentinel"} {
set prgname redis-sentinel
} else {
error "Unknown instance type."
}
set pid [exec ../../../src/${prgname} $cfgfile &]
lappend ::pids $pid
# Check availability
@@ -112,7 +98,6 @@ proc cleanup {} {
proc abort_sentinel_test msg {
puts "WARNING: Aborting the test."
puts ">>>>>>>> $msg"
if {$::pause_on_error} pause_on_error
cleanup
exit 1
}
@@ -128,8 +113,6 @@ proc parse_options {} {
set ::pause_on_error 1
} elseif {$opt eq "--fail"} {
set ::simulate_error 1
} elseif {$opt eq {--valgrind}} {
set ::valgrind 1
} elseif {$opt eq "--help"} {
puts "Hello, I'm sentinel.tcl and I run Sentinel unit tests."
puts "\nOptions:"
@@ -232,7 +215,6 @@ proc test {descr code} {
flush stdout
if {[catch {set retval [uplevel 1 $code]} error]} {
incr ::failed
if {[string match "assertion:*" $error]} {
set msg [string range $error 10 end]
puts [colorstr red $msg]
@@ -248,7 +230,6 @@ proc test {descr code} {
}
}
# Execute all the units inside the 'tests' directory.
proc run_tests {} {
set tests [lsort [glob ../tests/*]]
foreach test $tests {
@@ -261,17 +242,6 @@ proc run_tests {} {
}
}
# Print a message and exists with 0 / 1 according to zero or more failures.
proc end_tests {} {
if {$::failed == 0} {
puts "GOOD! No errors."
exit 0
} else {
puts "WARNING $::failed tests faield."
exit 1
}
}
# The "S" command is used to interact with the N-th Sentinel.
# The general form is:
#
@@ -390,31 +360,15 @@ proc get_instance_id_by_port {type port} {
# The instance can be restarted with restart-instance.
proc kill_instance {type id} {
set pid [get_instance_attrib $type $id pid]
set port [get_instance_attrib $type $id port]
if {$pid == -1} {
error "You tried to kill $type $id twice."
}
exec kill -9 $pid
set_instance_attrib $type $id pid -1
set_instance_attrib $type $id link you_tried_to_talk_with_killed_instance
# Remove the PID from the list of pids to kill at exit.
set ::pids [lsearch -all -inline -not -exact $::pids $pid]
# Wait for the port it was using to be available again, so that's not
# an issue to start a new server ASAP with the same port.
set retry 10
while {[incr retry -1]} {
set port_is_free [catch {set s [socket 127.0.01 $port]}]
if {$port_is_free} break
catch {close $s}
after 1000
}
if {$retry == 0} {
error "Port $port does not return available after killing instance."
}
}
# Return true of the instance of the specified type/id is killed.
@@ -431,7 +385,12 @@ proc restart_instance {type id} {
# Execute the instance with its old setup and append the new pid
# file for cleanup.
set pid [exec_instance $type $cfgfile]
if {$type eq "redis"} {
set prgname redis-server
} else {
set prgname redis-sentinel
}
set pid [exec ../../../src/${prgname} $cfgfile &]
set_instance_attrib $type $id pid $pid
lappend ::pids $pid
tests/integration/logging.tcl
-24
View File
@@ -1,24 +0,0 @@
set server_path [tmpdir server.log]
set system_name [string tolower [exec uname -s]]
if {$system_name eq {linux} || $system_name eq {darwin}} {
start_server [list overrides [list dir $server_path]] {
test "Server is able to generate a stack trace on selected systems" {
r config set watchdog-period 200
r debug sleep 1
set pattern "*debugCommand*"
set retry 10
while {$retry} {
set result [exec tail -100 < [srv 0 stdout]]
if {[string match $pattern $result]} {
break
}
incr retry -1
after 1000
}
if {$retry == 0} {
error "assertion:expected stack trace not found into log file"
}
}
}
}
tests/integration/rdb.tcl
+13 -13
View File
@@ -7,19 +7,19 @@ start_server [list overrides [list "dir" $server_path "dbfilename" "encodings.rd
test "RDB encoding loading test" {
r select 0
csvdump r
} {"compressible","string","aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
"hash","hash","a","1","aa","10","aaa","100","b","2","bb","20","bbb","200","c","3","cc","30","ccc","300","ddd","400","eee","5000000000",
"hash_zipped","hash","a","1","b","2","c","3",
"list","list","1","2","3","a","b","c","100000","6000000000","1","2","3","a","b","c","100000","6000000000","1","2","3","a","b","c","100000","6000000000",
"list_zipped","list","1","2","3","a","b","c","100000","6000000000",
"number","string","10"
"set","set","1","100000","2","3","6000000000","a","b","c",
"set_zipped_1","set","1","2","3","4",
"set_zipped_2","set","100000","200000","300000","400000",
"set_zipped_3","set","1000000000","2000000000","3000000000","4000000000","5000000000","6000000000",
"string","string","Hello World"
"zset","zset","a","1","b","2","c","3","aa","10","bb","20","cc","30","aaa","100","bbb","200","ccc","300","aaaa","1000","cccc","123456789","bbbb","5000000000",
"zset_zipped","zset","a","1","b","2","c","3",
} {"0","compressible","string","aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
"0","hash","hash","a","1","aa","10","aaa","100","b","2","bb","20","bbb","200","c","3","cc","30","ccc","300","ddd","400","eee","5000000000",
"0","hash_zipped","hash","a","1","b","2","c","3",
"0","list","list","1","2","3","a","b","c","100000","6000000000","1","2","3","a","b","c","100000","6000000000","1","2","3","a","b","c","100000","6000000000",
"0","list_zipped","list","1","2","3","a","b","c","100000","6000000000",
"0","number","string","10"
"0","set","set","1","100000","2","3","6000000000","a","b","c",
"0","set_zipped_1","set","1","2","3","4",
"0","set_zipped_2","set","100000","200000","300000","400000",
"0","set_zipped_3","set","1000000000","2000000000","3000000000","4000000000","5000000000","6000000000",
"0","string","string","Hello World"
"0","zset","zset","a","1","b","2","c","3","aa","10","bb","20","cc","30","aaa","100","bbb","200","ccc","300","aaaa","1000","cccc","123456789","bbbb","5000000000",
"0","zset_zipped","zset","a","1","b","2","c","3",
}
}
tests/integration/replication-psync.tcl
+38 -25
View File
@@ -13,7 +13,11 @@ proc stop_bg_complex_data {handle} {
#
# You can specifiy backlog size, ttl, delay before reconnection, test duration
# in seconds, and an additional condition to verify at the end.
proc test_psync {descr duration backlog_size backlog_ttl delay cond} {
#
# If reconnect is > 0, the test actually try to break the connection and
# reconnect with the master, otherwise just the initial synchronization is
# checked for consistency.
proc test_psync {descr duration backlog_size backlog_ttl delay cond diskless reconnect} {
start_server {tags {"repl"}} {
start_server {} {
@@ -24,6 +28,8 @@ proc test_psync {descr duration backlog_size backlog_ttl delay cond} {
$master config set repl-backlog-size $backlog_size
$master config set repl-backlog-ttl $backlog_ttl
$master config set repl-diskless-sync $diskless
$master config set repl-diskless-sync-delay 1
set load_handle0 [start_bg_complex_data $master_host $master_port 9 100000]
set load_handle1 [start_bg_complex_data $master_host $master_port 11 100000]
@@ -48,22 +54,24 @@ proc test_psync {descr duration backlog_size backlog_ttl delay cond} {
}
}
test "Test replication partial resync: $descr" {
test "Test replication partial resync: $descr (diskless: $diskless, reconnect: $reconnect)" {
# Now while the clients are writing data, break the maste-slave
# link multiple times.
for {set j 0} {$j < $duration*10} {incr j} {
after 100
# catch {puts "MASTER [$master dbsize] keys, SLAVE [$slave dbsize] keys"}
if ($reconnect) {
for {set j 0} {$j < $duration*10} {incr j} {
after 100
# catch {puts "MASTER [$master dbsize] keys, SLAVE [$slave dbsize] keys"}
if {($j % 20) == 0} {
catch {
if {$delay} {
$slave multi
$slave client kill $master_host:$master_port
$slave debug sleep $delay
$slave exec
} else {
$slave client kill $master_host:$master_port
if {($j % 20) == 0} {
catch {
if {$delay} {
$slave multi
$slave client kill $master_host:$master_port
$slave debug sleep $delay
$slave exec
} else {
$slave client kill $master_host:$master_port
}
}
}
}
@@ -98,18 +106,23 @@ proc test_psync {descr duration backlog_size backlog_ttl delay cond} {
}
}
test_psync {ok psync} 6 1000000 3600 0 {
assert {[s -1 sync_partial_ok] > 0}
}
foreach diskless {no yes} {
test_psync {no reconnection, just sync} 6 1000000 3600 0 {
} $diskless 0
test_psync {no backlog} 6 100 3600 0.5 {
assert {[s -1 sync_partial_err] > 0}
}
test_psync {ok psync} 6 1000000 3600 0 {
assert {[s -1 sync_partial_ok] > 0}
} $diskless 1
test_psync {ok after delay} 3 100000000 3600 3 {
assert {[s -1 sync_partial_ok] > 0}
}
test_psync {no backlog} 6 100 3600 0.5 {
assert {[s -1 sync_partial_err] > 0}
} $diskless 1
test_psync {backlog expired} 3 100000000 1 3 {
assert {[s -1 sync_partial_err] > 0}
test_psync {ok after delay} 3 100000000 3600 3 {
assert {[s -1 sync_partial_ok] > 0}
} $diskless 1
test_psync {backlog expired} 3 100000000 1 3 {
assert {[s -1 sync_partial_err] > 0}
} $diskless 1
}
tests/sentinel/run.tcl
-1
View File
@@ -13,7 +13,6 @@ proc main {} {
spawn_instance redis $::redis_base_port $::instances_count
run_tests
cleanup
end_tests
}
if {[catch main e]} {
tests/sentinel/tests/06-ckquorum.tcl
+34
View File
@@ -0,0 +1,34 @@
# Test for the SENTINEL CKQUORUM command
source "../tests/includes/init-tests.tcl"
set num_sentinels [llength $::sentinel_instances]
test "CKQUORUM reports OK and the right amount of Sentinels" {
foreach_sentinel_id id {
assert_match "*OK $num_sentinels usable*" [S $id SENTINEL CKQUORUM mymaster]
}
}
test "CKQUORUM detects quorum cannot be reached" {
set orig_quorum [expr {$num_sentinels/2+1}]
S 0 SENTINEL SET mymaster quorum [expr {$num_sentinels+1}]
catch {[S 0 SENTINEL CKQUORUM mymaster]} err
assert_match "*NOQUORUM*" $err
S 0 SENTINEL SET mymaster quorum $orig_quorum
}
test "CKQUORUM detects failover authorization cannot be reached" {
set orig_quorum [expr {$num_sentinels/2+1}]
S 0 SENTINEL SET mymaster quorum 1
kill_instance sentinel 1
kill_instance sentinel 2
kill_instance sentinel 3
after 5000
catch {[S 0 SENTINEL CKQUORUM mymaster]} err
assert_match "*NOQUORUM*" $err
S 0 SENTINEL SET mymaster quorum $orig_quorum
restart_instance sentinel 1
restart_instance sentinel 2
restart_instance sentinel 3
}
tests/support/cluster.tcl
-305
View File
@@ -1,305 +0,0 @@
# Tcl redis cluster client as a wrapper of redis.rb.
# Copyright (C) 2014 Salvatore Sanfilippo
# Released under the BSD license like Redis itself
#
# Example usage:
#
# set c [redis_cluster 127.0.0.1 6379 127.0.0.1 6380]
# $c set foo
# $c get foo
# $c close
package require Tcl 8.5
package provide redis_cluster 0.1
namespace eval redis_cluster {}
set ::redis_cluster::id 0
array set ::redis_cluster::startup_nodes {}
array set ::redis_cluster::nodes {}
array set ::redis_cluster::slots {}
# List of "plain" commands, which are commands where the sole key is always
# the first argument.
set ::redis_cluster::plain_commands {
get set setnx setex psetex append strlen exists setbit getbit
setrange getrange substr incr decr rpush lpush rpushx lpushx
linsert rpop lpop brpop llen lindex lset lrange ltrim lrem
sadd srem sismember scard spop srandmember smembers sscan zadd
zincrby zrem zremrangebyscore zremrangebyrank zremrangebylex zrange
zrangebyscore zrevrangebyscore zrangebylex zrevrangebylex zcount
zlexcount zrevrange zcard zscore zrank zrevrank zscan hset hsetnx
hget hmset hmget hincrby hincrbyfloat hdel hlen hkeys hvals
hgetall hexists hscan incrby decrby incrbyfloat getset move
expire expireat pexpire pexpireat type ttl pttl persist restore
dump bitcount bitpos pfadd pfcount
}
proc redis_cluster {nodes} {
set id [incr ::redis_cluster::id]
set ::redis_cluster::startup_nodes($id) $nodes
set ::redis_cluster::nodes($id) {}
set ::redis_cluster::slots($id) {}
set handle [interp alias {} ::redis_cluster::instance$id {} ::redis_cluster::__dispatch__ $id]
$handle refresh_nodes_map
return $handle
}
# Totally reset the slots / nodes state for the client, calls
# CLUSTER NODES in the first startup node available, populates the
# list of nodes ::redis_cluster::nodes($id) with an hash mapping node
# ip:port to a representation of the node (another hash), and finally
# maps ::redis_cluster::slots($id) with an hash mapping slot numbers
# to node IDs.
#
# This function is called when a new Redis Cluster client is initialized
# and every time we get a -MOVED redirection error.
proc ::redis_cluster::__method__refresh_nodes_map {id} {
# Contact the first responding startup node.
set idx 0; # Index of the node that will respond.
set errmsg {}
foreach start_node $::redis_cluster::startup_nodes($id) {
lassign [split $start_node :] start_host start_port
if {[catch {
set r {}
set r [redis $start_host $start_port]
set nodes_descr [$r cluster nodes]
$r close
} e]} {
if {$r ne {}} {catch {$r close}}
incr idx
if {[string length $errmsg] < 200} {
append errmsg " $start_node: $e"
}
continue ; # Try next.
} else {
break; # Good node found.
}
}
if {$idx == [llength $::redis_cluster::startup_nodes($id)]} {
error "No good startup node found. $errmsg"
}
# Put the node that responded as first in the list if it is not
# already the first.
if {$idx != 0} {
set l $::redis_cluster::startup_nodes($id)
set left [lrange $l 0 [expr {$idx-1}]]
set right [lrange $l [expr {$idx+1}] end]
set l [concat [lindex $l $idx] $left $right]
set ::redis_cluster::startup_nodes($id) $l
}
# Parse CLUSTER NODES output to populate the nodes description.
set nodes {} ; # addr -> node description hash.
foreach line [split $nodes_descr "\n"] {
set line [string trim $line]
if {$line eq {}} continue
set args [split $line " "]
lassign $args nodeid addr flags slaveof pingsent pongrecv configepoch linkstate
set slots [lrange $args 8 end]
if {$addr eq {:0}} {
set addr $start_host:$start_port
}
lassign [split $addr :] host port
# Connect to the node
set link {}
catch {set link [redis $host $port]}
# Build this node description as an hash.
set node [dict create \
id $nodeid \
addr $addr \
host $host \
port $port \
flags $flags \
slaveof $slaveof \
slots $slots \
link $link \
]
dict set nodes $addr $node
lappend ::redis_cluster::startup_nodes($id) $addr
}
# Close all the existing links in the old nodes map, and set the new
# map as current.
foreach n $::redis_cluster::nodes($id) {
catch {
[dict get $n link] close
}
}
set ::redis_cluster::nodes($id) $nodes
# Populates the slots -> nodes map.
dict for {addr node} $nodes {
foreach slotrange [dict get $node slots] {
lassign [split $slotrange -] start end
if {$end == {}} {set end $start}
for {set j $start} {$j <= $end} {incr j} {
dict set ::redis_cluster::slots($id) $j $addr
}
}
}
# Only retain unique entries in the startup nodes list
set ::redis_cluster::startup_nodes($id) [lsort -unique $::redis_cluster::startup_nodes($id)]
}
# Free a redis_cluster handle.
proc ::redis_cluster::__method__close {id} {
catch {
set nodes $::redis_cluster::nodes($id)
dict for {addr node} $nodes {
catch {
[dict get $node link] close
}
}
}
catch {unset ::redis_cluster::startup_nodes($id)}
catch {unset ::redis_cluster::nodes($id)}
catch {unset ::redis_cluster::slots($id)}
catch {interp alias {} ::redis_cluster::instance$id {}}
}
proc ::redis_cluster::__dispatch__ {id method args} {
if {[info command ::redis_cluster::__method__$method] eq {}} {
# Get the keys from the command.
set keys [::redis_cluster::get_keys_from_command $method $args]
if {$keys eq {}} {
error "Redis command '$method' is not supported by redis_cluster."
}
# Resolve the keys in the corresponding hash slot they hash to.
set slot [::redis_cluster::get_slot_from_keys $keys]
if {$slot eq {}} {
error "Invalid command: multiple keys not hashing to the same slot."
}
# Get the node mapped to this slot.
set node_addr [dict get $::redis_cluster::slots($id) $slot]
if {$node_addr eq {}} {
error "No mapped node for slot $slot."
}
# Execute the command in the node we think is the slot owner.
set retry 100
while {[incr retry -1]} {
if {$retry < 5} {after 100}
set node [dict get $::redis_cluster::nodes($id) $node_addr]
set link [dict get $node link]
if {[catch {$link $method {*}$args} e]} {
if {$link eq {} || \
[string range $e 0 4] eq {MOVED} || \
[string range $e 0 2] eq {I/O} \
} {
# MOVED redirection.
::redis_cluster::__method__refresh_nodes_map $id
set node_addr [dict get $::redis_cluster::slots($id) $slot]
continue
} elseif {[string range $e 0 2] eq {ASK}} {
# ASK redirection.
set node_addr [lindex $e 2]
continue
} else {
# Non redirecting error.
error $e $::errorInfo $::errorCode
}
} else {
# OK query went fine
return $e
}
}
error "Too many redirections or failures contacting Redis Cluster."
} else {
uplevel 1 [list ::redis_cluster::__method__$method $id] $args
}
}
proc ::redis_cluster::get_keys_from_command {cmd argv} {
set cmd [string tolower $cmd]
# Most Redis commands get just one key as first argument.
if {[lsearch -exact $::redis_cluster::plain_commands $cmd] != -1} {
return [list [lindex $argv 0]]
}
# Special handling for other commands
switch -exact $cmd {
mget {return $argv}
eval {return [lrange $argv 2 1+[lindex $argv 1]]}
evalsha {return [lrange $argv 2 1+[lindex $argv 1]]}
}
# All the remaining commands are not handled.
return {}
}
# Returns the CRC16 of the specified string.
# The CRC parameters are described in the Redis Cluster specification.
set ::redis_cluster::XMODEMCRC16Lookup {
0x0000 0x1021 0x2042 0x3063 0x4084 0x50a5 0x60c6 0x70e7
0x8108 0x9129 0xa14a 0xb16b 0xc18c 0xd1ad 0xe1ce 0xf1ef
0x1231 0x0210 0x3273 0x2252 0x52b5 0x4294 0x72f7 0x62d6
0x9339 0x8318 0xb37b 0xa35a 0xd3bd 0xc39c 0xf3ff 0xe3de
0x2462 0x3443 0x0420 0x1401 0x64e6 0x74c7 0x44a4 0x5485
0xa56a 0xb54b 0x8528 0x9509 0xe5ee 0xf5cf 0xc5ac 0xd58d
0x3653 0x2672 0x1611 0x0630 0x76d7 0x66f6 0x5695 0x46b4
0xb75b 0xa77a 0x9719 0x8738 0xf7df 0xe7fe 0xd79d 0xc7bc
0x48c4 0x58e5 0x6886 0x78a7 0x0840 0x1861 0x2802 0x3823
0xc9cc 0xd9ed 0xe98e 0xf9af 0x8948 0x9969 0xa90a 0xb92b
0x5af5 0x4ad4 0x7ab7 0x6a96 0x1a71 0x0a50 0x3a33 0x2a12
0xdbfd 0xcbdc 0xfbbf 0xeb9e 0x9b79 0x8b58 0xbb3b 0xab1a
0x6ca6 0x7c87 0x4ce4 0x5cc5 0x2c22 0x3c03 0x0c60 0x1c41
0xedae 0xfd8f 0xcdec 0xddcd 0xad2a 0xbd0b 0x8d68 0x9d49
0x7e97 0x6eb6 0x5ed5 0x4ef4 0x3e13 0x2e32 0x1e51 0x0e70
0xff9f 0xefbe 0xdfdd 0xcffc 0xbf1b 0xaf3a 0x9f59 0x8f78
0x9188 0x81a9 0xb1ca 0xa1eb 0xd10c 0xc12d 0xf14e 0xe16f
0x1080 0x00a1 0x30c2 0x20e3 0x5004 0x4025 0x7046 0x6067
0x83b9 0x9398 0xa3fb 0xb3da 0xc33d 0xd31c 0xe37f 0xf35e
0x02b1 0x1290 0x22f3 0x32d2 0x4235 0x5214 0x6277 0x7256
0xb5ea 0xa5cb 0x95a8 0x8589 0xf56e 0xe54f 0xd52c 0xc50d
0x34e2 0x24c3 0x14a0 0x0481 0x7466 0x6447 0x5424 0x4405
0xa7db 0xb7fa 0x8799 0x97b8 0xe75f 0xf77e 0xc71d 0xd73c
0x26d3 0x36f2 0x0691 0x16b0 0x6657 0x7676 0x4615 0x5634
0xd94c 0xc96d 0xf90e 0xe92f 0x99c8 0x89e9 0xb98a 0xa9ab
0x5844 0x4865 0x7806 0x6827 0x18c0 0x08e1 0x3882 0x28a3
0xcb7d 0xdb5c 0xeb3f 0xfb1e 0x8bf9 0x9bd8 0xabbb 0xbb9a
0x4a75 0x5a54 0x6a37 0x7a16 0x0af1 0x1ad0 0x2ab3 0x3a92
0xfd2e 0xed0f 0xdd6c 0xcd4d 0xbdaa 0xad8b 0x9de8 0x8dc9
0x7c26 0x6c07 0x5c64 0x4c45 0x3ca2 0x2c83 0x1ce0 0x0cc1
0xef1f 0xff3e 0xcf5d 0xdf7c 0xaf9b 0xbfba 0x8fd9 0x9ff8
0x6e17 0x7e36 0x4e55 0x5e74 0x2e93 0x3eb2 0x0ed1 0x1ef0
}
proc ::redis_cluster::crc16 {s} {
set s [encoding convertto ascii $s]
set crc 0
foreach char [split $s {}] {
scan $char %c byte
set crc [expr {(($crc<<8)&0xffff) ^ [lindex $::redis_cluster::XMODEMCRC16Lookup [expr {(($crc>>8)^$byte) & 0xff}]]}]
}
return $crc
}
# Hash a single key returning the slot it belongs to, Implemented hash
# tags as described in the Redis Cluster specification.
proc ::redis_cluster::hash {key} {
# TODO: Handle hash slots.
expr {[::redis_cluster::crc16 $key] & 16383}
}
# Return the slot the specified keys hash to.
# If the keys hash to multiple slots, an empty string is returned to
# signal that the command can't be run in Redis Cluster.
proc ::redis_cluster::get_slot_from_keys {keys} {
set slot {}
foreach k $keys {
set s [::redis_cluster::hash $k]
if {$slot eq {}} {
set slot $s
} elseif {$slot != $s} {
return {} ; # Error
}
}
return $slot
}
tests/support/redis.tcl
+2 -2
View File
@@ -1,5 +1,5 @@
# Tcl client library - used by the Redis test
# Copyright (C) 2009-2014 Salvatore Sanfilippo
# Tcl clinet library - used by test-redis.tcl script for now
# Copyright (C) 2009 Salvatore Sanfilippo
# Released under the BSD license like Redis itself
#
# Example usage:
tests/support/server.tcl
+2 -7
View File
@@ -54,15 +54,10 @@ proc kill_server config {
# kill server and wait for the process to be totally exited
catch {exec kill $pid}
if {$::valgrind} {
set max_wait 60000
} else {
set max_wait 10000
}
while {[is_alive $config]} {
incr wait 10
if {$wait >= $max_wait} {
if {$wait >= 5000} {
puts "Forcing process $pid to exit..."
catch {exec kill -KILL $pid}
} elseif {$wait % 1000 == 0} {
@@ -212,7 +207,7 @@ proc start_server {options {code undefined}} {
set stderr [format "%s/%s" [dict get $config "dir"] "stderr"]
if {$::valgrind} {
set pid [exec valgrind --track-origins=yes --suppressions=src/valgrind.sup --show-reachable=no --show-possibly-lost=no --leak-check=full src/redis-server $config_file > $stdout 2> $stderr &]
set pid [exec valgrind --suppressions=src/valgrind.sup --show-reachable=no --show-possibly-lost=no --leak-check=full src/redis-server $config_file > $stdout 2> $stderr &]
} else {
set pid [exec src/redis-server $config_file > $stdout 2> $stderr &]
}
tests/support/util.tcl
+40 -38
View File
@@ -262,46 +262,50 @@ proc formatCommand {args} {
proc csvdump r {
set o {}
foreach k [lsort [{*}$r keys *]] {
set type [{*}$r type $k]
append o [csvstring $k] , [csvstring $type] ,
switch $type {
string {
append o [csvstring [{*}$r get $k]] "\n"
}
list {
foreach e [{*}$r lrange $k 0 -1] {
append o [csvstring $e] ,
for {set db 0} {$db < 16} {incr db} {
{*}$r select $db
foreach k [lsort [{*}$r keys *]] {
set type [{*}$r type $k]
append o [csvstring $db] , [csvstring $k] , [csvstring $type] ,
switch $type {
string {
append o [csvstring [{*}$r get $k]] "\n"
}
append o "\n"
}
set {
foreach e [lsort [{*}$r smembers $k]] {
append o [csvstring $e] ,
list {
foreach e [{*}$r lrange $k 0 -1] {
append o [csvstring $e] ,
}
append o "\n"
}
append o "\n"
}
zset {
foreach e [{*}$r zrange $k 0 -1 withscores] {
append o [csvstring $e] ,
set {
foreach e [lsort [{*}$r smembers $k]] {
append o [csvstring $e] ,
}
append o "\n"
}
append o "\n"
}
hash {
set fields [{*}$r hgetall $k]
set newfields {}
foreach {k v} $fields {
lappend newfields [list $k $v]
zset {
foreach e [{*}$r zrange $k 0 -1 withscores] {
append o [csvstring $e] ,
}
append o "\n"
}
set fields [lsort -index 0 $newfields]
foreach kv $fields {
append o [csvstring [lindex $kv 0]] ,
append o [csvstring [lindex $kv 1]] ,
hash {
set fields [{*}$r hgetall $k]
set newfields {}
foreach {k v} $fields {
lappend newfields [list $k $v]
}
set fields [lsort -index 0 $newfields]
foreach kv $fields {
append o [csvstring [lindex $kv 0]] ,
append o [csvstring [lindex $kv 1]] ,
}
append o "\n"
}
append o "\n"
}
}
}
{*}$r select 9
return $o
}
@@ -315,14 +319,12 @@ proc roundFloat f {
proc find_available_port start {
for {set j $start} {$j < $start+1024} {incr j} {
if {[catch {set fd1 [socket 127.0.0.1 $j]}] &&
[catch {set fd2 [socket 127.0.0.1 [expr $j+10000]]}]} {
if {[catch {
set fd [socket 127.0.0.1 $j]
}]} {
return $j
} else {
catch {
close $fd1
close $fd2
}
close $fd
}
}
if {$j == $start+1024} {
tests/test_helper.tcl
+1 -2
View File
@@ -13,7 +13,6 @@ source tests/support/util.tcl
set ::all_tests {
unit/printver
unit/dump
unit/auth
unit/protocol
unit/basic
@@ -38,7 +37,6 @@ set ::all_tests {
integration/aof
integration/rdb
integration/convert-zipmap-hash-on-load
integration/logging
unit/pubsub
unit/slowlog
unit/scripting
@@ -46,6 +44,7 @@ set ::all_tests {
unit/introspection
unit/limits
unit/obuf-limits
unit/dump
unit/bitops
unit/memefficiency
unit/hyperloglog
tests/unit/basic.tcl
-23
View File
@@ -150,29 +150,6 @@ start_server {tags {"basic"}} {
r decrby novar 17179869185
} {-1}
test {INCR uses shared objects in the 0-9999 range} {
r set foo -1
r incr foo
assert {[r object refcount foo] > 1}
r set foo 9998
r incr foo
assert {[r object refcount foo] > 1}
r incr foo
assert {[r object refcount foo] == 1}
}
test {INCR can modify objects in-place} {
r set foo 20000
r incr foo
assert {[r object refcount foo] == 1}
set old [lindex [split [r debug object foo]] 1]
r incr foo
set new [lindex [split [r debug object foo]] 1]
assert {[string range $old 0 2] eq "at:"}
assert {[string range $new 0 2] eq "at:"}
assert {$old eq $new}
}
test {INCRBYFLOAT against non existing key} {
r del novar
list [roundFloat [r incrbyfloat novar 1]] \
tests/unit/dump.tcl
+3 -81
View File
@@ -31,49 +31,12 @@ start_server {tags {"dump"}} {
set e {}
catch {r restore foo 0 "..."} e
set e
} {*BUSYKEY*}
test {RESTORE can overwrite an existing key with REPLACE} {
r set foo bar1
set encoded1 [r dump foo]
r set foo bar2
set encoded2 [r dump foo]
r del foo
r restore foo 0 $encoded1
r restore foo 0 $encoded2 replace
r get foo
} {bar2}
test {RESTORE can detect a syntax error for unrecongized options} {
catch {r restore foo 0 "..." invalid-option} e
set e
} {*syntax*}
} {*is busy*}
test {DUMP of non existing key returns nil} {
r dump nonexisting_key
} {}
test {MIGRATE is caching connections} {
# Note, we run this as first test so that the connection cache
# is empty.
set first [srv 0 client]
r set key "Some Value"
start_server {tags {"repl"}} {
set second [srv 0 client]
set second_host [srv 0 host]
set second_port [srv 0 port]
assert_match {*migrate_cached_sockets:0*} [r -1 info]
r -1 migrate $second_host $second_port key 9 1000
assert_match {*migrate_cached_sockets:1*} [r -1 info]
}
}
test {MIGRATE cached connections are released after some time} {
after 15000
assert_match {*migrate_cached_sockets:0*} [r info]
}
test {MIGRATE is able to migrate a key between two instances} {
set first [srv 0 client]
r set key "Some Value"
@@ -93,47 +56,6 @@ start_server {tags {"dump"}} {
}
}
test {MIGRATE is able to copy a key between two instances} {
set first [srv 0 client]
r del list
r lpush list a b c d
start_server {tags {"repl"}} {
set second [srv 0 client]
set second_host [srv 0 host]
set second_port [srv 0 port]
assert {[$first exists list] == 1}
assert {[$second exists list] == 0}
set ret [r -1 migrate $second_host $second_port list 9 5000 copy]
assert {$ret eq {OK}}
assert {[$first exists list] == 1}
assert {[$second exists list] == 1}
assert {[$first lrange list 0 -1] eq [$second lrange list 0 -1]}
}
}
test {MIGRATE will not overwrite existing keys, unless REPLACE is used} {
set first [srv 0 client]
r del list
r lpush list a b c d
start_server {tags {"repl"}} {
set second [srv 0 client]
set second_host [srv 0 host]
set second_port [srv 0 port]
assert {[$first exists list] == 1}
assert {[$second exists list] == 0}
$second set list somevalue
catch {r -1 migrate $second_host $second_port list 9 5000 copy} e
assert_match {ERR*} $e
set res [r -1 migrate $second_host $second_port list 9 5000 copy replace]
assert {$ret eq {OK}}
assert {[$first exists list] == 1}
assert {[$second exists list] == 1}
assert {[$first lrange list 0 -1] eq [$second lrange list 0 -1]}
}
}
test {MIGRATE propagates TTL correctly} {
set first [srv 0 client]
r set key "Some Value"
@@ -211,9 +133,9 @@ start_server {tags {"dump"}} {
assert {[$second exists key] == 0}
set rd [redis_deferring_client]
$rd debug sleep 1.0 ; # Make second server unable to reply.
$rd debug sleep 5.0 ; # Make second server unable to reply.
set e {}
catch {r -1 migrate $second_host $second_port key 9 500} e
catch {r -1 migrate $second_host $second_port key 9 1000} e
assert_match {IOERR*} $e
}
}
tests/unit/sort.tcl
-8
View File
@@ -95,14 +95,6 @@ start_server {
assert_encoding ziplist sort-res
}
test "SORT extracts STORE correctly" {
r command getkeys sort abc store def
} {abc def}
test "SORT extracts multiple STORE correctly" {
r command getkeys sort abc store invalid store stillbad store def
} {abc def}
test "SORT DESC" {
assert_equal [lsort -decreasing -integer $result] [r sort tosort DESC]
}
tests/unit/type/set.tcl
+1
View File
@@ -450,6 +450,7 @@ start_server {
test "SMOVE non existing key" {
setup_move
assert_equal 0 [r smove myset1 myset2 foo]
assert_equal 0 [r smove myset1 myset1 foo]
assert_equal {1 a b} [lsort [r smembers myset1]]
assert_equal {2 3 4} [lsort [r smembers myset2]]
}
utils/cluster_fail_time.tcl
-50
View File
@@ -1,50 +0,0 @@
# This simple script is used in order to estimate the average PFAIL->FAIL
# state switch after a failure.
set ::sleep_time 10 ; # How much to sleep to trigger PFAIL.
set ::fail_port 30016 ; # Node to put in sleep.
set ::other_port 30001 ; # Node to use to monitor the flag switch.
proc avg vector {
set sum 0.0
foreach x $vector {
set sum [expr {$sum+$x}]
}
expr {$sum/[llength $vector]}
}
set samples {}
while 1 {
exec redis-cli -p $::fail_port debug sleep $::sleep_time > /dev/null &
# Wait for fail? to appear.
while 1 {
set output [exec redis-cli -p $::other_port cluster nodes]
if {[string match {*fail\?*} $output]} break
after 100
}
puts "FAIL?"
set start [clock milliseconds]
# Wait for fail? to disappear.
while 1 {
set output [exec redis-cli -p $::other_port cluster nodes]
if {![string match {*fail\?*} $output]} break
after 100
}
puts "FAIL"
set now [clock milliseconds]
set elapsed [expr {$now-$start}]
puts $elapsed
lappend samples $elapsed
puts "AVG([llength $samples]): [avg $samples]"
# Wait for the instance to be available again.
exec redis-cli -p $::fail_port ping
# Wait for the fail flag to be cleared.
after 2000
}
utils/create-cluster/.gitignore
-5
View File
@@ -1,5 +0,0 @@
config.sh
*.rdb
*.aof
*.conf
*.log
utils/create-cluster/README
-27
View File
@@ -1,27 +0,0 @@
Create-custer is a small script used to easily start a big number of Redis
instances configured to run in cluster mode. Its main goal is to allow manual
testing in a condition which is not easy to replicate with the Redis cluster
unit tests, for example when a lot of instances are needed in order to trigger
a give bug.
The tool can also be used just to easily create a number of instances in a
Redis Cluster in order to experiment a bit with the system.
USAGE
---
To create a cluster, follow this steps:
1. Edit create-cluster and change the start / end port, depending on the
number of instances you want to create.
2. Use "./create-cluster start" in order to run the instances.
3. Use "./create-cluster create" in order to execute redis-trib create, so that
an actual Redis cluster will be created.
4. Now you are ready to play with the cluster. AOF files and logs for each instances are created in the current directory.
In order to stop a cluster:
1. Use "./craete-cluster stop" to stop all the instances. After you stopped the instances you can use "./create-cluster start" to restart them if you change ideas.
2. Use "./create-cluster clean" to remove all the AOF / log files to restat with a clean environment.
Use the command "./create-cluster help" to get the full list of features.
utils/create-cluster/create-cluster
-95
View File
@@ -1,95 +0,0 @@
#!/bin/bash
# Settings
PORT=30000
TIMEOUT=2000
NODES=6
REPLICAS=1
# You may want to put the above config parameters into config.sh in order to
# override the defaults without modifying this script.
if [ -a config.sh ]
then
source "config.sh"
fi
# Computed vars
ENDPORT=$((PORT+NODES))
if [ "$1" == "start" ]
then
while [ $((PORT < ENDPORT)) != "0" ]; do
PORT=$((PORT+1))
echo "Starting $PORT"
../../src/redis-server --port $PORT --cluster-enabled yes --cluster-config-file nodes-${PORT}.conf --cluster-node-timeout $TIMEOUT --appendonly yes --appendfilename appendonly-${PORT}.aof --dbfilename dump-${PORT}.rdb --logfile ${PORT}.log --daemonize yes
done
exit 0
fi
if [ "$1" == "create" ]
then
HOSTS=""
while [ $((PORT < ENDPORT)) != "0" ]; do
PORT=$((PORT+1))
HOSTS="$HOSTS 127.0.0.1:$PORT"
done
../../src/redis-trib.rb create --replicas $REPLICAS $HOSTS
exit 0
fi
if [ "$1" == "stop" ]
then
while [ $((PORT < ENDPORT)) != "0" ]; do
PORT=$((PORT+1))
echo "Stopping $PORT"
../../src/redis-cli -p $PORT shutdown nosave
done
exit 0
fi
if [ "$1" == "watch" ]
then
PORT=$((PORT+1))
while [ 1 ]; do
clear
date
../../src/redis-cli -p $PORT cluster nodes | head -30
sleep 1
done
exit 0
fi
if [ "$1" == "tail" ]
then
INSTANCE=$2
PORT=$((PORT+INSTANCE))
tail -f ${PORT}.log
exit 0
fi
if [ "$1" == "call" ]
then
while [ $((PORT < ENDPORT)) != "0" ]; do
PORT=$((PORT+1))
../../src/redis-cli -p $PORT $2 $3 $4 $5 $6 $7 $8 $9
done
exit 0
fi
if [ "$1" == "clean" ]
then
rm -rf *.log
rm -rf appendonly*.aof
rm -rf dump*.rdb
rm -rf nodes*.conf
exit 0
fi
echo "Usage: $0 [start|create|stop|watch|tail|clean]"
echo "start -- Launch Redis Cluster instances."
echo "create -- Create a cluster using redis-trib create."
echo "stop -- Stop Redis Cluster instances."
echo "watch -- Show CLUSTER NODES output (first 30 lines) of first node."
echo "tail <id> -- Run tail -f of instance at base port + ID."
echo "clean -- Remove all instances data, logs, configs."
utils/lru/README
-13
View File
@@ -1,13 +0,0 @@
The test-lru.rb program can be used in order to check the behavior of the
Redis approximated LRU algorithm against the theoretical output of true
LRU algorithm.
In order to use the program you need to recompile Redis setting the define
REDIS_LRU_CLOCK_RESOLUTION to 1, by editing redis.h.
This allows to execute the program in a fast way since the 1 ms resolution
is enough for all the objects to have a different enough time stamp during
the test.
The program is executed like this:
ruby test-lru.rb > /tmp/lru.html
utils/lru/test-lru.rb
-112
View File
@@ -1,112 +0,0 @@
require 'rubygems'
require 'redis'
r = Redis.new
r.config("SET","maxmemory","2000000")
r.config("SET","maxmemory-policy","allkeys-lru")
r.config("SET","maxmemory-samples",5)
r.config("RESETSTAT")
r.flushall
puts <<EOF
<html>
<body>
<style>
.box {
width:5px;
height:5px;
float:left;
margin: 1px;
}
.old {
border: 1px black solid;
}
.new {
border: 1px green solid;
}
.ex {
background-color: #666;
}
</style>
<pre>
EOF
# Fill
oldsize = r.dbsize
id = 0
while true
id += 1
r.set(id,"foo")
newsize = r.dbsize
break if newsize == oldsize
oldsize = newsize
end
inserted = r.dbsize
first_set_max_id = id
puts "#{r.dbsize} keys inserted"
# Access keys sequentially
puts "Access keys sequentially"
(1..first_set_max_id).each{|id|
r.get(id)
# sleep 0.001
}
# Insert more 50% keys. We expect that the new keys
half = inserted/2
puts "Insert enough keys to evict half the keys we inserted"
add = 0
while true
add += 1
id += 1
r.set(id,"foo")
break if r.info['evicted_keys'].to_i >= half
end
puts "#{add} additional keys added."
puts "#{r.dbsize} keys in DB"
# Check if evicted keys respect LRU
# We consider errors from 1 to N progressively more serious as they violate
# more the access pattern.
errors = 0
e = 1
edecr = 1.0/(first_set_max_id/2)
(1..(first_set_max_id/2)).each{|id|
e -= edecr if e > 0
e = 0 if e < 0
if r.exists(id)
errors += e
end
}
puts "#{errors} errors!"
puts "</pre>"
# Generate the graphical representation
(1..id).each{|id|
# Mark first set and added items in a different way.
c = "box"
if id <= first_set_max_id
c << " old"
else
c << " new"
end
# Add class if exists
c << " ex" if r.exists(id)
puts "<div class=\"#{c}\"></div>"
}
# Close HTML page
puts <<EOF
</body>
</html>
EOF