Commit Graph
9234 Commits
Author SHA1 Message Date
Oran Agra 0b2f7c6407 Daily github action: run cluster and sentinel tests with tls (#7575)
(cherry picked from commit 6d92eee69b)
2020-09-01 09:27:58 +03:00
Yossi Gottlieb 8a4240b1fa TLS: support cluster/replication without tls-port.
Initialize and configure OpenSSL even when tls-port is not used, because
we may still have tls-cluster or tls-replication.

Also, make sure to reconfigure OpenSSL when these parameters are changed
as TLS could have been enabled for the first time.

(cherry picked from commit c75512d89d)
2020-09-01 09:27:58 +03:00
grishaf 395d339234 Fix prepareForShutdown function declaration (#7566)
(cherry picked from commit 4126ca466f)
2020-09-01 09:27:58 +03:00
Oran Agra 6daa8b9adb Stabilize bgsave test that sometimes fails with valgrind (#7559)
on ci.redis.io the test fails a lot, reporting that bgsave didn't end.
increaseing the timeout we wait for that bgsave to get aborted.
in addition to that, i also verify that it indeed got aborted by
checking that the save counter wasn't reset.

add another test to verify that a successful bgsave indeed resets the
change counter.

(cherry picked from commit 8a57969fd7)
2020-09-01 09:27:58 +03:00
Madelyn Olson e00b6248c9 Properly reset errno for rdbLoad (#7542)
(cherry picked from commit 818dc3a089)
2020-09-01 09:27:58 +03:00
Oran Agra e8aa5583d0 testsuite may leave servers alive on error (#7549)
in cases where you have
test name {
  start_server {
    start_server {
      assert
    }
  }
}

the exception will be thrown to the test proc, and the servers are
supposed to be killed on the way out. but it seems there was always a
bug of not cleaning the server stack, and recently (#7404) we started
relying on that stack in order to kill them, so with that bug sometimes
we would have tried to kill the same server twice, and leave one alive.

luckly, in most cases the pattern is:
start_server {
  test name {
  }
}

(cherry picked from commit 36b9494385)
2020-09-01 09:27:58 +03:00
Yossi Gottlieb f1d5d5d28e Tests: drop TCL 8.6 dependency. (#7548)
This re-implements the redis-cli --pipe test so it no longer depends on a close feature available only in TCL 8.6.

Basically what this test does is run redis-cli --pipe, generates a bunch of commands and pipes them through redis-cli, and inspects the result in both Redis and the redis-cli output.

To do that, we need to close stdin for redis-cli to indicate we're done so it can flush its buffers and exit. TCL has bi-directional channels can only offers a way to "one-way close" a channel with TCL 8.6. To work around that, we now generate the commands into a file and feed that file to redis-cli directly.

As we're writing to an actual file, the number of commands is now reduced.

(cherry picked from commit f57e844b2e)
2020-09-01 09:27:58 +03:00
Oran Agra cc7f4ba67e Fixes to release scripts (#7547)
(cherry picked from commit 343dd9bcce)
2020-09-01 09:27:58 +03:00
Remi Collet af907e4b6d Fix deprecated tail syntax in tests (#7543)
(cherry picked from commit 3f2fbc4c61)
2020-09-01 09:27:58 +03:00
Wen Hui d2f09a054c Add missing calls to raxStop (#7532)
Since the dynamic allocations in raxIterator are only used for deep walks, memory
leak due to missing call to raxStop can only happen for rax with key names longer
than 32 bytes.

Out of all the missing calls, the only ones that may lead to a leak are the rax
for consumer groups and consumers, and these were only in AOFRW and rdbSave, which
normally only happen in fork or at shutdown.

(cherry picked from commit 4e8f2d6881)
2020-09-01 09:27:58 +03:00
Wen Hui cef46edebc add missing caching command in client help (#7399)
(cherry picked from commit 2fbd0271f6)
2020-09-01 09:27:58 +03:00
zhaozhao.zz 9e997cd0d1 replication: need handle -NOPERM error after send ping (#7538)
(cherry picked from commit 13e50935a8)
2020-09-01 09:27:58 +03:00
Scott Brenner 171aa22b0b GitHub Actions workflows - use latest version of actions/checkout (#7534)
(cherry picked from commit 2f4e9c3f9f)
2020-09-01 09:27:58 +03:00
Oran Agra 7bf665f125 Redis 6.0.6.
CI / test-ubuntu-latest (push) Has been cancelled
CI / build-ubuntu-old (push) Has been cancelled
CI / build-macos-latest (push) Has been cancelled
CI / build-32bit (push) Has been cancelled
CI / build-libc-malloc (push) Has been cancelled
6.0.6
2020-07-20 21:08:26 +03:00
Oran Agra a5696bdf4f Run daily CI on PRs to release a branch 2020-07-20 21:08:26 +03:00
Itamar Haber e15528bf1d Adds SHA256SUM to redis-stable tarball upload
(cherry picked from commit 5df0a64d30)
2020-07-20 21:08:26 +03:00
yoav-steinberg e28aa99af1 Support passing stack allocated module strings to moduleCreateArgvFromUserFormat (#7528)
Specifically, the key passed to the module aof_rewrite callback is a stack allocated robj. When passing it to RedisModule_EmitAOF (with appropriate "s" fmt string) redis used to panic when trying to inc the ref count of the stack allocated robj. Now support such robjs by coying them to a new heap robj. This doesn't affect performance because using the alternative "c" or "b" format strings also copies the input to a new heap robj.

(cherry picked from commit d484b8a04e)
2020-07-20 21:08:26 +03:00
Luke Palmer 3051430040 Send null for invalidate on flush (#7469)
(cherry picked from commit 5f716ea467)
2020-07-20 21:08:26 +03:00
dmurnane 29b20fd528 Notify systemd on sentinel startup (#7168)
Co-authored-by: Daniel Murnane <dmurnane@eitccorp.com>
(cherry picked from commit 9242ccf238)
2020-07-20 21:08:26 +03:00
Developer-Ecosystem-Engineering 5b36681213 Add registers dump support for Apple silicon (#7453)
Export following environment variables before building on macOS on Apple silicon

export ARCH_FLAGS="-arch arm64"
export SDK_NAME=macosx
export SDK_PATH=$(xcrun --show-sdk-path --sdk $SDK_NAME)
export CFLAGS="$ARCH_FLAGS -isysroot $SDK_PATH -I$SDK_PATH/usr/include"
export CXXFLAGS=$CFLAGS
export LDFLAGS="$ARCH_FLAGS"
export CC="$(xcrun -sdk $SDK_PATH --find clang) $CFLAGS"
export CXX="$(xcrun -sdk $SDK_PATH --find clang++) $CXXFLAGS"
export LD="$(xcrun -sdk $SDK_PATH --find ld) $LDFLAGS"

make
make test
..
All tests passed without errors!

Backtrack logging assumes x86 and required updating

(cherry picked from commit c2b5f1c15b)
2020-07-20 21:08:26 +03:00
Wen Hui 0f75036c07 correct error msg for num connections reaching maxclients in cluster mode (#7444)
(cherry picked from commit d85af4d6f5)
2020-07-20 21:08:26 +03:00
WuYunlong b1a01fda91 Fix command help for unexpected options (#7476)
(cherry picked from commit 93bdbf5aa4)
2020-07-20 21:08:26 +03:00
WuYunlong 83f55f61a6 Refactor RM_KeyType() by using macro. (#7486)
(cherry picked from commit dc690161d5)
2020-07-20 21:08:26 +03:00
Oran Agra f89f50dbd0 diskless master disconnect replicas when rdb child failed (#7518)
in case the rdb child failed, crashed or terminated unexpectedly redis
would have marked the replica clients with repl_put_online_on_ack and
then kill them only after a minute when no ack was received.

it would not stream anything to these connections, so the only effect of
this bug is a delay of 1 minute in the replicas attempt to re-connect.

(cherry picked from commit a176cb56a3)
2020-07-20 21:08:26 +03:00
Oran Agra 05f8975d21 redis-cli tests, fix valgrind timing issue (#7519)
this test when run with valgrind on github actions takes 160 seconds

(cherry picked from commit 254c962554)
2020-07-20 21:08:26 +03:00
WuYunlong 4780cc5efa Fix out of update help info in tcl tests. (#7516)
Before this commit, the output of "./runtest-cluster --help" is incorrect.
After this commit, the format of the following 3 output is consistent:
./runtest --help
./runtest-cluster --help
./runtest-sentinel --help

(cherry picked from commit 8128d39737)
2020-07-20 21:08:26 +03:00
Qu Chen 9bc6af2449 Replica always reports master's config epoch in CLUSTER NODES output. (#7235)
(cherry picked from commit 938c35302f)
2020-07-20 21:08:26 +03:00
Oran Agra c4b428a388 RESTORE ABSTTL skip expired keys - leak (#7511)
(cherry picked from commit 6a81450144)
2020-07-20 21:08:26 +03:00
Oran Agra aea4db2f5a fix recently added time sensitive tests failing with valgrind (#7512)
interestingly the latency monitor test fails because valgrind is slow
enough so that the time inside PEXPIREAT command from the moment of
the first mstime() call to get the basetime until checkAlreadyExpired
calls mstime() again is more than 1ms, and that test was too sensitive.

using this opportunity to speed up the test (unrelated to the failure)
the fix is just the longer time passed to PEXPIRE.

(cherry picked from commit e5227aab89)
2020-07-20 21:08:26 +03:00
Oran Agra b5c5f870a4 runtest --stop pause stops before terminating the redis server (#7513)
in the majority of the cases (on this rarely used feature) we want to
stop and be able to connect to the shard with redis-cli.
since these are two different processes interracting with the tty we
need to stop both, and we'll have to hit enter twice, but it's not that
bad considering it is rarely used.

(cherry picked from commit 02ef355f98)
2020-07-20 21:08:26 +03:00
Oran Agra 23bf3d1cb9 update release scripts for new hosts, and CI to run more tests (#7480)
* update daily CI to include cluster and sentinel tests
* update daily CI to run when creating a new release
* update release scripts to work on the new redis.io hosts

(cherry picked from commit 7f19a04f0f)
2020-07-20 21:08:26 +03:00
jimgreen2013 1be6dbbdf6 fix description about ziplist, the code is ok (#6318)
* fix description about ZIP_BIG_PREVLEN(the code is ok), it's similar to
antirez#4705

* fix description about ziplist entry encoding field (the code is ok),
the max length should be 2^32 - 1 when encoding is 5 bytes

(cherry picked from commit 67660881ed)
2020-07-20 21:08:26 +03:00
马永泽 1acdc84d10 fix benchmark in cluster mode fails to authenticate (#7488)
Co-authored-by: Oran Agra <oran@redislabs.com> (styling)
(cherry picked from commit 279b4a1464)
2020-07-20 21:08:26 +03:00
Abhishek Soni 52929b61f8 fix: typo in CI job name (#7466)
(cherry picked from commit d5648d617e)
2020-07-20 21:08:26 +03:00
Jiayuan Chen b0faf6113e Fix typo in deps README (#7500)
(cherry picked from commit af39d750cd)
2020-07-20 21:08:26 +03:00
WuYunlong f838df92d2 Add missing latency-monitor tcl test to test_helper.tcl. (#6782)
(cherry picked from commit d792db7948)
2020-07-20 21:08:26 +03:00
Yossi Gottlieb 7a536c2912 TLS: Session caching configuration support. (#7420)
* TLS: Session caching configuration support.
* TLS: Remove redundant config initialization.

(cherry picked from commit 3e6f2b1a45)
2020-07-20 21:08:26 +03:00
Yossi Gottlieb 6f1e828454 TLS: Ignore client cert when tls-auth-clients off. (#7457)
(cherry picked from commit 5266293a0f)
2020-07-20 21:08:26 +03:00
James Hilliard 2d3687655b Use pkg-config to properly detect libssl and libcrypto libraries (#7452)
(cherry picked from commit 6a014af79a)
2020-07-20 21:08:26 +03:00
Yossi Gottlieb b057ff81ee TLS: Add missing redis-cli options. (#7456)
* Tests: fix and reintroduce redis-cli tests.

These tests have been broken and disabled for 10 years now!

* TLS: add remaining redis-cli support.

This adds support for the redis-cli --pipe, --rdb and --replica options
previously unsupported in --tls mode.

* Fix writeConn().

(cherry picked from commit d9f970d8d3)
2020-07-20 21:08:26 +03:00
Oran Agra a9d1014fb2 redis-cli --hotkeys fixed to handle non-printable key names
(cherry picked from commit b23e251036)
2020-07-20 21:08:26 +03:00
Oran Agra 183bdc11c0 redis-cli --bigkeys fixed to handle non-printable key names
(cherry picked from commit 6f8a8647de)
2020-07-20 21:08:26 +03:00
Oran Agra 95ba01b538 RESTORE ABSTTL won't store expired keys into the db (#7472)
Similarly to EXPIREAT with TTL in the past, which implicitly deletes the
key and return success, RESTORE should not store key that are already
expired into the db.
When used together with REPLACE it should emit a DEL to keyspace
notification and replication stream.

(cherry picked from commit 5977a94842)
2020-07-20 21:08:26 +03:00
huangzhw 7b21b8c3fb defrag.c activeDefragSdsListAndDict when defrag sdsele, We can't use (#7492)
it to calculate hash, we should use newsds.

(cherry picked from commit d6180c8c86)
2020-07-20 21:08:26 +03:00
Oran Agra 33ca884cc5 skip a test that uses +inf on valgrind (#7440)
On some platforms strtold("+inf") with valgrind returns a non-inf result

[err]: INCRBYFLOAT does not allow NaN or Infinity in tests/unit/type/incr.tcl
Expected 'ERR*would produce*' to equal or match '1189731495357231765085759.....'

(cherry picked from commit 909bc97c52)
2020-07-20 21:08:26 +03:00
Oran Agra 2b5f23197c stabilize tests that look for log lines (#7367)
tests were sensitive to additional log lines appearing in the log
causing the search to come empty handed.

instead of just looking for the n last log lines, capture the log lines
before performing the action, and then search from that offset.

(cherry picked from commit 8e76e13472)
2020-07-20 21:08:26 +03:00
Oran Agra 1104113c07 tests/valgrind: don't use debug restart (#7404)
* tests/valgrind: don't use debug restart

DEBUG REATART causes two issues:
1. it uses execve which replaces the original process and valgrind doesn't
   have a chance to check for errors, so leaks go unreported.
2. valgrind report invalid calls to close() which we're unable to resolve.

So now the tests use restart_server mechanism in the tests, that terminates
the old server and starts a new one, new PID, but same stdout, stderr.

since the stderr can contain two or more valgrind report, it is not enough
to just check for the absence of leaks, we also need to check for some known
errors, we do both, and fail if we either find an error, or can't find a
report saying there are no leaks.

other changes:
- when killing a server that was already terminated we check for leaks too.
- adding DEBUG LEAK which was used to test it.
- adding --trace-children to valgrind, although no longer needed.
- since the stdout contains two or more runs, we need slightly different way
  of checking if the new process is up (explicitly looking for the new PID)
- move the code that handles --wait-server to happen earlier (before
  watching the startup message in the log), and serve the restarted server too.

* squashme - CR fixes

(cherry picked from commit 69ade87325)
2020-07-20 21:08:26 +03:00
Oran Agra 62f8583445 change references to the github repo location (#7479)
(cherry picked from commit 9bbf768d3c)
2020-07-20 21:08:26 +03:00
zhaozhao.zz 3155adb299 BITOP: propagate only when it really SET or DEL targetkey (#5783)
For example:
    BITOP not targetkey sourcekey

If targetkey and sourcekey doesn't exist, BITOP has no effect,
we do not propagate it, thus can save aof and replica flow.

(cherry picked from commit 1978f996d8)
2020-07-20 21:08:26 +03:00
antirez 14a59d4ce7 Update comment to clarify change in #7398.
(cherry picked from commit ad0a9df77a)
2020-07-20 21:08:26 +03:00