Compare commits

...
31 Commits
Author SHA1 Message Date
Oran Agra ae6a2aa95c Release Redis 7.2.6
CI / test-ubuntu-latest (push) Has been cancelled
CI / test-sanitizer-address (push) Has been cancelled
CI / build-debian-old (push) Has been cancelled
CI / build-macos-latest (push) Has been cancelled
CI / build-32bit (push) Has been cancelled
CI / build-libc-malloc (push) Has been cancelled
CI / build-centos7-jemalloc (push) Has been cancelled
External Server Tests / test-external-standalone (push) Has been cancelled
External Server Tests / test-external-cluster (push) Has been cancelled
External Server Tests / test-external-nodebug (push) Has been cancelled
Spellcheck / Spellcheck (push) Has been cancelled
2024-10-02 22:13:33 +03:00
Oran Agra c8649f8e85 Prevent pattern matching abuse (CVE-2024-31228) 2024-10-02 22:13:33 +03:00
Oran Agra b351d5a321 Fix ACL SETUSER Read/Write key pattern selector (CVE-2024-31227)
The '%' rule must contain one or both of R/W
2024-10-02 22:13:33 +03:00
Oran Agra fe8de4313f Fix lua bit.tohex (CVE-2024-31449)
INT_MIN value must be explicitly checked, and cannot be negated.
2024-10-02 22:13:33 +03:00
debing.sun 2ad2548747 Fixed crashes due to missed slotToKeyInit() and missed expires_cursor reset (#13315)
this PR fixes two crashes:

1. Fix missing slotToKeyInit() when using `flushdb async` under cluster
mode.
    https://github.com/redis/redis/issues/13205

2. Fix missing expires_cursor reset when stopping active defrag in the
middle of defragment.
    https://github.com/redis/redis/issues/13307
If we stop active defrag in the middle of defragging db->expires, if
`expires_cursor` is not reset to 0, the next time we enable active
defrag again, defragLaterStep(db, ...) will be entered. However, at this
time, `db` has been reset to NULL, which results in crash.

The affected code were removed by #11695 and #13058 in usntable, so we
just need backport this to 7.2.
2024-06-18 18:02:22 +08:00
YaacovHazan f60370ce28 Redis 7.2.5
CI / test-ubuntu-latest (push) Has been cancelled
CI / test-sanitizer-address (push) Has been cancelled
CI / build-debian-old (push) Has been cancelled
CI / build-macos-latest (push) Has been cancelled
CI / build-32bit (push) Has been cancelled
CI / build-libc-malloc (push) Has been cancelled
CI / build-centos7-jemalloc (push) Has been cancelled
External Server Tests / test-external-standalone (push) Has been cancelled
External Server Tests / test-external-cluster (push) Has been cancelled
External Server Tests / test-external-nodebug (push) Has been cancelled
Spellcheck / Spellcheck (push) Has been cancelled
2024-05-19 09:12:35 +03:00
Yanqi Lv 464aad9ee7 fix wrong data type conversion in zrangeResultBeginStore (#13148)
In `beginResultEmission`, -1 means the result length is not known in
advance. But after #12185, if we pass -1 to `zrangeResultBeginStore`, it
will convert to SIZE_MAX in `zsetTypeCreate` and try to `dictExpand`.
Although `dictExpand` won't succeed because the size overflows, I think
we'd better to avoid this wrong conversion.

This bug can be triggered when the source of `zrangestore` doesn't exist
or we use `zrangestore` command with `byscore` or `bylex`.
The impact is that dst keys will be converted to use skiplist instead of
listpack.

(cherry picked from commit bad33f8738)
2024-05-19 09:12:35 +03:00
Binbin 439b8da475 Fix redis-check-aof incorrectly considering data in manifest format as MP-AOF (#12958)
The check in fileIsManifest misjudged the manifest file. For example,
if resp aof contains "file", it will be considered a manifest file and
the check will fail:
```
*3
$3
set
$4
file
$4
file
```

In #12951, if the preamble aof also contains it, it will also fail.
Fixes #12951.

the bug was happening if the the word "file" is mentioned
in the first 1024 lines of the AOF. and now as soon as it finds
a non-comment line it'll break (if it contains "file" or doesn't)

(cherry picked from commit da727ad445)
2024-05-19 09:12:35 +03:00
Matthew Douglass 1caaf581f1 Fix conversion of numbers in lua args to redis args (#13115)
Since lua_Number is not explicitly an integer or a double, we need to
make an effort
to convert it as an integer when that's possible, since the string could
later be used
in a context that doesn't support scientific notation (e.g. 1e9 instead
of 100000000).

Since fpconv_dtoa converts numbers with the equivalent of `%f` or `%e`,
which ever is shorter,
this would break if we try to pass a long integer number to a command
that takes integer.
we'll get an implicit conversion to string in Lua, and then the parsing
in getLongLongFromObjectOrReply will fail.

```
> eval "redis.call('hincrby', 'key', 'field', '1000000000')" 0
(nil)
> eval "redis.call('hincrby', 'key', 'field', tonumber('1000000000'))" 0
(error) ERR value is not an integer or out of range script: ac99c32e4daf7e300d593085b611de261954a946, on @user_script:1.
```

Switch to using ll2string if the number can be safely represented as a
long long.

The problem was introduced in #10587 (Redis 7.2).
closes #13113.

---------

Co-authored-by: Binbin <binloveplay1314@qq.com>
Co-authored-by: debing.sun <debing.sun@redis.com>
Co-authored-by: Oran Agra <oran@redislabs.com>
(cherry picked from commit 5fdaa53d20)
2024-05-19 09:12:35 +03:00
debing.sun e46ddde28d Check user's oom_score_adj write permission for oom-score-adj test (#13111)
`CONFIG SET oom-score-adj handles configuration failures` test failed in
some CI jobs today.
Failed CI: https://github.com/redis/redis/actions/runs/8152519326

Not sure why the github action's docker image perssions have changed,
but the issue is similar to #12887,
where we can't assume the range of oom_score_adj that a user can change.

## Solution:
Modify the way of determining whether the current user has no privileges
or not,
instead of relying on whether the user id is 0 or not.

(cherry picked from commit 9738ba9841)
2024-05-19 09:12:35 +03:00
LiiNen 8f70fcc65b Fix redis-cli --count (for --scan, --bigkeys, etc) was ignored unless --pattern was also used (#13092)
The --count option for redis-cli has been released in redis 7.2.
https://github.com/redis/redis/pull/12042
But I have found in code, that some logic was missing for using this
'count' option.

```
static redisReply *sendScan(unsigned long long *it) {
    redisReply *reply;

    if (config.pattern)
        reply = redisCommand(context, "SCAN %llu MATCH %b COUNT %d",
            *it, config.pattern, sdslen(config.pattern), config.count);
    else
        reply = redisCommand(context,"SCAN %llu",*it);
```

The intention was being able to using scan count.
But in this case, the --count will be only applied when 'pattern' is
declared.
So, I had fix it simply, to be worked properly - even if --pattern
option is not being used.

I tested it simply with time() command several times, and I could see it
works as intended with this commit.
The examples of test results are below:
```
# unstable build

time(./redis-cli -a $AUTH -p $PORT -h $HOST --scan >/dev/null 2>/dev/null)

real    0m1.287s
user    0m0.011s
sys     0m0.022s

# count is not applied
time(./redis-cli -a $AUTH -p $PORT -h $HOST --scan --count 1000 >/dev/null 2>/dev/null)

real    0m1.117s
user    0m0.011s
sys     0m0.020s

# count is applied with --pattern

time(./redis-cli -a $AUTH -p $PORT -h $HOST --scan --count 1000 --pattern "hash:*" >/dev/null 2>/dev/null)

real    0m0.045s
user    0m0.002s
sys     0m0.002s
```

```
# fix-redis-cli-scan-count build
time(./redis-cli -a $AUTH -p $PORT -h $HOST --scan >/dev/null 2>/dev/null)

real    0m1.084s
user    0m0.008s
sys     0m0.024s

# count is applied even if --pattern is not declared
time(./redis-cli -a $AUTH -p $PORT -h $HOST --scan --count 1000 >/dev/null 2>/dev/null)

real    0m0.043s
user    0m0.000s
sys     0m0.004s

# of course this also applied
time(./redis-cli -a $AUTH -p $PORT -h $HOST --scan --count 1000 --pattern "hash:*" >/dev/null 2>/dev/null)

real    0m0.031s
user    0m0.002s
sys     0m0.002s
```

Thanks a lot.

(cherry picked from commit 763827c981)
2024-05-19 09:12:35 +03:00
Binbin 8d3a1c978a Increase tolerance range to block reprocess tests to avoid timing issues (#13053)
These tests have all failed in daily CI:
```
*** [err]: Blocking XREADGROUP for stream key that has clients blocked on stream - reprocessing command in tests/unit/type/stream-cgroups.tcl
Expected '1101' to be between to '1000' and '1100' (context: type eval line 23 cmd {assert_range [expr $end-$start] 1000 1100} proc ::test)

*** [err]: BLPOP unblock but the key is expired and then block again - reprocessing command in tests/unit/type/list.tcl
Expected '1101' to be between to '1000' and '1100' (context: type eval line 23 cmd {assert_range [expr $end-$start] 1000 1100} proc ::test)

*** [err]: BZPOPMIN unblock but the key is expired and then block again - reprocessing command in tests/unit/type/zset.tcl
Expected '1103' to be between to '1000' and '1100' (context: type eval line 23 cmd {assert_range [expr $end-$start] 1000 1100} proc ::test)
```

Increase the range to avoid failures, and improve the comment to be
clearer.
tests was introduced in #13004.

(cherry picked from commit 32f44da510)
2024-05-19 09:12:35 +03:00
debing.sun c34e64848f Fix crash due to merge of quicklist node introduced by #12955 (#13040)
Fix two crash introducted by #12955

When a quicklist node can't be inserted and split, we eventually merge
the current node with its neighboring
nodes after inserting, and compress the current node and its siblings.

1. When the current node is merged with another node, the current node
may become invalid and can no longer be used.

   Solution: let `_quicklistMergeNodes()` return the merged nodes.

3. If the current node is a LZF quicklist node, its recompress will be
1. If the split node can be merged with a sibling node to become head or
tail, recompress may cause the head and tail to be compressed, which is
not allowed.

    Solution: always recompress to 0 after merging.

(cherry picked from commit 1e8dc1da0d)
2024-05-19 09:12:35 +03:00
debing.sun 1be66b98f5 Prevent LSET command from causing quicklist plain node size to exceed 4GB (#12955)
Fix #12864

The main reason for this crash is that when replacing a element of a
quicklist packed node with lpReplace() method,
if the final size is larger than 4GB, lpReplace() will fail and returns
NULL, causing `node->entry` to be incorrectly set to NULL.

Since the inserted data is not a large element, we can't just replace it
like a large element, first quicklistInsertAfter()
and then quicklistDelIndex(), because the current node may be merged and
invalidated in quicklistInsertAfter().

The solution of this PR:
When replacing a node fails (listpack exceeds 4GB), split the current
node, create a new node to put in the middle, and try to merge them.
This is the same as inserting a large element.
In the worst case, its size will not exceed 4GB.

(cherry picked from commit 1f00c951c2)
2024-05-19 09:12:35 +03:00
Binbin 423d1909c5 Fix timeout not being set in module blockClient case (#13011)
This was introduced in #13004, missing this assignment.
It causes timeout to be a random value (may be less than now),
and then in `Unblock by timer` test, the client is unblocked
and then it call timeout_callback, since the callback is NULL,
the server will crash.

The crash stack is:
```
beforesleep
handleBlockedClientsTimeout
checkBlockedClientTimeout
unblockClientOnTimeout
replyToBlockedClientTimedOut
moduleBlockedClientTimedOut
-- the timeout_callback is NULL, invalidFunctionWasCalled
bc->timeout_callback(&ctx,(void**)c->argv,c->argc);
```

(cherry picked from commit 45a35a79c7)
2024-05-19 09:12:35 +03:00
Binbin 1bda797dc0 Fix blocking commands timeout is reset due to re-processing command (#13004)
In #11012, we will reprocess command when client is unblocked on keys,
in some blocking commands, for example, in the XREADGROUP BLOCK
scenario,
because of the re-processing command, we will recalculate the block
timeout,
causing the blocking time to be reset.

This commit add a new CLIENT_REPROCESSING_COMMAND clent flag, explicitly
let the command know that it is being re-processed, later in
blockForKeys
we will not reset the timeout.

Affected BLOCK cases:
- list / zset / stream, added test cases for each.

Unaffected cases:
- module (never re-process the commands).
- WAIT / WAITAOF (never re-process the commands).

Fixes #12998.

(cherry picked from commit 492021db95)
2024-05-19 09:12:35 +03:00
Oran Agra d5bae505ba update redis-check-rdb types (#12969)
seems that we forgot to update the array in redis-check rdb.

(cherry picked from commit f9a0eb60f7)
2024-05-19 09:12:35 +03:00
bentotten 099a2f40ec When one shard, sole primary node marks potentially failed replica as FAIL instead of PFAIL (#12824)
Fixes issue where a single primary cannot mark a replica as failed in a
single-shard cluster.

(cherry picked from commit b3aaa0a136)
2024-05-19 09:12:35 +03:00
Binbin 4bd614c52d Add announced-endpoints test to all_tests and fix tls related tests (#12927)
The test was introduced in #10745, but we forgot to add it to the
test_helper.tcl, so our CI did not actually run it. This PR adds it
and ensures it passes CI tests.

(cherry picked from commit b351a04b1e)
2024-05-19 09:12:35 +03:00
Oran Agra d2c8a4b91e Redis 7.2.4
CI / test-ubuntu-latest (push) Has been cancelled
CI / test-sanitizer-address (push) Has been cancelled
CI / build-debian-old (push) Has been cancelled
CI / build-macos-latest (push) Has been cancelled
CI / build-32bit (push) Has been cancelled
CI / build-libc-malloc (push) Has been cancelled
CI / build-centos7-jemalloc (push) Has been cancelled
External Server Tests / test-external-standalone (push) Has been cancelled
External Server Tests / test-external-cluster (push) Has been cancelled
External Server Tests / test-external-nodebug (push) Has been cancelled
Spellcheck / Spellcheck (push) Has been cancelled
2024-01-09 13:51:49 +02:00
Binbin 85408b7391 Fix CLUSTER SHARDS crash in 7.0/7.2 mixed clusters where shard ids are not sync (#12832)
Crash reported in #12695. In the process of upgrading the cluster from
7.0 to 7.2, because the 7.0 nodes will not gossip shard id, in 7.2 we
will rely on shard id to build the server.cluster->shards dict.

In some cases, for example, the 7.0 master node and the 7.2 replica node.
From the view of 7.2 replica node, the cluster->shards dictionary does not
have its master node. In this case calling CLUSTER SHARDS on the 7.2 replica
node may crash.

We should fix the underlying assumption of updateShardId, which is that the
shard dict should be always in sync with the node's shard_id. The fix was
suggested by PingXie, see more details in #12695.

(cherry picked from commit 5b0c6a8255)
2024-01-09 13:51:49 +02:00
Binbin 5a2f4a1e94 Use shard-id of the master if the replica does not support shard-id (#12805)
If there are nodes in the cluster that do not support shard-id, they
will gossip shard-id. From the perspective of nodes that support shard-id,
their shard-id is meaningless (since shard-id is randomly generated when
we create a node.)

Nodes that support shard-id will save the shard-id information in nodes.conf.
If the node is restarted according to nodes.conf, the server will report a
corrupted cluster config file error. Because auxShardIdSetter will reject
configurations with inconsistent master-replica shard-ids.

A cluster-wide consensus for the node's shard_id is not necessary. The key
is maintaining consistency of the shard_id on each individual 7.2 node.
As the cluster progressively upgrades to version 7.2, we can expect the
shard_ids across all nodes to naturally converge and align.

In this PR, when processing the gossip, if sender is a replica and does not
support shard-id, set the shard_id to the shard_id of its master.

(cherry picked from commit 4cae66f5e8)
2024-01-09 13:51:49 +02:00
Binbin c4776cafcf Un-register notification and server event when RedisModule_OnLoad fails (#12809)
When we register notification or server event in RedisModule_OnLoad, but
RedisModule_OnLoad eventually fails, triggering notification or server
event
will cause the server to crash.

If the loading fails on a later stage of moduleLoad, we do call
moduleUnload
which handles all un-registration, but when it fails on the
RedisModule_OnLoad
call, we only un-register several specific things and these were
missing:

- moduleUnsubscribeNotifications
- moduleUnregisterFilters
- moduleUnsubscribeAllServerEvents

Refactored the code to reuse the code from moduleUnload.

Fixes #12808.

(cherry picked from commit d6f19539d2)
2024-01-09 13:51:49 +02:00
Meir Shpilraien (Spielrein) 4cbf903083 Before evicted and before expired server events are not executed inside an execution unit. (#12733)
Redis 7.2 (#9406) introduced a new modules event, `RedisModuleEvent_Key`.
This new event allows the module to read the key data just before it is removed
from the database (either deleted, expired, evicted, or overwritten).

When the key is removed from the database, either by active expire or eviction.
The new event was not called as part of an execution unit. This can cause an
issue if the module registers a post notification job inside the event. This job will
not be executed atomically with the expiration/eviction operation and will not
replicated inside a Multi/Exec. Moreover, the post notification job will be executed
right after the event where it is still not safe to perform any write operation, this will
violate the promise that post notification job will be called atomically with the
operation that triggered it and **only when it is safe to write**.

This PR fixes the issue by wrapping each expiration/eviction of a key with an execution
unit. This makes sure the entire operation will run atomically and all the post notification
jobs will be executed at the end where it is safe to write.

Tests were modified to verify the fix.

(cherry picked from commit 0ffb9d2ea9)
2024-01-09 13:51:49 +02:00
Sankar a91b57eff7 Clear owner_not_claiming_slot bit for the slot in clusterDelSlot (#12564)
Clear owner_not_claiming_slot bit for the slot in clusterDelSlot to keep it
consistent with slot ownership information.

(cherry picked from commit 8cdeddc81c)
2024-01-09 13:51:49 +02:00
Binbin 8359ce266c Use CLZ in _dictNextExp to get the next power of two (#12815)
In the past, we did not call _dictNextExp frequently. It was only
called when the dictionary was expanded.

Later, dictTypeExpandAllowed was introduced in #7954, which is 6.2.
For the data dict and the expire dict, we can check maxmemory before
actually expanding the dict. This is a good optimization to avoid
maxmemory being exceeded due to the dict expansion.

And in #11692, we moved the dictTypeExpandAllowed check before the
threshold check, this caused a bit of performance degradation, every
time a key is added to the dict, dictTypeExpandAllowed is called to
check.

The main reason for degradation is that in a large dict, we need to
call _dictNextExp frequently, that is, every time we add a key, we
need to call _dictNextExp once. Then the threshold is checked to see
if the dict needs to be expanded. We can see that the order of checks
here can be optimized.

So we moved the dictTypeExpandAllowed check back to after the threshold
check in #12789. In this way, before the dict is actually expanded (that
is, before the threshold is reached), we will not do anything extra
compared to before, that is, we will not call _dictNextExp frequently.

But note we'll still hit the degradation when we over the thresholds.
When the threshold is reached, because #7954, we may delay the dict
expansion due to maxmemory limitations. In this case, we will call
_dictNextExp every time we add a key during this period.

This PR use CLZ in _dictNextExp to get the next power of two. CLZ (count
leading zeros) can easily give you the next power of two. It should be
noted that we have actually introduced the use of __builtin_clzl in
#8687,
which is 7.0. So i suppose all the platforms we use have it (even if the
CPU doesn't have an instruction).

We build 67108864 (2**26) keys through DEBUG POPULTE, which will use
approximately 5.49G memory (used_memory:5898522936). If expansion is
triggered, the additional hash table will consume approximately 1G
memory (2 ** 27 * 8). So we set maxmemory to 6871947673 (that is, 6.4G),
which will be less than 5.49G + 1G, so we will delay the dict rehash
while addint the keys.

After that, each time an element is added to the dict, an allow check
will be performed, that is, we can frequently call _dictNextExp to test
the comparison before and after the optimization. Using DEBUG HTSTATS 0
to
check and make sure that our dict expansion is dealyed.

Using `./src/redis-server redis.conf --save "" --maxmemory 6871947673`.
Using `./src/redis-benchmark -P 100 -r 1000000000 -t set -n 5000000`.
After ten rounds of testing:
```
unstable:           this PR:
769585.94           816860.00
771724.00           818196.69
775674.81           822368.44
781983.12           822503.69
783576.25           828088.75
784190.75           828637.75
791389.69           829875.50
794659.94           835660.69
798212.00           830013.25
801153.62           833934.56
```

We can see there is about 4-5% performance improvement in this case.

(cherry picked from commit 22cc9b5122)
2024-01-09 13:51:49 +02:00
Binbin 856c8a47d0 Optimize dict expand check, move allow check after the thresholds check (#12789)
dictExpandAllowed (for the main db dict and the expire dict) seems to
involve a few function calls and memory accesses, and we can do it only
after the thresholds checks and can get some performance improvements.

A simple benchmark test: there are 11032768 fixed keys in the database,
start a redis-server with `--maxmemory big_number --save ""`,
start a redis-benchmark with `-P 100 -r 1000000000 -t set -n 5000000`,
collect `throughput summary: n requests per second` result.

After five rounds of testing:
```
unstable     this PR
848032.56    897988.56
854408.69    913408.88
858663.94    914076.81
871839.56    916758.31
882612.56    920640.75
```

We can see a 5% performance improvement in general condition.
But note we'll still hit the degradation when we over the thresholds.

(cherry picked from commit 463476933c)
2024-01-09 13:51:49 +02:00
Oran Agra 5f5f298a4a Fix possible corruption in sdsResize (CVE-2023-41056)
#11766 introduced a bug in sdsResize where it could forget to update
the sds type in the sds header and then cause an overflow in sdsalloc.
it looks like the only implication of that is a possible assertion in HLL,
but it's hard to rule out possible heap corruption issues with clientsCronResizeQueryBuffer
2024-01-09 13:51:49 +02:00
Oran Agra 7f4bae8176 Redis 7.2.3
CI / test-ubuntu-latest (push) Has been cancelled
CI / test-sanitizer-address (push) Has been cancelled
CI / build-debian-old (push) Has been cancelled
CI / build-macos-latest (push) Has been cancelled
CI / build-32bit (push) Has been cancelled
CI / build-libc-malloc (push) Has been cancelled
CI / build-centos7-jemalloc (push) Has been cancelled
External Server Tests / test-external-standalone (push) Has been cancelled
External Server Tests / test-external-cluster (push) Has been cancelled
External Server Tests / test-external-nodebug (push) Has been cancelled
Spellcheck / Spellcheck (push) Has been cancelled
2023-11-01 14:38:13 +02:00
Viktor Söderqvist cee6d309d9 Don't crash when adding a forgotten node to blacklist twice (#12702)
Add a defensive checks to prevent double freeing a node from the cluster blacklist.

(cherry picked from commit 8d675950e6)
2023-11-01 14:38:13 +02:00
Oran Agra cf013b83df Fix fd leak causing deleted files to remain open and eat disk space (#12693)
This was introduced in v7.2 by #11248

(cherry picked from commit ba900f6cb8)
2023-11-01 14:38:13 +02:00
47 changed files with 825 additions and 117 deletions
+72
View File
@@ -12,6 +12,78 @@ SECURITY: There are security fixes in the release.
--------------------------------------------------------------------------------
Redis Community Edition 7.2.6 Released Wed 02 Oct 2024 20:17:04 IDT
================================================================================
Upgrade urgency SECURITY: See security fixes below.
Security fixes
==============
* (CVE-2024-31449) Lua library commands may lead to stack overflow and potential RCE.
* (CVE-2024-31227) Potential Denial-of-service due to malformed ACL selectors.
* (CVE-2024-31228) Potential Denial-of-service due to unbounded pattern matching.
Bug fixes
=========
* Fixed crashes in cluster mode (#13315)
================================================================================
Redis 7.2.5 Released Thu 16 May 2024 12:00:00 IST
================================================================================
Upgrade urgency MODERATE: Program an upgrade of the server, but it's not urgent.
Bug fixes
=========
* A single shard cluster leaves failed replicas in CLUSTER SLOTS instead of removing them (#12824)
* Crash in LSET command when replacing small items and exceeding 4GB (#12955)
* Blocking commands timeout is reset due to re-processing command (#13004)
* Conversion of numbers in Lua args to redis args can fail. Bug introduced in 7.2.0 (#13115)
Bug fixes in CLI tools
======================
* redis-cli: --count (for --scan, --bigkeys, etc) was ignored unless --pattern was also used (#13092)
* redis-check-aof: incorrectly considering data in manifest format as MP-AOF (#12958)
================================================================================
Redis 7.2.4 Released Tue 09 Jan 2024 10:45:52 IST
================================================================================
Upgrade urgency SECURITY: See security fixes below.
Security fixes
==============
* (CVE-2023-41056) In some cases, Redis may incorrectly handle resizing of memory
buffers which can result in incorrect accounting of buffer sizes and lead to
heap overflow and potential remote code execution.
Bug fixes
=========
* Fix crashes of cluster commands clusters with mixed versions of 7.0 and 7.2 (#12805, #12832)
* Fix slot ownership not being properly handled when deleting a slot from a node (#12564)
* Fix atomicity issues with the RedisModuleEvent_Key module API event (#12733)
================================================================================
Redis 7.2.3 Released Wed 01 Nov 2023 12:00:00 IST
================================================================================
Upgrade urgency: HIGH, Fixes critical bugs affecting most users.
Bug fixes
=========
* Fix file descriptor leak preventing deleted files from freeing disk space on
replicas (#12693)
* Fix a possible crash after cluster node removal (#12702)
================================================================================
Redis 7.2.2 Released Wed 18 Oct 2023 10:33:40 IDT
================================================================================
+1
View File
@@ -132,6 +132,7 @@ static int bit_tohex(lua_State *L)
const char *hexdigits = "0123456789abcdef";
char buf[8];
int i;
if (n == INT32_MIN) n = INT32_MIN+1;
if (n < 0) { n = -n; hexdigits = "0123456789ABCDEF"; }
if (n > 8) n = 8;
for (i = (int)n; --i >= 0; ) { buf[i] = hexdigits[b & 15]; b >>= 4; }
+1 -1
View File
@@ -1051,7 +1051,7 @@ int ACLSetSelector(aclSelector *selector, const char* op, size_t oplen) {
flags |= ACL_READ_PERMISSION;
} else if (toupper(op[offset]) == 'W' && !(flags & ACL_WRITE_PERMISSION)) {
flags |= ACL_WRITE_PERMISSION;
} else if (op[offset] == '~') {
} else if (op[offset] == '~' && flags) {
offset++;
break;
} else {
+3 -1
View File
@@ -117,7 +117,9 @@ aofInfo *aofInfoDup(aofInfo *orig) {
return ai;
}
/* Format aofInfo as a string and it will be a line in the manifest. */
/* Format aofInfo as a string and it will be a line in the manifest.
*
* When update this format, make sure to update redis-check-aof as well. */
sds aofInfoFormat(sds buf, aofInfo *ai) {
sds filename_repr = NULL;
+6 -2
View File
@@ -370,7 +370,12 @@ void blockForKeys(client *c, int btype, robj **keys, int numkeys, mstime_t timeo
list *l;
int j;
c->bstate.timeout = timeout;
if (!(c->flags & CLIENT_REPROCESSING_COMMAND)) {
/* If the client is re-processing the command, we do not set the timeout
* because we need to retain the client's original timeout. */
c->bstate.timeout = timeout;
}
for (j = 0; j < numkeys; j++) {
/* If the key already exists in the dictionary ignore it. */
if (!(client_blocked_entry = dictAddRaw(c->bstate.keys,keys[j],NULL))) {
@@ -392,7 +397,6 @@ void blockForKeys(client *c, int btype, robj **keys, int numkeys, mstime_t timeo
listAddNodeTail(l,c);
dictSetVal(c->bstate.keys,client_blocked_entry,listLast(l));
/* We need to add the key to blocking_keys_unblock_on_nokey, if the client
* wants to be awakened if key is deleted (like XREADGROUP) */
if (unblock_on_nokey) {
+37 -7
View File
@@ -988,7 +988,7 @@ void clusterInit(void) {
server.cluster->myself = NULL;
server.cluster->currentEpoch = 0;
server.cluster->state = CLUSTER_FAIL;
server.cluster->size = 1;
server.cluster->size = 0;
server.cluster->todo_before_sleep = 0;
server.cluster->nodes = dictCreate(&clusterNodesDictType);
server.cluster->shards = dictCreate(&clusterSdsToListType);
@@ -1134,6 +1134,9 @@ void clusterReset(int hard) {
}
dictReleaseIterator(di);
/* Empty the nodes blacklist. */
dictEmpty(server.cluster->nodes_black_list, NULL);
/* Hard reset only: set epochs to 0, change node ID. */
if (hard) {
sds oldname;
@@ -1684,6 +1687,7 @@ void clusterRenameNode(clusterNode *node, char *newname) {
serverAssert(retval == DICT_OK);
memcpy(node->name, newname, CLUSTER_NAMELEN);
clusterAddNode(node);
clusterAddNodeToShard(node->shard_id, node);
}
void clusterAddNodeToShard(const char *shard_id, clusterNode *node) {
@@ -2231,6 +2235,7 @@ void clusterProcessGossipSection(clusterMsg *hdr, clusterLink *link) {
node->tls_port = msg_tls_port;
node->cport = ntohs(g->cport);
clusterAddNode(node);
clusterAddNodeToShard(node->shard_id, node);
}
}
@@ -2408,7 +2413,6 @@ void clusterUpdateSlotsConfigWith(clusterNode *sender, uint64_t senderConfigEpoc
}
clusterDelSlot(j);
clusterAddSlot(sender,j);
bitmapClearBit(server.cluster->owner_not_claiming_slot, j);
clusterDoBeforeSleep(CLUSTER_TODO_SAVE_CONFIG|
CLUSTER_TODO_UPDATE_STATE|
CLUSTER_TODO_FSYNC_CONFIG);
@@ -2655,8 +2659,7 @@ void clusterProcessPingExtensions(clusterMsg *hdr, clusterLink *link) {
clusterNode *n = clusterLookupNode(forgotten_node_ext->name, CLUSTER_NAMELEN);
if (n && n != myself && !(nodeIsSlave(myself) && myself->slaveof == n)) {
sds id = sdsnewlen(forgotten_node_ext->name, CLUSTER_NAMELEN);
dictEntry *de = dictAddRaw(server.cluster->nodes_black_list, id, NULL);
serverAssert(de != NULL);
dictEntry *de = dictAddOrFind(server.cluster->nodes_black_list, id);
uint64_t expire = server.unixtime + ntohu64(forgotten_node_ext->ttl);
dictSetUnsignedIntegerVal(de, expire);
clusterDelNode(n);
@@ -2674,11 +2677,24 @@ void clusterProcessPingExtensions(clusterMsg *hdr, clusterLink *link) {
/* We know this will be valid since we validated it ahead of time */
ext = getNextPingExt(ext);
}
/* If the node did not send us a hostname extension, assume
* they don't have an announced hostname. Otherwise, we'll
* set it now. */
updateAnnouncedHostname(sender, ext_hostname);
updateAnnouncedHumanNodename(sender, ext_humannodename);
/* If the node did not send us a shard-id extension, it means the sender
* does not support it (old version), node->shard_id is randomly generated.
* A cluster-wide consensus for the node's shard_id is not necessary.
* The key is maintaining consistency of the shard_id on each individual 7.2 node.
* As the cluster progressively upgrades to version 7.2, we can expect the shard_ids
* across all nodes to naturally converge and align.
*
* If sender is a replica, set the shard_id to the shard_id of its master.
* Otherwise, we'll set it now. */
if (ext_shardid == NULL) ext_shardid = clusterNodeGetMaster(sender)->shard_id;
updateShardId(sender, ext_shardid);
}
@@ -3022,6 +3038,10 @@ int clusterProcessPacket(clusterLink *link) {
clusterNodeAddSlave(master,sender);
sender->slaveof = master;
/* Update the shard_id when a replica is connected to its
* primary in the very first time. */
updateShardId(sender, master->shard_id);
/* Update config. */
clusterDoBeforeSleep(CLUSTER_TODO_SAVE_CONFIG);
}
@@ -4751,10 +4771,13 @@ void clusterCron(void) {
/* Timeout reached. Set the node as possibly failing if it is
* not already in this state. */
if (!(node->flags & (CLUSTER_NODE_PFAIL|CLUSTER_NODE_FAIL))) {
serverLog(LL_DEBUG,"*** NODE %.40s possibly failing",
node->name);
node->flags |= CLUSTER_NODE_PFAIL;
update_state = 1;
if (myself->flags & CLUSTER_NODE_MASTER && server.cluster->size == 1) {
markNodeAsFailingIfNeeded(node);
} else {
serverLog(LL_DEBUG,"*** NODE %.40s possibly failing", node->name);
}
}
}
}
@@ -4940,6 +4963,8 @@ int clusterDelSlot(int slot) {
/* Clear the slot bit. */
serverAssert(clusterNodeClearSlotBit(n,slot) == 1);
server.cluster->slots[slot] = NULL;
/* Make owner_not_claiming_slot flag consistent with slot ownership information. */
bitmapClearBit(server.cluster->owner_not_claiming_slot, slot);
return C_OK;
}
@@ -5707,7 +5732,7 @@ void addShardReplyForClusterShards(client *c, list *nodes) {
addReplyBulkCString(c, "slots");
/* Use slot_info_pairs from the primary only */
while (n->slaveof != NULL) n = n->slaveof;
n = clusterNodeGetMaster(n);
if (n->slot_info_pairs != NULL) {
serverAssert((n->slot_info_pairs_count % 2) == 0);
@@ -7641,6 +7666,11 @@ unsigned int countKeysInSlot(unsigned int hashslot) {
return (*server.db->slots_to_keys).by_slot[hashslot].count;
}
clusterNode *clusterNodeGetMaster(clusterNode *node) {
while (node->slaveof != NULL) node = node->slaveof;
return node;
}
/* -----------------------------------------------------------------------------
* Operation(s) on channel rax tree.
* -------------------------------------------------------------------------- */
+1
View File
@@ -413,6 +413,7 @@ void clusterInit(void);
void clusterInitListeners(void);
void clusterCron(void);
void clusterBeforeSleep(void);
clusterNode *clusterNodeGetMaster(clusterNode *node);
clusterNode *getNodeByQuery(client *c, struct redisCommand *cmd, robj **argv, int argc, int *hashslot, int *ask);
int verifyClusterNodeId(const char *name, int length);
clusterNode *clusterLookupNode(const char *name, int length);
+1
View File
@@ -941,6 +941,7 @@ void activeDefragCycle(void) {
defrag_later_cursor = 0;
current_db = -1;
cursor = 0;
expires_cursor = 0;
db = NULL;
goto update_metrics;
}
+5 -10
View File
@@ -1414,30 +1414,25 @@ static int _dictExpandIfNeeded(dict *d)
* table (global setting) or we should avoid it but the ratio between
* elements/buckets is over the "safe" threshold, we resize doubling
* the number of buckets. */
if (!dictTypeExpandAllowed(d))
return DICT_OK;
if ((dict_can_resize == DICT_RESIZE_ENABLE &&
d->ht_used[0] >= DICTHT_SIZE(d->ht_size_exp[0])) ||
(dict_can_resize != DICT_RESIZE_FORBID &&
d->ht_used[0] / DICTHT_SIZE(d->ht_size_exp[0]) > dict_force_resize_ratio))
{
if (!dictTypeExpandAllowed(d))
return DICT_OK;
return dictExpand(d, d->ht_used[0] + 1);
}
return DICT_OK;
}
/* TODO: clz optimization */
/* Our hash table capability is a power of two */
static signed char _dictNextExp(unsigned long size)
{
unsigned char e = DICT_HT_INITIAL_EXP;
if (size <= DICT_HT_INITIAL_SIZE) return DICT_HT_INITIAL_EXP;
if (size >= LONG_MAX) return (8*sizeof(long)-1);
while(1) {
if (((unsigned long)1<<e) >= size)
return e;
e++;
}
return 8*sizeof(long) - __builtin_clzl(size-1);
}
/* Finds and returns the position within the dict where the provided key should
+2
View File
@@ -667,6 +667,7 @@ int performEvictions(void) {
*
* AOF and Output buffer memory will be freed eventually so
* we only care about memory used by the key space. */
enterExecutionUnit(1, 0);
delta = (long long) zmalloc_used_memory();
latencyStartMonitor(eviction_latency);
dbGenericDelete(db,keyobj,server.lazyfree_lazy_eviction,DB_FLAG_KEY_EVICTED);
@@ -679,6 +680,7 @@ int performEvictions(void) {
notifyKeyspaceEvent(NOTIFY_EVICTED, "evicted",
keyobj, db->id);
propagateDeletion(db,keyobj,server.lazyfree_lazy_eviction);
exitExecutionUnit();
postExecutionUnitOperations();
decrRefCount(keyobj);
keys_freed++;
+2
View File
@@ -54,10 +54,12 @@
int activeExpireCycleTryExpire(redisDb *db, dictEntry *de, long long now) {
long long t = dictGetSignedIntegerVal(de);
if (now > t) {
enterExecutionUnit(1, 0);
sds key = dictGetKey(de);
robj *keyobj = createStringObject(key,sdslen(key));
deleteExpiredKeyAndPropagate(db,keyobj);
decrRefCount(keyobj);
exitExecutionUnit();
return 1;
} else {
return 0;
+5
View File
@@ -2,6 +2,7 @@
#include "bio.h"
#include "atomicvar.h"
#include "functions.h"
#include "cluster.h"
static redisAtomic size_t lazyfree_objects = 0;
static redisAtomic size_t lazyfreed_objects = 0;
@@ -177,6 +178,10 @@ void emptyDbAsync(redisDb *db) {
dict *oldht1 = db->dict, *oldht2 = db->expires;
db->dict = dictCreate(&dbDictType);
db->expires = dictCreate(&dbExpiresDictType);
if (server.cluster_enabled) {
slotToKeyDestroy(db);
slotToKeyInit(db);
}
atomicIncr(lazyfree_objects,dictSize(oldht1));
bioCreateLazyFreeJob(lazyfreeFreeDatabase,2,oldht1,oldht2);
}
+20 -22
View File
@@ -7706,15 +7706,15 @@ RedisModuleBlockedClient *moduleBlockClient(RedisModuleCtx *ctx, RedisModuleCmdF
bc->background_timer = 0;
bc->background_duration = 0;
c->bstate.timeout = 0;
mstime_t timeout = 0;
if (timeout_ms) {
mstime_t now = mstime();
if (timeout_ms > LLONG_MAX - now) {
if (timeout_ms > LLONG_MAX - now) {
c->bstate.module_blocked_handle = NULL;
addReplyError(c, "timeout is out of range"); /* 'timeout_ms+now' would overflow */
return bc;
}
c->bstate.timeout = timeout_ms + now;
timeout = timeout_ms + now;
}
if (islua || ismulti) {
@@ -7730,8 +7730,9 @@ RedisModuleBlockedClient *moduleBlockClient(RedisModuleCtx *ctx, RedisModuleCmdF
addReplyError(c, "Clients undergoing module based authentication can only be blocked on auth");
} else {
if (keys) {
blockForKeys(c,BLOCKED_MODULE,keys,numkeys,c->bstate.timeout,flags&REDISMODULE_BLOCK_UNBLOCK_DELETED);
blockForKeys(c,BLOCKED_MODULE,keys,numkeys,timeout,flags&REDISMODULE_BLOCK_UNBLOCK_DELETED);
} else {
c->bstate.timeout = timeout;
blockClient(c,BLOCKED_MODULE);
}
}
@@ -12115,6 +12116,19 @@ int parseLoadexArguments(RedisModuleString ***module_argv, int *module_argc) {
return REDISMODULE_OK;
}
/* Unregister module-related things, called when moduleLoad fails or moduleUnload. */
void moduleUnregisterCleanup(RedisModule *module) {
moduleFreeAuthenticatedClients(module);
moduleUnregisterCommands(module);
moduleUnsubscribeNotifications(module);
moduleUnregisterSharedAPI(module);
moduleUnregisterUsedAPI(module);
moduleUnregisterFilters(module);
moduleUnsubscribeAllServerEvents(module);
moduleRemoveConfigs(module);
moduleUnregisterAuthCBs(module);
}
/* Load a module and initialize it. On success C_OK is returned, otherwise
* C_ERR is returned. */
int moduleLoad(const char *path, void **module_argv, int module_argc, int is_loadex) {
@@ -12149,11 +12163,7 @@ int moduleLoad(const char *path, void **module_argv, int module_argc, int is_loa
serverLog(LL_WARNING,
"Module %s initialization failed. Module not loaded",path);
if (ctx.module) {
moduleUnregisterCommands(ctx.module);
moduleUnregisterSharedAPI(ctx.module);
moduleUnregisterUsedAPI(ctx.module);
moduleRemoveConfigs(ctx.module);
moduleUnregisterAuthCBs(ctx.module);
moduleUnregisterCleanup(ctx.module);
moduleFreeModuleStructure(ctx.module);
}
moduleFreeContext(&ctx);
@@ -12194,8 +12204,6 @@ int moduleLoad(const char *path, void **module_argv, int module_argc, int is_loa
}
if (post_load_err) {
/* Unregister module auth callbacks (if any exist) that this Module registered onload. */
moduleUnregisterAuthCBs(ctx.module);
moduleUnload(ctx.module->name, NULL);
moduleFreeContext(&ctx);
return C_ERR;
@@ -12253,17 +12261,7 @@ int moduleUnload(sds name, const char **errmsg) {
}
}
moduleFreeAuthenticatedClients(module);
moduleUnregisterCommands(module);
moduleUnregisterSharedAPI(module);
moduleUnregisterUsedAPI(module);
moduleUnregisterFilters(module);
moduleUnregisterAuthCBs(module);
moduleRemoveConfigs(module);
/* Remove any notification subscribers this module might have */
moduleUnsubscribeNotifications(module);
moduleUnsubscribeAllServerEvents(module);
moduleUnregisterCleanup(module);
/* Unload the dynamic library. */
if (dlclose(module->handle) == -1) {
+64 -18
View File
@@ -104,6 +104,9 @@ quicklistBookmark *_quicklistBookmarkFindByName(quicklist *ql, const char *name)
quicklistBookmark *_quicklistBookmarkFindByNode(quicklist *ql, quicklistNode *node);
void _quicklistBookmarkDelete(quicklist *ql, quicklistBookmark *bm);
quicklistNode *_quicklistSplitNode(quicklistNode *node, int offset, int after);
quicklistNode *_quicklistMergeNodes(quicklist *quicklist, quicklistNode *center);
/* Simple way to give quicklistEntry structs default values with one call. */
#define initEntry(e) \
do { \
@@ -378,6 +381,15 @@ REDIS_STATIC void __quicklistCompress(const quicklist *quicklist,
quicklistCompressNode(reverse);
}
/* This macro is used to compress a node.
*
* If the 'recompress' flag of the node is true, we compress it directly without
* checking whether it is within the range of compress depth.
* However, it's important to ensure that the 'recompress' flag of head and tail
* is always false, as we always assume that head and tail are not compressed.
*
* If the 'recompress' flag of the node is false, we check whether the node is
* within the range of compress depth before compressing it. */
#define quicklistCompress(_ql, _node) \
do { \
if ((_node)->recompress) \
@@ -529,19 +541,25 @@ REDIS_STATIC int _quicklistNodeAllowMerge(const quicklistNode *a,
(node)->sz = lpBytes((node)->entry); \
} while (0)
static quicklistNode* __quicklistCreatePlainNode(void *value, size_t sz) {
static quicklistNode* __quicklistCreateNode(int container, void *value, size_t sz) {
quicklistNode *new_node = quicklistCreateNode();
new_node->entry = zmalloc(sz);
new_node->container = QUICKLIST_NODE_CONTAINER_PLAIN;
memcpy(new_node->entry, value, sz);
new_node->container = container;
if (container == QUICKLIST_NODE_CONTAINER_PLAIN) {
new_node->entry = zmalloc(sz);
memcpy(new_node->entry, value, sz);
} else {
new_node->entry = lpPrepend(lpNew(0), value, sz);
}
new_node->sz = sz;
new_node->count++;
return new_node;
}
static void __quicklistInsertPlainNode(quicklist *quicklist, quicklistNode *old_node,
void *value, size_t sz, int after) {
__quicklistInsertNode(quicklist, old_node, __quicklistCreatePlainNode(value, sz), after);
void *value, size_t sz, int after)
{
quicklistNode *new_node = __quicklistCreateNode(QUICKLIST_NODE_CONTAINER_PLAIN, value, sz);
__quicklistInsertNode(quicklist, old_node, new_node, after);
quicklist->count++;
}
@@ -741,9 +759,13 @@ void quicklistReplaceEntry(quicklistIter *iter, quicklistEntry *entry,
void *data, size_t sz)
{
quicklist* quicklist = iter->quicklist;
quicklistNode *node = entry->node;
unsigned char *newentry;
if (likely(!QL_NODE_IS_PLAIN(entry->node) && !isLargeElement(sz))) {
entry->node->entry = lpReplace(entry->node->entry, &entry->zi, data, sz);
if (likely(!QL_NODE_IS_PLAIN(entry->node) && !isLargeElement(sz) &&
(newentry = lpReplace(entry->node->entry, &entry->zi, data, sz)) != NULL))
{
entry->node->entry = newentry;
quicklistNodeUpdateSz(entry->node);
/* quicklistNext() and quicklistGetIteratorEntryAtIdx() provide an uncompressed node */
quicklistCompress(quicklist, entry->node);
@@ -758,17 +780,37 @@ void quicklistReplaceEntry(quicklistIter *iter, quicklistEntry *entry,
quicklistInsertAfter(iter, entry, data, sz);
__quicklistDelNode(quicklist, entry->node);
}
} else {
entry->node->dont_compress = 1; /* Prevent compression in quicklistInsertAfter() */
quicklistInsertAfter(iter, entry, data, sz);
} else { /* The node is full or data is a large element */
quicklistNode *split_node = NULL, *new_node;
node->dont_compress = 1; /* Prevent compression in __quicklistInsertNode() */
/* If the entry is not at the tail, split the node at the entry's offset. */
if (entry->offset != node->count - 1 && entry->offset != -1)
split_node = _quicklistSplitNode(node, entry->offset, 1);
/* Create a new node and insert it after the original node.
* If the original node was split, insert the split node after the new node. */
new_node = __quicklistCreateNode(isLargeElement(sz) ?
QUICKLIST_NODE_CONTAINER_PLAIN : QUICKLIST_NODE_CONTAINER_PACKED, data, sz);
__quicklistInsertNode(quicklist, node, new_node, 1);
if (split_node) __quicklistInsertNode(quicklist, new_node, split_node, 1);
quicklist->count++;
/* Delete the replaced element. */
if (entry->node->count == 1) {
__quicklistDelNode(quicklist, entry->node);
} else {
unsigned char *p = lpSeek(entry->node->entry, -1);
quicklistDelIndex(quicklist, entry->node, &p);
entry->node->dont_compress = 0; /* Re-enable compression */
quicklistCompress(quicklist, entry->node);
quicklistCompress(quicklist, entry->node->next);
new_node = _quicklistMergeNodes(quicklist, new_node);
/* We can't know if the current node and its sibling nodes are correctly compressed,
* and we don't know if they are within the range of compress depth, so we need to
* use quicklistCompress() for compression, which checks if node is within compress
* depth before compressing. */
quicklistCompress(quicklist, new_node);
quicklistCompress(quicklist, new_node->prev);
if (new_node->next) quicklistCompress(quicklist, new_node->next);
}
}
@@ -826,6 +868,8 @@ REDIS_STATIC quicklistNode *_quicklistListpackMerge(quicklist *quicklist,
}
keep->count = lpLength(keep->entry);
quicklistNodeUpdateSz(keep);
keep->recompress = 0; /* Prevent 'keep' from being recompressed if
* it becomes head or tail after merging. */
nokeep->count = 0;
__quicklistDelNode(quicklist, nokeep);
@@ -844,9 +888,10 @@ REDIS_STATIC quicklistNode *_quicklistListpackMerge(quicklist *quicklist,
* - (center->next, center->next->next)
* - (center->prev, center)
* - (center, center->next)
*
* Returns the new 'center' after merging.
*/
REDIS_STATIC void _quicklistMergeNodes(quicklist *quicklist,
quicklistNode *center) {
REDIS_STATIC quicklistNode *_quicklistMergeNodes(quicklist *quicklist, quicklistNode *center) {
int fill = quicklist->fill;
quicklistNode *prev, *prev_prev, *next, *next_next, *target;
prev = prev_prev = next = next_next = target = NULL;
@@ -886,8 +931,9 @@ REDIS_STATIC void _quicklistMergeNodes(quicklist *quicklist,
/* Use result of center merge (or original) to merge with next node. */
if (_quicklistNodeAllowMerge(target, target->next, fill)) {
_quicklistListpackMerge(quicklist, target, target->next);
target = _quicklistListpackMerge(quicklist, target, target->next);
}
return target;
}
/* Split 'node' into two parts, parameterized by 'offset' and 'after'.
@@ -1002,7 +1048,7 @@ REDIS_STATIC void _quicklistInsert(quicklistIter *iter, quicklistEntry *entry,
} else {
quicklistDecompressNodeForUse(node);
new_node = _quicklistSplitNode(node, entry->offset, after);
quicklistNode *entry_node = __quicklistCreatePlainNode(value, sz);
quicklistNode *entry_node = __quicklistCreateNode(QUICKLIST_NODE_CONTAINER_PLAIN, value, sz);
__quicklistInsertNode(quicklist, node, entry_node, after);
__quicklistInsertNode(quicklist, entry_node, new_node, after);
quicklist->count++;
@@ -3224,7 +3270,7 @@ int quicklistTest(int argc, char *argv[], int flags) {
memcpy(s, "helloworld", 10);
memcpy(s + sz - 10, "1234567890", 10);
quicklistNode *node = __quicklistCreatePlainNode(s, sz);
quicklistNode *node = __quicklistCreateNode(QUICKLIST_NODE_CONTAINER_PLAIN, s, sz);
/* Just to avoid triggering the assertion in __quicklistCompressNode(),
* it disables the passing of quicklist head or tail node. */
+1 -4
View File
@@ -81,9 +81,6 @@
#define RDB_TYPE_MODULE_PRE_GA 6 /* Used in 4.0 release candidates */
#define RDB_TYPE_MODULE_2 7 /* Module value with annotations for parsing without
the generating module being loaded. */
/* NOTE: WHEN ADDING NEW RDB TYPE, UPDATE rdbIsObjectType() BELOW */
/* Object types for encoded objects. */
#define RDB_TYPE_HASH_ZIPMAP 9
#define RDB_TYPE_LIST_ZIPLIST 10
#define RDB_TYPE_SET_INTSET 11
@@ -97,7 +94,7 @@
#define RDB_TYPE_STREAM_LISTPACKS_2 19
#define RDB_TYPE_SET_LISTPACK 20
#define RDB_TYPE_STREAM_LISTPACKS_3 21
/* NOTE: WHEN ADDING NEW RDB TYPE, UPDATE rdbIsObjectType() BELOW */
/* NOTE: WHEN ADDING NEW RDB TYPE, UPDATE rdbIsObjectType(), and rdb_type_string[] */
/* Test if a type is an object type. */
#define rdbIsObjectType(t) (((t) >= 0 && (t) <= 7) || ((t) >= 9 && (t) <= 21))
+9 -1
View File
@@ -233,6 +233,7 @@ int checkSingleAof(char *aof_filename, char *aof_filepath, int last_file, int fi
struct redis_stat sb;
if (redis_fstat(fileno(fp),&sb) == -1) {
printf("Cannot stat file: %s, aborting...\n", aof_filename);
fclose(fp);
exit(1);
}
@@ -343,6 +344,7 @@ int fileIsRDB(char *filepath) {
struct redis_stat sb;
if (redis_fstat(fileno(fp), &sb) == -1) {
printf("Cannot stat file: %s\n", filepath);
fclose(fp);
exit(1);
}
@@ -379,6 +381,7 @@ int fileIsManifest(char *filepath) {
struct redis_stat sb;
if (redis_fstat(fileno(fp), &sb) == -1) {
printf("Cannot stat file: %s\n", filepath);
fclose(fp);
exit(1);
}
@@ -395,15 +398,20 @@ int fileIsManifest(char *filepath) {
break;
} else {
printf("Cannot read file: %s\n", filepath);
fclose(fp);
exit(1);
}
}
/* Skip comments lines */
/* We will skip comments lines.
* At present, the manifest format is fixed, see aofInfoFormat.
* We will break directly as long as it encounters other items. */
if (buf[0] == '#') {
continue;
} else if (!memcmp(buf, "file", strlen("file"))) {
is_manifest = 1;
} else {
break;
}
}
+2
View File
@@ -98,7 +98,9 @@ char *rdb_type_string[] = {
"hash-listpack",
"zset-listpack",
"quicklist-v2",
"stream-v2",
"set-listpack",
"stream-v3",
};
/* Show a few stats collected into 'rdbstate' */
+2 -1
View File
@@ -8839,7 +8839,8 @@ static redisReply *sendScan(unsigned long long *it) {
reply = redisCommand(context, "SCAN %llu MATCH %b COUNT %d",
*it, config.pattern, sdslen(config.pattern), config.count);
else
reply = redisCommand(context,"SCAN %llu",*it);
reply = redisCommand(context, "SCAN %llu COUNT %d",
*it, config.count);
/* Handle any error conditions */
if(reply == NULL) {
+1
View File
@@ -2250,6 +2250,7 @@ void readSyncBulkPayload(connection *conn) {
}
zfree(server.repl_transfer_tmpfile);
close(server.repl_transfer_fd);
server.repl_transfer_fd = -1;
server.repl_transfer_tmpfile = NULL;
}
+11 -2
View File
@@ -818,8 +818,17 @@ static robj **luaArgsToRedisArgv(lua_State *lua, int *argc, int *argv_len) {
/* We can't use lua_tolstring() for number -> string conversion
* since Lua uses a format specifier that loses precision. */
lua_Number num = lua_tonumber(lua,j+1);
obj_len = fpconv_dtoa((double)num, dbuf);
dbuf[obj_len] = '\0';
/* Integer printing function is much faster, check if we can safely use it.
* Since lua_Number is not explicitly an integer or a double, we need to make an effort
* to convert it as an integer when that's possible, since the string could later be used
* in a context that doesn't support scientific notation (e.g. 1e9 instead of 100000000). */
long long lvalue;
if (double2ll((double)num, &lvalue))
obj_len = ll2string(dbuf, sizeof(dbuf), lvalue);
else {
obj_len = fpconv_dtoa((double)num, dbuf);
dbuf[obj_len] = '\0';
}
obj_s = dbuf;
} else {
obj_s = (char*)lua_tolstring(lua,j+1,&obj_len);
+15 -13
View File
@@ -349,20 +349,22 @@ sds sdsResize(sds s, size_t size, int would_regrow) {
* type. */
int use_realloc = (oldtype==type || (type < oldtype && type > SDS_TYPE_8));
size_t newlen = use_realloc ? oldhdrlen+size+1 : hdrlen+size+1;
int alloc_already_optimal = 0;
#if defined(USE_JEMALLOC)
/* je_nallocx returns the expected allocation size for the newlen.
* We aim to avoid calling realloc() when using Jemalloc if there is no
* change in the allocation size, as it incurs a cost even if the
* allocation size stays the same. */
alloc_already_optimal = (je_nallocx(newlen, 0) == zmalloc_size(sh));
#endif
if (use_realloc && !alloc_already_optimal) {
newsh = s_realloc(sh, newlen);
if (newsh == NULL) return NULL;
s = (char*)newsh+oldhdrlen;
} else if (!alloc_already_optimal) {
if (use_realloc) {
int alloc_already_optimal = 0;
#if defined(USE_JEMALLOC)
/* je_nallocx returns the expected allocation size for the newlen.
* We aim to avoid calling realloc() when using Jemalloc if there is no
* change in the allocation size, as it incurs a cost even if the
* allocation size stays the same. */
alloc_already_optimal = (je_nallocx(newlen, 0) == zmalloc_size(sh));
#endif
if (!alloc_already_optimal) {
newsh = s_realloc(sh, newlen);
if (newsh == NULL) return NULL;
s = (char*)newsh+oldhdrlen;
}
} else {
newsh = s_malloc(newlen);
if (newsh == NULL) return NULL;
memcpy((char*)newsh+hdrlen, s, len);
+9 -1
View File
@@ -3512,12 +3512,20 @@ void call(client *c, int flags) {
* re-processing and unblock the client.*/
c->flags |= CLIENT_EXECUTING_COMMAND;
/* Setting the CLIENT_REPROCESSING_COMMAND flag so that during the actual
* processing of the command proc, the client is aware that it is being
* re-processed. */
if (reprocessing_command) c->flags |= CLIENT_REPROCESSING_COMMAND;
monotime monotonic_start = 0;
if (monotonicGetType() == MONOTONIC_CLOCK_HW)
monotonic_start = getMonotonicUs();
c->cmd->proc(c);
/* Clear the CLIENT_REPROCESSING_COMMAND flag after the proc is executed. */
if (reprocessing_command) c->flags &= ~CLIENT_REPROCESSING_COMMAND;
exitExecutionUnit();
/* In case client is blocked after trying to execute the command,
@@ -3575,7 +3583,7 @@ void call(client *c, int flags) {
/* Send the command to clients in MONITOR mode if applicable,
* since some administrative commands are considered too dangerous to be shown.
* Other exceptions is a client which is unblocked and retring to process the command
* Other exceptions is a client which is unblocked and retrying to process the command
* or we are currently in the process of loading AOF. */
if (update_command_stats && !reprocessing_command &&
!(c->cmd->flags & (CMD_SKIP_MONITOR|CMD_ADMIN))) {
+1
View File
@@ -400,6 +400,7 @@ extern int configOOMScoreAdjValuesDefaults[CONFIG_OOM_COUNT];
auth had been authenticated from the Module. */
#define CLIENT_MODULE_PREVENT_AOF_PROP (1ULL<<48) /* Module client do not want to propagate to AOF */
#define CLIENT_MODULE_PREVENT_REPL_PROP (1ULL<<49) /* Module client do not want to propagate to replica */
#define CLIENT_REPROCESSING_COMMAND (1ULL<<50) /* The client is re-processing the command. */
/* Client block type (btype field in client structure)
* if CLIENT_BLOCKED flag is set. */
+3 -2
View File
@@ -1172,7 +1172,8 @@ unsigned long zsetLength(const robj *zobj) {
* and the value len hint indicates the approximate individual size of the added elements,
* they are used to determine the initial representation.
*
* If the hints are not known, and underestimation or 0 is suitable. */
* If the hints are not known, and underestimation or 0 is suitable.
* We should never pass a negative value because it will convert to a very large unsigned number. */
robj *zsetTypeCreate(size_t size_hint, size_t val_len_hint) {
if (size_hint <= server.zset_max_listpack_entries &&
val_len_hint <= server.zset_max_listpack_value)
@@ -3001,7 +3002,7 @@ static void zrangeResultFinalizeClient(zrange_result_handler *handler,
/* Result handler methods for storing the ZRANGESTORE to a zset. */
static void zrangeResultBeginStore(zrange_result_handler *handler, long length)
{
handler->dstobj = zsetTypeCreate(length, 0);
handler->dstobj = zsetTypeCreate(length >= 0 ? length : 0, 0);
}
static void zrangeResultEmitCBufferForStore(zrange_result_handler *handler,
+6 -3
View File
@@ -54,8 +54,11 @@
/* Glob-style pattern matching. */
static int stringmatchlen_impl(const char *pattern, int patternLen,
const char *string, int stringLen, int nocase, int *skipLongerMatches)
const char *string, int stringLen, int nocase, int *skipLongerMatches, int nesting)
{
/* Protection against abusive patterns. */
if (nesting > 1000) return 0;
while(patternLen && stringLen) {
switch(pattern[0]) {
case '*':
@@ -67,7 +70,7 @@ static int stringmatchlen_impl(const char *pattern, int patternLen,
return 1; /* match */
while(stringLen) {
if (stringmatchlen_impl(pattern+1, patternLen-1,
string, stringLen, nocase, skipLongerMatches))
string, stringLen, nocase, skipLongerMatches, nesting+1))
return 1; /* match */
if (*skipLongerMatches)
return 0; /* no match */
@@ -189,7 +192,7 @@ static int stringmatchlen_impl(const char *pattern, int patternLen,
int stringmatchlen(const char *pattern, int patternLen,
const char *string, int stringLen, int nocase) {
int skipLongerMatches = 0;
return stringmatchlen_impl(pattern,patternLen,string,stringLen,nocase,&skipLongerMatches);
return stringmatchlen_impl(pattern,patternLen,string,stringLen,nocase,&skipLongerMatches,0);
}
int stringmatch(const char *pattern, const char *string, int nocase) {
+2 -2
View File
@@ -1,2 +1,2 @@
#define REDIS_VERSION "7.2.2"
#define REDIS_VERSION_NUM 0x00070202
#define REDIS_VERSION "7.2.6"
#define REDIS_VERSION_NUM 0x00070206
+25
View File
@@ -529,6 +529,18 @@ tags {"aof external:skip"} {
assert_match "*Start checking Old-Style AOF*is valid*" $result
}
test {Test redis-check-aof for old style resp AOF - has data in the same format as manifest} {
create_aof $aof_dirpath $aof_file {
append_to_aof [formatCommand set file file]
append_to_aof [formatCommand set "file appendonly.aof.2.base.rdb seq 2 type b" "file appendonly.aof.2.base.rdb seq 2 type b"]
}
catch {
exec src/redis-check-aof $aof_file
} result
assert_match "*Start checking Old-Style AOF*is valid*" $result
}
test {Test redis-check-aof for old style rdb-preamble AOF} {
catch {
exec src/redis-check-aof tests/assets/rdb-preamble.aof
@@ -577,6 +589,19 @@ tags {"aof external:skip"} {
assert_match "*Start checking Multi Part AOF*Start to check BASE AOF (RDB format)*DB preamble is OK, proceeding with AOF tail*BASE AOF*is valid*Start to check INCR files*INCR AOF*is valid*All AOF files and manifest are valid*" $result
}
test {Test redis-check-aof for Multi Part AOF contains a format error} {
create_aof_manifest $aof_dirpath $aof_manifest_file {
append_to_manifest "file appendonly.aof.1.base.aof seq 1 type b\n"
append_to_manifest "file appendonly.aof.1.incr.aof seq 1 type i\n"
append_to_manifest "!!!\n"
}
catch {
exec src/redis-check-aof $aof_manifest_file
} result
assert_match "*Invalid AOF manifest file format*" $result
}
test {Test redis-check-aof only truncates the last file for Multi Part AOF in fix mode} {
create_aof $aof_dirpath $aof_base_file {
append_to_aof [formatCommand set foo hello]
+14 -3
View File
@@ -1,6 +1,7 @@
#include "redismodule.h"
#include <string.h>
#include <strings.h>
static RedisModuleString *log_key_name;
@@ -92,7 +93,7 @@ int CommandFilter_LogCommand(RedisModuleCtx *ctx, RedisModuleString **argv, int
return REDISMODULE_OK;
}
int CommandFilter_UnfilteredClientdId(RedisModuleCtx *ctx, RedisModuleString **argv, int argc)
int CommandFilter_UnfilteredClientId(RedisModuleCtx *ctx, RedisModuleString **argv, int argc)
{
if (argc < 2)
return RedisModule_WrongArity(ctx);
@@ -192,7 +193,7 @@ int RedisModule_OnLoad(RedisModuleCtx *ctx, RedisModuleString **argv, int argc)
if (RedisModule_Init(ctx,"commandfilter",1,REDISMODULE_APIVER_1)
== REDISMODULE_ERR) return REDISMODULE_ERR;
if (argc != 2) {
if (argc != 2 && argc != 3) {
RedisModule_Log(ctx, "warning", "Log key name not specified");
return REDISMODULE_ERR;
}
@@ -219,7 +220,7 @@ int RedisModule_OnLoad(RedisModuleCtx *ctx, RedisModuleString **argv, int argc)
return REDISMODULE_ERR;
if (RedisModule_CreateCommand(ctx, unfiltered_clientid_name,
CommandFilter_UnfilteredClientdId, "admin", 1,1,1) == REDISMODULE_ERR)
CommandFilter_UnfilteredClientId, "admin", 1,1,1) == REDISMODULE_ERR)
return REDISMODULE_ERR;
if ((filter = RedisModule_RegisterCommandFilter(ctx, CommandFilter_CommandFilter,
@@ -229,6 +230,16 @@ int RedisModule_OnLoad(RedisModuleCtx *ctx, RedisModuleString **argv, int argc)
if ((filter1 = RedisModule_RegisterCommandFilter(ctx, CommandFilter_BlmoveSwap, 0)) == NULL)
return REDISMODULE_ERR;
if (argc == 3) {
const char *ptr = RedisModule_StringPtrLen(argv[2], NULL);
if (!strcasecmp(ptr, "noload")) {
/* This is a hint that we return ERR at the last moment of OnLoad. */
RedisModule_FreeString(ctx, log_key_name);
if (retained) RedisModule_FreeString(NULL, retained);
return REDISMODULE_ERR;
}
}
return REDISMODULE_OK;
}
+13 -3
View File
@@ -33,6 +33,7 @@
#include "redismodule.h"
#include <stdio.h>
#include <string.h>
#include <strings.h>
#include <assert.h>
/* We need to store events to be able to test and see what we got, and we can't
@@ -407,9 +408,6 @@ int RedisModule_OnLoad(RedisModuleCtx *ctx, RedisModuleString **argv, int argc)
return REDISMODULE_ERR; \
}
REDISMODULE_NOT_USED(argv);
REDISMODULE_NOT_USED(argc);
if (RedisModule_Init(ctx,"testhook",1,REDISMODULE_APIVER_1)
== REDISMODULE_ERR) return REDISMODULE_ERR;
@@ -471,6 +469,18 @@ int RedisModule_OnLoad(RedisModuleCtx *ctx, RedisModuleString **argv, int argc)
if (RedisModule_CreateCommand(ctx,"hooks.pexpireat", cmdKeyExpiry,"",0,0,0) == REDISMODULE_ERR)
return REDISMODULE_ERR;
if (argc == 1) {
const char *ptr = RedisModule_StringPtrLen(argv[0], NULL);
if (!strcasecmp(ptr, "noload")) {
/* This is a hint that we return ERR at the last moment of OnLoad. */
RedisModule_FreeDict(ctx, event_log);
RedisModule_FreeDict(ctx, removed_event_log);
RedisModule_FreeDict(ctx, removed_subevent_type);
RedisModule_FreeDict(ctx, removed_expiry_log);
return REDISMODULE_ERR;
}
}
return REDISMODULE_OK;
}
+11 -3
View File
@@ -36,6 +36,7 @@
#include "redismodule.h"
#include <stdio.h>
#include <string.h>
#include <strings.h>
#include <unistd.h>
ustime_t cached_time = 0;
@@ -318,9 +319,6 @@ static int cmdGetDels(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) {
/* This function must be present on each Redis module. It is used in order to
* register the commands into the Redis server. */
int RedisModule_OnLoad(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) {
REDISMODULE_NOT_USED(argv);
REDISMODULE_NOT_USED(argc);
if (RedisModule_Init(ctx,"testkeyspace",1,REDISMODULE_APIVER_1) == REDISMODULE_ERR){
return REDISMODULE_ERR;
}
@@ -405,6 +403,16 @@ int RedisModule_OnLoad(RedisModuleCtx *ctx, RedisModuleString **argv, int argc)
return REDISMODULE_ERR;
}
if (argc == 1) {
const char *ptr = RedisModule_StringPtrLen(argv[0], NULL);
if (!strcasecmp(ptr, "noload")) {
/* This is a hint that we return ERR at the last moment of OnLoad. */
RedisModule_FreeDict(ctx, loaded_event_log);
RedisModule_FreeDict(ctx, module_event_log);
return REDISMODULE_ERR;
}
}
return REDISMODULE_OK;
}
+67
View File
@@ -90,6 +90,10 @@ static int KeySpace_NotificationEvicted(RedisModuleCtx *ctx, int type, const cha
return REDISMODULE_OK; /* do not count the evicted key */
}
if (strncmp(key_str, "before_evicted", 14) == 0) {
return REDISMODULE_OK; /* do not count the before_evicted key */
}
RedisModuleString *new_key = RedisModule_CreateString(NULL, "evicted", 7);
RedisModule_AddPostNotificationJob(ctx, KeySpace_PostNotificationString, new_key, KeySpace_PostNotificationStringFreePD);
return REDISMODULE_OK;
@@ -186,6 +190,55 @@ static int KeySpace_PostNotificationsAsyncSet(RedisModuleCtx *ctx, RedisModuleSt
return REDISMODULE_OK;
}
typedef struct KeySpace_EventPostNotificationCtx {
RedisModuleString *triggered_on;
RedisModuleString *new_key;
} KeySpace_EventPostNotificationCtx;
static void KeySpace_ServerEventPostNotificationFree(void *pd) {
KeySpace_EventPostNotificationCtx *pn_ctx = pd;
RedisModule_FreeString(NULL, pn_ctx->new_key);
RedisModule_FreeString(NULL, pn_ctx->triggered_on);
RedisModule_Free(pn_ctx);
}
static void KeySpace_ServerEventPostNotification(RedisModuleCtx *ctx, void *pd) {
REDISMODULE_NOT_USED(ctx);
KeySpace_EventPostNotificationCtx *pn_ctx = pd;
RedisModuleCallReply* rep = RedisModule_Call(ctx, "lpush", "!ss", pn_ctx->new_key, pn_ctx->triggered_on);
RedisModule_FreeCallReply(rep);
}
static void KeySpace_ServerEventCallback(RedisModuleCtx *ctx, RedisModuleEvent eid, uint64_t subevent, void *data) {
REDISMODULE_NOT_USED(eid);
REDISMODULE_NOT_USED(data);
if (subevent > 3) {
RedisModule_Log(ctx, "warning", "Got an unexpected subevent '%ld'", subevent);
return;
}
static const char* events[] = {
"before_deleted",
"before_expired",
"before_evicted",
"before_overwritten",
};
const RedisModuleString *key_name = RedisModule_GetKeyNameFromModuleKey(((RedisModuleKeyInfo*)data)->key);
const char *key_str = RedisModule_StringPtrLen(key_name, NULL);
for (int i = 0 ; i < 4 ; ++i) {
const char *event = events[i];
if (strncmp(key_str, event , strlen(event)) == 0) {
return; /* don't log any event on our tracking keys */
}
}
KeySpace_EventPostNotificationCtx *pn_ctx = RedisModule_Alloc(sizeof(*pn_ctx));
pn_ctx->triggered_on = RedisModule_HoldString(NULL, (RedisModuleString*)key_name);
pn_ctx->new_key = RedisModule_CreateString(NULL, events[subevent], strlen(events[subevent]));
RedisModule_AddPostNotificationJob(ctx, KeySpace_ServerEventPostNotification, pn_ctx, KeySpace_ServerEventPostNotificationFree);
}
/* This function must be present on each Redis module. It is used in order to
* register the commands into the Redis server. */
int RedisModule_OnLoad(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) {
@@ -200,6 +253,14 @@ int RedisModule_OnLoad(RedisModuleCtx *ctx, RedisModuleString **argv, int argc)
return REDISMODULE_ERR;
}
int with_key_events = 0;
if (argc >= 1) {
const char *arg = RedisModule_StringPtrLen(argv[0], 0);
if (strcmp(arg, "with_key_events") == 0) {
with_key_events = 1;
}
}
RedisModule_SetModuleOptions(ctx, REDISMODULE_OPTIONS_ALLOW_NESTED_KEYSPACE_NOTIFICATIONS);
if(RedisModule_SubscribeToKeyspaceEvents(ctx, REDISMODULE_NOTIFY_STRING, KeySpace_NotificationString) != REDISMODULE_OK){
@@ -222,6 +283,12 @@ int RedisModule_OnLoad(RedisModuleCtx *ctx, RedisModuleString **argv, int argc)
return REDISMODULE_ERR;
}
if (with_key_events) {
if(RedisModule_SubscribeToServerEvent(ctx, RedisModuleEvent_Key, KeySpace_ServerEventCallback) != REDISMODULE_OK){
return REDISMODULE_ERR;
}
}
if (RedisModule_CreateCommand(ctx, "postnotification.async_set", KeySpace_PostNotificationsAsyncSet,
"write", 0, 0, 0) == REDISMODULE_ERR){
return REDISMODULE_ERR;
+27
View File
@@ -199,3 +199,30 @@ proc are_hostnames_propagated {match_string} {
}
return 1
}
proc wait_node_marked_fail {ref_node_index instance_id_to_check} {
wait_for_condition 1000 50 {
[check_cluster_node_mark fail $ref_node_index $instance_id_to_check]
} else {
fail "Replica node never marked as FAIL ('fail')"
}
}
proc wait_node_marked_pfail {ref_node_index instance_id_to_check} {
wait_for_condition 1000 50 {
[check_cluster_node_mark fail\? $ref_node_index $instance_id_to_check]
} else {
fail "Replica node never marked as PFAIL ('fail?')"
}
}
proc check_cluster_node_mark {flag ref_node_index instance_id_to_check} {
set nodes [get_cluster_nodes $ref_node_index]
foreach n $nodes {
if {[dict get $n id] eq $instance_id_to_check} {
return [cluster_has_flag $n $flag]
}
}
fail "Unable to find instance id in cluster nodes. ID: $instance_id_to_check"
}
+2
View File
@@ -94,6 +94,7 @@ set ::all_tests {
unit/client-eviction
unit/violations
unit/replybufsize
unit/cluster/announced-endpoints
unit/cluster/misc
unit/cluster/cli
unit/cluster/scripting
@@ -103,6 +104,7 @@ set ::all_tests {
unit/cluster/slot-ownership
unit/cluster/links
unit/cluster/cluster-response-tls
unit/cluster/failure-marking
}
# Index to the next test to run in the ::all_tests list.
set ::next_test 0
+13 -5
View File
@@ -1,8 +1,12 @@
start_cluster 2 2 {tags {external:skip cluster}} {
test "Test change cluster-announce-port and cluster-announce-tls-port at runtime" {
set baseport [lindex [R 0 config get port] 1]
set count [expr [llength $::servers] +1 ]
if {$::tls} {
set baseport [lindex [R 0 config get tls-port] 1]
} else {
set baseport [lindex [R 0 config get port] 1]
}
set count [expr [llength $::servers] + 1]
set used_port [find_available_port $baseport $count]
R 0 config set cluster-announce-tls-port $used_port
@@ -17,12 +21,16 @@ start_cluster 2 2 {tags {external:skip cluster}} {
R 0 config set cluster-announce-tls-port 0
R 0 config set cluster-announce-port 0
assert_match "*:$baseport@*" [R 0 CLUSTER NODES]
assert_match "*:$baseport@*" [R 0 CLUSTER NODES]
}
test "Test change cluster-announce-bus-port at runtime" {
set baseport [lindex [R 0 config get port] 1]
set count [expr [llength $::servers] +1 ]
if {$::tls} {
set baseport [lindex [R 0 config get tls-port] 1]
} else {
set baseport [lindex [R 0 config get port] 1]
}
set count [expr [llength $::servers] + 1]
set used_port [find_available_port $baseport $count]
# Verify config set cluster-announce-bus-port
+53
View File
@@ -0,0 +1,53 @@
# Test a single primary can mark replica as `fail`
start_cluster 1 1 {tags {external:skip cluster}} {
test "Verify that single primary marks replica as failed" {
set primary [srv -0 client]
set replica1 [srv -1 client]
set replica1_pid [srv -1 pid]
set replica1_instance_id [dict get [cluster_get_myself 1] id]
assert {[lindex [$primary role] 0] eq {master}}
assert {[lindex [$replica1 role] 0] eq {slave}}
wait_for_sync $replica1
pause_process $replica1_pid
wait_node_marked_fail 0 $replica1_instance_id
}
}
# Test multiple primaries wait for a quorum and then mark a replica as `fail`
start_cluster 2 1 {tags {external:skip cluster}} {
test "Verify that multiple primaries mark replica as failed" {
set primary1 [srv -0 client]
set primary2 [srv -1 client]
set primary2_pid [srv -1 pid]
set replica1 [srv -2 client]
set replica1_pid [srv -2 pid]
set replica1_instance_id [dict get [cluster_get_myself 2] id]
assert {[lindex [$primary1 role] 0] eq {master}}
assert {[lindex [$primary2 role] 0] eq {master}}
assert {[lindex [$replica1 role] 0] eq {slave}}
wait_for_sync $replica1
pause_process $replica1_pid
# Pause other primary to allow time for pfail flag to appear
pause_process $primary2_pid
wait_node_marked_pfail 0 $replica1_instance_id
# Resume other primary and wait for to show replica as failed
resume_process $primary2_pid
wait_node_marked_fail 0 $replica1_instance_id
}
}
+6
View File
@@ -499,4 +499,10 @@ foreach {type large} [array get largevalue] {
r SET aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa 1
r KEYS "a*a*a*a*a*a*a*a*a*a*a*a*a*a*a*a*a*a*a*a*b"
} {}
test {Regression for pattern matching very long nested loops} {
r flushdb
r SET [string repeat "a" 50000] 1
r KEYS [string repeat "*?" 50000]
} {}
}
+53
View File
@@ -577,4 +577,57 @@ start_server {tags {"defrag external:skip"} overrides {appendonly yes auto-aof-r
}
}
}
start_cluster 1 0 {tags {"defrag external:skip"} overrides {appendonly yes auto-aof-rewrite-percentage 0 save ""}} {
if {[string match {*jemalloc*} [s mem_allocator]] && [r debug mallctl arenas.page] <= 8192} {
test "Active defrag stability during async flush or mid stopping in cluster mode" {
# Verify that asynchronously empting db doesn't cause defragmentation crashes, see issue #13205.
r flushdb async
r config set hz 100
r config set activedefrag no
r config set active-defrag-threshold-lower 1
r config set active-defrag-cycle-min 65
r config set active-defrag-cycle-max 75
r config set active-defrag-ignore-bytes 100k
# create big keys with 10k items
set rd [redis_deferring_client]
for {set j 0} {$j < 100000} {incr j} {
$rd set $j a
$rd expire $j 99999
}
for {set j 0} {$j < 100000} {incr j} {
$rd read ; # Discard replies
$rd read ; # Discard replies
}
catch {r config set activedefrag yes} e
if {[r config get activedefrag] eq "activedefrag yes"} {
# It repeatedly enables and disables active defragmentation,
# and checks if it crashes, see issue #13307.
for {set i 0} {$i < 10} {incr i} {
r config set activedefrag no
# Wait for the active defrag to start working (decision once a second).
wait_for_condition 50 100 {
[s active_defrag_running] eq 0
} else {
after 120 ;# serverCron only updates the info once in 100ms
puts [r info memory]
puts [r memory malloc-stats]
fail "defrag didn't stop."
}
# Wait for the active defrag to stop working.
r config set activedefrag yes
wait_for_condition 50 100 {
[s active_defrag_running] ne 0
} else {
fail "defrag not started."
}
}
}
r ping
}
}
}
} ;# run_solo
+27
View File
@@ -314,6 +314,14 @@ start_server {tags {"modules"}} {
r lpush l a
assert_equal [$rd read] {OK}
# Explanation of the first multi exec block:
# {lpop l} - pop the value by our blocking command 'blpop_and_set_multiple_keys'
# {set string_foo 1} - the action of our blocking command 'blpop_and_set_multiple_keys'
# {set string_bar 2} - the action of our blocking command 'blpop_and_set_multiple_keys'
# {incr string_changed{string_foo}} - post notification job that was registered when 'string_foo' changed
# {incr string_changed{string_bar}} - post notification job that was registered when 'string_bar' changed
# {incr string_total} - post notification job that was registered when string_changed{string_foo} changed
# {incr string_total} - post notification job that was registered when string_changed{string_bar} changed
assert_replication_stream $repl {
{select *}
{lpush l a}
@@ -355,6 +363,25 @@ start_server {tags {"modules"}} {
r lpush l a
assert_equal [$rd read] {OK}
# Explanation of the first multi exec block:
# {lpop l} - pop the value by our blocking command 'blpop_and_set_multiple_keys'
# {set string_foo 1} - the action of our blocking command 'blpop_and_set_multiple_keys'
# {set string_bar 2} - the action of our blocking command 'blpop_and_set_multiple_keys'
# {incr string_changed{string_foo}} - post notification job that was registered when 'string_foo' changed
# {incr string_changed{string_bar}} - post notification job that was registered when 'string_bar' changed
# {incr string_total} - post notification job that was registered when string_changed{string_foo} changed
# {incr string_total} - post notification job that was registered when string_changed{string_bar} changed
#
# Explanation of the second multi exec block:
# {lpop l} - pop the value by our blocking command 'blpop_and_set_multiple_keys'
# {del string_foo} - lazy expiration of string_foo when 'blpop_and_set_multiple_keys' tries to write to it.
# {set string_foo 1} - the action of our blocking command 'blpop_and_set_multiple_keys'
# {set string_bar 2} - the action of our blocking command 'blpop_and_set_multiple_keys'
# {incr expired} - the post notification job, registered after string_foo got expired
# {incr string_changed{string_foo}} - post notification job triggered when we set string_foo
# {incr string_changed{string_bar}} - post notification job triggered when we set string_bar
# {incr string_total} - post notification job triggered when we incr 'string_changed{string_foo}'
# {incr string_total} - post notification job triggered when we incr 'string_changed{string_bar}'
assert_replication_stream $repl {
{select *}
{lpush l a}
+12 -2
View File
@@ -95,7 +95,7 @@ start_server {tags {"modules"}} {
test "Unload the module - commandfilter" {
assert_equal {OK} [r module unload commandfilter]
}
}
}
test {RM_CommandFilterArgInsert and script argv caching} {
# coverage for scripts calling commands that expand the argv array
@@ -162,4 +162,14 @@ test {Filtering based on client id} {
$rr close
}
}
}
start_server {} {
test {OnLoad failure will handle un-registration} {
catch {r module load $testmodule log-key 0 noload}
r set mykey @log
assert_equal [r lrange log-key 0 -1] {}
r rpush mylist elem1 @delme elem2
assert_equal [r lrange mylist 0 -1] {elem1 @delme elem2}
}
}
+8
View File
@@ -310,4 +310,12 @@ tags "modules" {
assert_equal [string match {*module-event-shutdown*} [exec tail -5 < $replica_stdout]] 1
}
}
start_server {} {
test {OnLoad failure will handle un-registration} {
catch {r module load $testmodule noload}
r flushall
r ping
}
}
}
+13
View File
@@ -102,4 +102,17 @@ tags "modules" {
assert_equal {OK} [r set x 1 EX 1]
}
}
start_server {} {
test {OnLoad failure will handle un-registration} {
catch {r module load $testmodule noload}
r set x 1
r hset y f v
r lpush z 1 2 3
r sadd p 1 2 3
r zadd t 1 f1 2 f2
r xadd s * f v
r ping
}
}
}
+21 -7
View File
@@ -1,7 +1,8 @@
set testmodule [file normalize tests/modules/postnotifications.so]
tags "modules" {
start_server [list overrides [list loadmodule "$testmodule"]] {
start_server {} {
r module load $testmodule with_key_events
test {Test write on post notification callback} {
set repl [attach_to_replication_stream]
@@ -9,11 +10,12 @@ tags "modules" {
r set string_x 1
assert_equal {1} [r get string_changed{string_x}]
assert_equal {1} [r get string_total]
r set string_x 2
assert_equal {2} [r get string_changed{string_x}]
assert_equal {2} [r get string_total]
# the {lpush before_overwritten string_x} is a post notification job registered when 'string_x' was overwritten
assert_replication_stream $repl {
{multi}
{select *}
@@ -23,6 +25,7 @@ tags "modules" {
{exec}
{multi}
{set string_x 2}
{lpush before_overwritten string_x}
{incr string_changed{string_x}}
{incr string_total}
{exec}
@@ -37,7 +40,7 @@ tags "modules" {
assert_equal {OK} [r postnotification.async_set]
assert_equal {1} [r get string_changed{string_x}]
assert_equal {1} [r get string_total]
assert_replication_stream $repl {
{multi}
{select *}
@@ -63,12 +66,14 @@ tags "modules" {
fail "Failed waiting for x to expired"
}
# the {lpush before_expired x} is a post notification job registered before 'x' got expired
assert_replication_stream $repl {
{select *}
{set x 1}
{pexpireat x *}
{multi}
{del x}
{lpush before_expired x}
{incr expired}
{exec}
}
@@ -85,12 +90,14 @@ tags "modules" {
after 10
assert_equal {} [r get x]
# the {lpush before_expired x} is a post notification job registered before 'x' got expired
assert_replication_stream $repl {
{select *}
{set x 1}
{pexpireat x *}
{multi}
{del x}
{lpush before_expired x}
{incr expired}
{exec}
}
@@ -108,6 +115,7 @@ tags "modules" {
after 10
assert_equal {OK} [r set read_x 1]
# the {lpush before_expired x} is a post notification job registered before 'x' got expired
assert_replication_stream $repl {
{select *}
{set x 1}
@@ -115,6 +123,7 @@ tags "modules" {
{multi}
{set read_x 1}
{del x}
{lpush before_expired x}
{incr expired}
{exec}
}
@@ -143,16 +152,18 @@ tags "modules" {
r flushall
set repl [attach_to_replication_stream]
r set x 1
r config set maxmemory-policy allkeys-random
r config set maxmemory-policy allkeys-random
r config set maxmemory 1
assert_error {OOM *} {r set y 1}
# the {lpush before_evicted x} is a post notification job registered before 'x' got evicted
assert_replication_stream $repl {
{select *}
{set x 1}
{multi}
{del x}
{lpush before_evicted x}
{incr evicted}
{exec}
}
@@ -164,7 +175,8 @@ tags "modules" {
set testmodule2 [file normalize tests/modules/keyspace_events.so]
tags "modules" {
start_server [list overrides [list loadmodule "$testmodule"]] {
start_server {} {
r module load $testmodule with_key_events
r module load $testmodule2
test {Test write on post notification callback} {
set repl [attach_to_replication_stream]
@@ -172,7 +184,7 @@ tags "modules" {
r set string_x 1
assert_equal {1} [r get string_changed{string_x}]
assert_equal {1} [r get string_total]
r set string_x 2
assert_equal {2} [r get string_changed{string_x}]
assert_equal {2} [r get string_total]
@@ -181,6 +193,7 @@ tags "modules" {
assert_equal {1} [r get string_changed{string1_x}]
assert_equal {3} [r get string_total]
# the {lpush before_overwritten string_x} is a post notification job registered before 'string_x' got overwritten
assert_replication_stream $repl {
{multi}
{select *}
@@ -190,6 +203,7 @@ tags "modules" {
{exec}
{multi}
{set string_x 2}
{lpush before_overwritten string_x}
{incr string_changed{string_x}}
{incr string_total}
{exec}
@@ -202,4 +216,4 @@ tags "modules" {
close_replication_stream $repl
}
}
}
}
+13 -2
View File
@@ -1,5 +1,4 @@
set system_name [string tolower [exec uname -s]]
set user_id [exec id -u]
if {$system_name eq {linux}} {
start_server {tags {"oom-score-adj external:skip"}} {
@@ -56,8 +55,15 @@ if {$system_name eq {linux}} {
}
}
# Determine whether the current user is unprivileged
set original_value [exec cat /proc/self/oom_score_adj]
catch {
set fd [open "/proc/self/oom_score_adj" "w"]
puts $fd -1000
close $fd
} e
# Failed oom-score-adj tests can only run unprivileged
if {$user_id != 0} {
if {[string match "*permission denied*" $e]} {
test {CONFIG SET oom-score-adj handles configuration failures} {
# Bad config
r config set oom-score-adj no
@@ -81,6 +87,11 @@ if {$system_name eq {linux}} {
# Make sure previous values remain
assert {[r config get oom-score-adj-values] == {oom-score-adj-values {0 100 100}}}
}
} else {
# Restore the original oom_score_adj value
set fd [open "/proc/self/oom_score_adj" "w"]
puts $fd $original_value
close $fd
}
test {CONFIG SET oom-score-adj-values doesn't touch proc when disabled} {
+14
View File
@@ -146,6 +146,14 @@ start_server {tags {"scripting"}} {
} 1 x
} {number 1}
test {EVAL - Lua number -> Redis integer conversion} {
r del hash
run_script {
local foo = redis.pcall('hincrby','hash','field',200000000)
return {type(foo),foo}
} 0
} {number 200000000}
test {EVAL - Redis bulk -> Lua type conversion} {
r set mykey myval
run_script {
@@ -605,6 +613,12 @@ start_server {tags {"scripting"}} {
set e
} {ERR *Attempt to modify a readonly table*}
test {lua bit.tohex bug} {
set res [run_script {return bit.tohex(65535, -2147483648)} 0]
r ping
set res
} {0000FFFF}
test {Test an example script DECR_IF_GT} {
set decr_if_gt {
local current
+88 -1
View File
@@ -220,6 +220,7 @@ start_server [list overrides [list save ""] ] {
# checking LSET in case ziplist needs to be split
test {Test LSET with packed is split in the middle} {
set original_config [config_get_set list-max-listpack-size 4]
r flushdb
r debug quicklist-packed-threshold 5b
r RPUSH lst "aa"
@@ -227,6 +228,7 @@ start_server [list overrides [list save ""] ] {
r RPUSH lst "cc"
r RPUSH lst "dd"
r RPUSH lst "ee"
assert_encoding quicklist lst
r lset lst 2 [string repeat e 10]
assert_equal [r lpop lst] "aa"
assert_equal [r lpop lst] "bb"
@@ -234,6 +236,7 @@ start_server [list overrides [list save ""] ] {
assert_equal [r lpop lst] "dd"
assert_equal [r lpop lst] "ee"
r debug quicklist-packed-threshold 0
r config set list-max-listpack-size $original_config
} {OK} {needs:debug}
@@ -381,7 +384,63 @@ if {[lindex [r config get proto-max-bulk-len] 1] == 10000000000} {
assert_equal [read_big_bulk {r rpop lst}] $str_length
} {} {large-memory}
test {Test LMOVE on plain nodes over 4GB} {
test {Test LSET on plain nodes with large elements under packed_threshold over 4GB} {
r flushdb
r rpush lst a b c d e
for {set i 0} {$i < 5} {incr i} {
r write "*4\r\n\$4\r\nlset\r\n\$3\r\nlst\r\n\$1\r\n$i\r\n"
write_big_bulk 1000000000
}
r ping
} {PONG} {large-memory}
test {Test LSET splits a quicklist node, and then merge} {
# Test when a quicklist node can't be inserted and is split, the split
# node merges with the node before it and the `before` node is kept.
r flushdb
r rpush lst [string repeat "x" 4096]
r lpush lst a b c d e f g
r lpush lst [string repeat "y" 4096]
# now: [y...] [g f e d c b a x...]
# (node0) (node1)
# Keep inserting elements into node1 until node1 is split into two
# nodes([g] [...]), eventually node0 will merge with the [g] node.
# Since node0 is larger, after the merge node0 will be kept and
# the [g] node will be deleted.
for {set i 7} {$i >= 3} {incr i -1} {
r write "*4\r\n\$4\r\nlset\r\n\$3\r\nlst\r\n\$1\r\n$i\r\n"
write_big_bulk 1000000000
}
assert_equal "g" [r lindex lst 1]
r ping
} {PONG} {large-memory}
test {Test LSET splits a LZF compressed quicklist node, and then merge} {
# Test when a LZF compressed quicklist node can't be inserted and is split,
# the split node merges with the node before it and the split node is kept.
r flushdb
r config set list-compress-depth 1
r lpush lst [string repeat "x" 2000]
r rpush lst [string repeat "y" 7000]
r rpush lst a b c d e f g
r rpush lst [string repeat "z" 8000]
r lset lst 0 h
# now: [h] [y... a b c d e f g] [z...]
# node0 node1(LZF)
# Keep inserting elements into node1 until node1 is split into two
# nodes([y...] [...]), eventually node0 will merge with the [y...] node.
# Since [y...] node is larger, after the merge node0 will be deleted and
# the [y...] node will be kept.
for {set i 7} {$i >= 3} {incr i -1} {
r write "*4\r\n\$4\r\nlset\r\n\$3\r\nlst\r\n\$1\r\n$i\r\n"
write_big_bulk 1000000000
}
assert_equal "h" [r lindex lst 0]
r config set list-compress-depth 0
r ping
} {PONG} {large-memory}
test {Test LMOVE on plain nodes over 4GB} {
r flushdb
r RPUSH lst2{t} "aa"
r RPUSH lst2{t} "bb"
@@ -1186,6 +1245,34 @@ foreach {pop} {BLPOP BLMPOP_LEFT} {
r select 9
} {OK} {singledb:skip needs:debug}
test {BLPOP unblock but the key is expired and then block again - reprocessing command} {
r flushall
r debug set-active-expire 0
set rd [redis_deferring_client]
set start [clock milliseconds]
$rd blpop mylist 1
wait_for_blocked_clients_count 1
# The exec will try to awake the blocked client, but the key is expired,
# so the client will be blocked again during the command reprocessing.
r multi
r rpush mylist a
r pexpire mylist 100
r debug sleep 0.2
r exec
assert_equal {} [$rd read]
set end [clock milliseconds]
# Before the fix in #13004, this time would have been 1200+ (i.e. more than 1200ms),
# now it should be 1000, but in order to avoid timing issues, we increase the range a bit.
assert_range [expr $end-$start] 1000 1150
r debug set-active-expire 1
$rd close
} {0} {needs:debug}
foreach {pop} {BLPOP BLMPOP_LEFT} {
test "$pop when new key is moved into place" {
set rd [redis_deferring_client]
+29 -1
View File
@@ -475,7 +475,7 @@ start_server {
$rd close
}
test {Blocking XREADGROUP for stream key that has clients blocked on list - avoid endless loop} {
test {Blocking XREADGROUP for stream key that has clients blocked on stream - avoid endless loop} {
r DEL mystream
r XGROUP CREATE mystream mygroup $ MKSTREAM
@@ -498,6 +498,34 @@ start_server {
assert_equal [r ping] {PONG}
}
test {Blocking XREADGROUP for stream key that has clients blocked on stream - reprocessing command} {
r DEL mystream
r XGROUP CREATE mystream mygroup $ MKSTREAM
set rd1 [redis_deferring_client]
set rd2 [redis_deferring_client]
$rd1 xreadgroup GROUP mygroup myuser BLOCK 0 STREAMS mystream >
wait_for_blocked_clients_count 1
set start [clock milliseconds]
$rd2 xreadgroup GROUP mygroup myuser BLOCK 1000 STREAMS mystream >
wait_for_blocked_clients_count 2
# After a while call xadd and let rd2 re-process the command.
after 200
r xadd mystream * field value
assert_equal {} [$rd2 read]
set end [clock milliseconds]
# Before the fix in #13004, this time would have been 1200+ (i.e. more than 1200ms),
# now it should be 1000, but in order to avoid timing issues, we increase the range a bit.
assert_range [expr $end-$start] 1000 1150
$rd1 close
$rd2 close
}
test {XGROUP DESTROY should unblock XREADGROUP with -NOGROUP} {
r config resetstat
r del mystream
+34
View File
@@ -1942,6 +1942,34 @@ start_server {tags {"zset"}} {
}
}
test {BZPOPMIN unblock but the key is expired and then block again - reprocessing command} {
r flushall
r debug set-active-expire 0
set rd [redis_deferring_client]
set start [clock milliseconds]
$rd bzpopmin zset{t} 1
wait_for_blocked_clients_count 1
# The exec will try to awake the blocked client, but the key is expired,
# so the client will be blocked again during the command reprocessing.
r multi
r zadd zset{t} 1 one
r pexpire zset{t} 100
r debug sleep 0.2
r exec
assert_equal {} [$rd read]
set end [clock milliseconds]
# Before the fix in #13004, this time would have been 1200+ (i.e. more than 1200ms),
# now it should be 1000, but in order to avoid timing issues, we increase the range a bit.
assert_range [expr $end-$start] 1000 1150
r debug set-active-expire 1
$rd close
} {0} {needs:debug}
test "BZPOPMIN with same key multiple times should work" {
set rd [redis_deferring_client]
r del z1{t} z2{t}
@@ -2211,12 +2239,18 @@ start_server {tags {"zset"}} {
} {b 2 c 3}
test {ZRANGESTORE BYLEX} {
set res [r zrangestore z3{t} z1{t} \[b \[c BYLEX]
assert_equal $res 2
assert_encoding listpack z3{t}
set res [r zrangestore z2{t} z1{t} \[b \[c BYLEX]
assert_equal $res 2
r zrange z2{t} 0 -1 withscores
} {b 2 c 3}
test {ZRANGESTORE BYSCORE} {
set res [r zrangestore z4{t} z1{t} 1 2 BYSCORE]
assert_equal $res 2
assert_encoding listpack z4{t}
set res [r zrangestore z2{t} z1{t} 1 2 BYSCORE]
assert_equal $res 2
r zrange z2{t} 0 -1 withscores