Make RM_DefragRedisModuleDict API support incremental defragmentation for dict leaf (#13840)

After https://github.com/redis/redis/pull/13816, we make a new API to
defrag RedisModuleDict.
Currently, we only support incremental defragmentation of the dictionary
itself, but the defragmentation of values is still not incremental. If
the values are very large, it could lead to significant blocking.
Therefore, in this PR, we have added incremental defragmentation for the
values.

The main change is to the `RedisModuleDefragDictValueCallback`, we
modified the return value of this callback.
When the callback returns 1, we will save the `seekTo` as the key of the
current unfinished node, and the next time we enter, we will continue
defragmenting this node.
When the return value is 0, we will proceed to the next node.

## Test
Since each dictionary in the global dict originally contained only 10
strings, but now it has been changed to a nested dictionary, each
dictionary now has 10 sub-dictionaries, with each sub-dictionary
containing 10 strings, this has led to a corresponding reduction in the
defragmentation time obtained from other tests.
Therefore, the other tests have been modified to always wait for
defragmentation to be turned off before the test begins, then start it
after creating fragmentation, ensuring that they can always run for a
full defragmentation cycle.

---------

Co-authored-by: ephraimfeldblum <ephraim.feldblum@redis.com>
This commit is contained in:
debing.sun
2025-03-04 17:19:41 +08:00
committed by GitHub
co-authored by ephraimfeldblum
parent 7939ba031d
commit f364dcca2d
4 changed files with 99 additions and 22 deletions
+29 -2
View File
@@ -12,9 +12,22 @@ start_server {tags {"modules"} overrides {{save ""}}} {
if {[r config get activedefrag] eq "activedefrag yes"} {
test {Module defrag: simple key defrag works} {
r config set activedefrag no
wait_for_condition 100 50 {
[s active_defrag_running] eq 0
} else {
fail "Unable to wait for active defrag to stop"
}
r frag.create key1 1 1000 0
after 2000
r config set activedefrag yes
wait_for_condition 100 50 {
[getInfoProperty [r info defragtest_stats] defragtest_defrag_ended] > 0
} else {
fail "Unable to wait for a complete defragmentation cycle to finish"
}
set info [r info defragtest_stats]
assert {[getInfoProperty $info defragtest_datatype_attempts] > 0}
assert_equal 0 [getInfoProperty $info defragtest_datatype_resumes]
@@ -24,6 +37,13 @@ start_server {tags {"modules"} overrides {{save ""}}} {
}
test {Module defrag: late defrag with cursor works} {
r config set activedefrag no
wait_for_condition 100 50 {
[s active_defrag_running] eq 0
} else {
fail "Unable to wait for active defrag to stop"
}
r flushdb
r frag.resetstats
@@ -31,7 +51,13 @@ start_server {tags {"modules"} overrides {{save ""}}} {
# due to maxstep
r frag.create key2 10000 100 1000
after 2000
r config set activedefrag yes
wait_for_condition 100 50 {
[getInfoProperty [r info defragtest_stats] defragtest_defrag_ended] > 0
} else {
fail "Unable to wait for a complete defragmentation cycle to finish"
}
set info [r info defragtest_stats]
assert {[getInfoProperty $info defragtest_datatype_resumes] > 10}
assert_equal 0 [getInfoProperty $info defragtest_datatype_wrong_cursor]
@@ -67,6 +93,7 @@ start_server {tags {"modules"} overrides {{save ""}}} {
assert_morethan [getInfoProperty $info defragtest_defrag_started] 0
assert_morethan [getInfoProperty $info defragtest_defrag_ended] 0
assert_morethan [getInfoProperty $info defragtest_global_dicts_resumes] [getInfoProperty $info defragtest_defrag_ended]
assert_morethan [getInfoProperty $info defragtest_global_subdicts_resumes] [getInfoProperty $info defragtest_defrag_ended]
}
}
}