improve performance for scan command when matching pattern or data type (#12209)

Optimized the performance of the SCAN command in a few ways:
1. Move the key filtering (by MATCH pattern) in the scan callback,
  so as to avoid collecting them for later filtering.
2. Reduce a many memory allocations and copying (use a reference
  to the original sds, instead of creating an robj, an excessive 2 mallocs
  and one string duplication)
3. Compare TYPE filter directly (as integers), instead of inefficient string
  compare per key.
4. fixed a small bug: when scan zset and hash types, maxiterations uses
  a more accurate number to avoid wrong double maxiterations.

Changes **postponed** for a later version (8.0):
1. Prepare to move the TYPE filtering to the scan callback as well. this was
  put on hold since it has side effects that can be considered a breaking
  change, which is that we will not attempt to do lazy expire (delete) a key
  that was filtered by not matching the TYPE (changing it would mean TYPE filter
  starts behaving the same as MATCH filter already does in that respect). 
2. when the specified key TYPE filter is an unknown type, server will reply a error
  immediately instead of doing a full scan that comes back empty handed. 

Benchmark result:
For different scenarios, we obtained about 30% or more performance improvement.

Co-authored-by: Oran Agra <oran@redislabs.com>
This commit is contained in:
judeng
2023-06-27 16:43:46 +03:00
committed by GitHub
co-authored by Oran Agra
parent b2cdf6bcc3
commit 07ed0eafa9
5 changed files with 353 additions and 119 deletions
+42
View File
@@ -89,4 +89,46 @@ start_server {tags {"modules"}} {
$rd read
$rd close
}
test {DataType: check the type name} {
r flushdb
r datatype.set foo 111 bar
assert_type test___dt foo
}
test {SCAN module datatype} {
r flushdb
populate 1000
r datatype.set foo 111 bar
set type [r type foo]
set cur 0
set keys {}
while 1 {
set res [r scan $cur type $type]
set cur [lindex $res 0]
set k [lindex $res 1]
lappend keys {*}$k
if {$cur == 0} break
}
assert_equal 1 [llength $keys]
}
test {SCAN module datatype with case sensitive} {
r flushdb
populate 1000
r datatype.set foo 111 bar
set type "tEsT___dT"
set cur 0
set keys {}
while 1 {
set res [r scan $cur type $type]
set cur [lindex $res 0]
set k [lindex $res 1]
lappend keys {*}$k
if {$cur == 0} break
}
assert_equal 1 [llength $keys]
}
}