From e1789e43687108a41cfa8e77d09138e2a296fb89 Mon Sep 17 00:00:00 2001 From: Moti Cohen Date: Mon, 12 May 2025 10:15:17 +0300 Subject: [PATCH] keyspace - Unify key and value & use dict no_value=1 (#13806) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The idea of packing the key (`sds`), value (`robj`) and optionally TTL into a single struct in memory was mentioned a few times in the past by the community in various flavors. This approach improves memory efficiency, reduces pointer dereferences for faster lookups, and simplifies expiration management by keeping all relevant data in one place. This change goes along with setting keyspace's dict to no_value=1, and saving considerable amount of memory. Two more motivations that well aligned with this unification are: - Prepare the groundwork for replacing EXPIRE scan based implementation and evaluate instead new `ebuckets` data structure that was introduced as part of [Hash Field Expiration feature](https://redis.io/blog/hash-field-expiration-architecture-and-benchmarks/). Using this data structure requires embedding the ExpireMeta structure within each object. - Consider replacing dict with a more space efficient open addressing approach hash table that might rely on keeping a single pointer to object. Before this PR, I POC'ed on a variant of open addressing hash-table and was surprised to find that dict with no_value actually could provide a good balance between performance, memory efficiency, and simplicity. This realization prompted the separation of the unification step from the evaluation of a new hash table to avoid introducing too many changes at once and to evaluate its impact independently before considering replacement of existing hash-table. On an earlier [commit](https://github.com/redis/redis/pull/13683) I extended dict no_value optimization (which saves keeping dictEntry where possible) to be relevant also for objects with even addresses in memory. Combining it with this unification saves a considerable amount of memory for keyspace. # kvobj This PR adopts Valkey’s [packing](https://github.com/valkey-io/valkey/commit/3eb8314be6af0777e69f852b65f933dd9186d30b) layout and logic for key, value, and TTL. However, unlike Valkey implementation, which retained a common `robj` throughout the project, this PR distinguishes between the general-purpose, overused `robj`, and the new `kvobj`, which embeds both the key and value and used by the keyspace. Conceptually, `robj` serves as a base class, while `kvobj` acts as a derived class. Two new flags introduced into redis object, `iskvobj` and `expirable`: ``` struct redisObject { unsigned type:4; unsigned encoding:4; unsigned lru:LRU_BITS; unsigned iskvobj : 1; /* new flag */ unsigned expirable : 1; /* new flag */ unsigned refcount : 30; /* modified: 32bits->30bits */ void *ptr; }; typedef struct redisObject robj; typedef struct redisObject kvobj; ``` When the `iskvobj` flag is set, the object includes also the key and it is appended to the end of the object. If the `expirable` flag is set, an additional 8 bytes are added to the object. If the object is of type string, and the string is rather short, then it will be embedded as well. As a result, all keys in the keyspace are promoted to be of type `kvobj`. This term attempts to align with the existing Redis object, robj, and the kvstore data structure. # EXPIRE Implementation As `kvobj` embeds expiration time as well, looking up expiration times is now an O(1) operation. And the hash-table of EXPIRE is set now to be `no_value` mode, directly referencing `kvobj` entries, and in turn, saves memory. Next, I plan to evaluate replacing the EXPIRE implementation with the [ebuckets](https://github.com/redis/redis/blob/unstable/src/ebuckets.h) data structure, which would eliminate keyspace scans for expired keys. This requires embedding `ExpireMeta` within each `kvobj` of each key with expiration. In such implementation, the `expirable` flag will be shifted to indicate whether `ExpireMeta` is attached. # Implementation notes ## Manipulating keyspace (find, modify, insert) Initially, unifying the key and value into a single object and storing it in dict with `no_value` optimization seemed like a quick win. However, it (quickly) became clear that this change required deeper modifications to how keys are manipulated. The challenge was handling cases where a dictEntry is opt-out due to no_value optimization. In such cases, many of the APIs that return the dictEntry from a lookup become insufficient, as it just might be the key itself. To address this issue, a new-old approach of returning a "link" to the looked-up key's `dictEntry` instead of the `dictEntry` itself. The term `link` was already somewhat available in dict API, and is well aligned with the new dictEntLink declaration: ``` typedef dictEntry **dictEntLink; ``` This PR introduces two new function APIs to dict to leverage returned link from the search: ``` dictEntLink dictFindLink(dict *d, const void *key, dictEntLink *bucket); void dictSetKeyAtLink(dict *d, void *key, dictEntLink *link, int newItem); ``` After calling `link = dictFindLink(...)`, any necessary updates must be performed immediately after by calling `dictSetKeyAtLink()` without any intervening operations on given dict. Otherwise, `dictEntLink` may become invalid. Example: ``` /* replace existing key */ link = dictFindLink(d, key, &bucket, 0); // ... Do something, but don't modify the dict ... // assert(link != NULL); dictSetKeyAtLink(d, kv, &link, 0); /* Add new value (If no space for the new key, dict will be expanded and bucket will be looked up again.) */ link = dictFindLink(d, key, &bucket); // ... Do something, but don't modify the dict ... // assert(link == NULL); dictSetKeyAtLink(d, kv, &bucket, 1); ``` ## dict.h - The dict API has became cluttered with many unused functions. I have removed these from dict.h. - Additionally, APIs specifically related to hash maps (no_value=0), primarily those handling key-value access, have been gathered and isolated. - Removed entirely internal functions ending with “*ByHash()” that were originally added for optimization and not required any more. - Few other legacy dict functions were adapted at API level to work with the term dictEntLink as well. - Simplified and generalized an optimization that related to comparison of length of keys of type strings. ## Hash Field Expiration Until now each hash object with expiration on fields needed to maintain a reference to its key-name (of the hash object), such that in case it will be active-expired, then it will be possible to resolve the key-name for the notification sake. Now there is no need anymore. --------- Co-authored-by: debing.sun --- src/aof.c | 16 +- src/bitops.c | 46 +- src/blocked.c | 2 +- src/cluster.c | 64 +- src/cluster_legacy.c | 2 +- src/db.c | 674 +++++++++++-------- src/debug.c | 81 ++- src/defrag.c | 146 ++-- src/dict.c | 491 ++++++++------ src/dict.h | 76 ++- src/evict.c | 32 +- src/expire.c | 44 +- src/geo.c | 9 +- src/hyperloglog.c | 62 +- src/kvstore.c | 93 ++- src/kvstore.h | 16 +- src/latency.c | 4 +- src/module.c | 430 ++++++------ src/multi.c | 8 +- src/object.c | 355 +++++++--- src/pubsub.c | 7 +- src/rdb.c | 36 +- src/redis-benchmark.c | 6 +- src/redis-cli.c | 6 +- src/redisassert.h | 6 + src/sds.c | 219 ++++-- src/sds.h | 60 +- src/sentinel.c | 2 +- src/server.c | 102 ++- src/server.h | 125 ++-- src/sort.c | 40 +- src/t_hash.c | 247 +++---- src/t_list.c | 59 +- src/t_set.c | 104 ++- src/t_stream.c | 99 ++- src/t_string.c | 130 ++-- src/t_zset.c | 48 +- tests/integration/replication-rdbchannel.tcl | 15 +- tests/unit/info.tcl | 48 +- tests/unit/maxmemory.tcl | 23 - tests/unit/memefficiency.tcl | 4 +- tests/unit/type/hash-field-expire.tcl | 30 + tests/unit/type/incr.tcl | 6 +- tests/unit/type/string.tcl | 33 + 44 files changed, 2346 insertions(+), 1760 deletions(-) diff --git a/src/aof.c b/src/aof.c index 06cd51c79..c534b36ae 100644 --- a/src/aof.c +++ b/src/aof.c @@ -2374,16 +2374,18 @@ int rewriteAppendOnlyFileRio(rio *aof) { kvs_it = kvstoreIteratorInit(db->keys); /* Iterate this DB writing every entry */ while((de = kvstoreIteratorNext(kvs_it)) != NULL) { - sds keystr; - robj key, *o; long long expiretime; size_t aof_bytes_before_key = aof->processed_bytes; - keystr = dictGetKey(de); - o = dictGetVal(de); - initStaticStringObject(key,keystr); - - expiretime = getExpire(db,&key); + /* Get the value object (of type kvobj) */ + kvobj *o = dictGetKV(de); + + /* Get the expire time */ + expiretime = kvobjGetExpire(o); + + /* Set on stack string object for key */ + robj key; + initStaticStringObject(key, kvobjGetKey(o)); /* Save the key and associated value */ if (o->type == OBJ_STRING) { diff --git a/src/bitops.c b/src/bitops.c index 4d33ff953..6286865f4 100644 --- a/src/bitops.c +++ b/src/bitops.c @@ -494,16 +494,17 @@ int getBitfieldTypeFromArgument(client *c, robj *o, int *sign, int *bits) { * * (Must provide all the arguments to the function) */ -static robj *lookupStringForBitCommand(client *c, uint64_t maxbit, +static kvobj *lookupStringForBitCommand(client *c, uint64_t maxbit, size_t *strOldSize, size_t *strGrowSize) { + dictEntryLink link; size_t byte = maxbit >> 3; - robj *o = lookupKeyWrite(c->db,c->argv[1]); + kvobj *o = lookupKeyWriteWithLink(c->db,c->argv[1],&link); if (checkType(c,o,OBJ_STRING)) return NULL; if (o == NULL) { o = createObject(OBJ_STRING,sdsnewlen(NULL, byte+1)); - dbAdd(c->db,c->argv[1],o); + dbAddByLink(c->db,c->argv[1],&o,&link); *strGrowSize = byte + 1; *strOldSize = 0; } else { @@ -548,7 +549,6 @@ unsigned char *getObjectReadOnlyString(robj *o, long *len, char *llbuf) { /* SETBIT key offset bitvalue */ void setbitCommand(client *c) { - robj *o; char *err = "bit is not an integer or out of range"; uint64_t bitoffset; ssize_t byte, bit; @@ -568,8 +568,8 @@ void setbitCommand(client *c) { } size_t strOldSize, strGrowSize; - if ((o = lookupStringForBitCommand(c,bitoffset,&strOldSize,&strGrowSize)) == NULL) - return; + kvobj *o = lookupStringForBitCommand(c, bitoffset, &strOldSize, &strGrowSize); + if (o == NULL) return; /* Get current values */ byte = bitoffset >> 3; @@ -603,7 +603,6 @@ void setbitCommand(client *c) { /* GETBIT key offset */ void getbitCommand(client *c) { - robj *o; char llbuf[32]; uint64_t bitoffset; size_t byte, bit; @@ -612,16 +611,16 @@ void getbitCommand(client *c) { if (getBitOffsetFromArgument(c,c->argv[2],&bitoffset,0,0) != C_OK) return; - if ((o = lookupKeyReadOrReply(c,c->argv[1],shared.czero)) == NULL || - checkType(c,o,OBJ_STRING)) return; + kvobj *kv = lookupKeyReadOrReply(c, c->argv[1], shared.czero); + if (kv == NULL || checkType(c,kv,OBJ_STRING)) return; byte = bitoffset >> 3; bit = 7 - (bitoffset & 0x7); - if (sdsEncodedObject(o)) { - if (byte < sdslen(o->ptr)) - bitval = ((uint8_t*)o->ptr)[byte] & (1 << bit); + if (sdsEncodedObject(kv)) { + if (byte < sdslen(kv->ptr)) + bitval = ((uint8_t*)kv->ptr)[byte] & (1 << bit); } else { - if (byte < (size_t)ll2string(llbuf,sizeof(llbuf),(long)o->ptr)) + if (byte < (size_t)ll2string(llbuf,sizeof(llbuf),(long)kv->ptr)) bitval = llbuf[byte] & (1 << bit); } @@ -632,7 +631,7 @@ void getbitCommand(client *c) { REDIS_NO_SANITIZE("alignment") void bitopCommand(client *c) { char *opname = c->argv[1]->ptr; - robj *o, *targetkey = c->argv[2]; + robj *targetkey = c->argv[2]; unsigned long op, j, numkeys; robj **objects; /* Array of source objects. */ unsigned char **src; /* Array of source strings pointers. */ @@ -667,9 +666,9 @@ void bitopCommand(client *c) { len = zmalloc(sizeof(long) * numkeys); objects = zmalloc(sizeof(robj*) * numkeys); for (j = 0; j < numkeys; j++) { - o = lookupKeyRead(c->db,c->argv[j+3]); + kvobj *kv = lookupKeyRead(c->db, c->argv[j + 3]); /* Handle non-existing keys as empty strings. */ - if (o == NULL) { + if (kv == NULL) { objects[j] = NULL; src[j] = NULL; len[j] = 0; @@ -677,7 +676,7 @@ void bitopCommand(client *c) { continue; } /* Return an error if one of the keys is not a string. */ - if (checkType(c,o,OBJ_STRING)) { + if (checkType(c, kv, OBJ_STRING)) { unsigned long i; for (i = 0; i < j; i++) { if (objects[i]) @@ -688,7 +687,7 @@ void bitopCommand(client *c) { zfree(objects); return; } - objects[j] = getDecodedObject(o); + objects[j] = getDecodedObject(kv); src[j] = objects[j]->ptr; len[j] = sdslen(objects[j]->ptr); if (len[j] > maxlen) maxlen = len[j]; @@ -805,10 +804,9 @@ void bitopCommand(client *c) { /* Store the computed value into the target key */ if (maxlen) { - o = createObject(OBJ_STRING,res); - setKey(c,c->db,targetkey,o,0); + robj *o = createObject(OBJ_STRING, res); + setKey(c, c->db, targetkey, &o, 0); notifyKeyspaceEvent(NOTIFY_STRING,"set",targetkey,c->db->id); - decrRefCount(o); server.dirty++; } else if (dbDelete(c->db,targetkey)) { signalModifiedKey(c,c->db,targetkey); @@ -820,7 +818,7 @@ void bitopCommand(client *c) { /* BITCOUNT key [start end [BIT|BYTE]] */ void bitcountCommand(client *c) { - robj *o; + kvobj *o; long long start, end; long strlen; unsigned char *p; @@ -912,7 +910,7 @@ void bitcountCommand(client *c) { /* BITPOS key bit [start [end [BIT|BYTE]]] */ void bitposCommand(client *c) { - robj *o; + kvobj *o; long long start, end; long bit, strlen; unsigned char *p; @@ -1077,7 +1075,7 @@ struct bitfieldOp { * when flags is set to BITFIELD_FLAG_READONLY: in this case only the * GET subcommand is allowed, other subcommands will return an error. */ void bitfieldGeneric(client *c, int flags) { - robj *o; + kvobj *o; uint64_t bitoffset; int j, numops = 0, changes = 0; size_t strOldSize, strGrowSize = 0; diff --git a/src/blocked.c b/src/blocked.c index 7830e9913..0d9406a89 100644 --- a/src/blocked.c +++ b/src/blocked.c @@ -575,7 +575,7 @@ static void handleClientsBlockedOnKey(readyList *rl) { long count = listLength(clients); while ((ln = listNext(&li)) && count--) { client *receiver = listNodeValue(ln); - robj *o = lookupKeyReadWithFlags(rl->db, rl->key, LOOKUP_NOEFFECTS); + kvobj *o = lookupKeyReadWithFlags(rl->db, rl->key, LOOKUP_NOEFFECTS); /* 1. In case new key was added/touched we need to verify it satisfy the * blocked type, since we might process the wrong key type. * 2. We want to serve clients blocked on module keys diff --git a/src/cluster.c b/src/cluster.c index 3d0e78404..070c278b9 100644 --- a/src/cluster.c +++ b/src/cluster.c @@ -131,7 +131,7 @@ int verifyDumpPayload(unsigned char *p, size_t len, uint16_t *rdbver_ptr) { * DUMP is actually not used by Redis Cluster but it is the obvious * complement of RESTORE and can be useful for different applications. */ void dumpCommand(client *c) { - robj *o; + kvobj *o; rio payload; /* Check if the key is here. */ @@ -239,14 +239,14 @@ void restoreCommand(client *c) { } /* Create the key and set the TTL if any */ - dictEntry *de = dbAdd(c->db,key,obj); + kvobj *kv = dbAdd(c->db, key, &obj); /* If minExpiredField was set, then the object is hash with expiration * on fields and need to register it in global HFE DS */ - if (obj->type == OBJ_HASH) { - uint64_t minExpiredField = hashTypeGetMinExpire(obj, 1); + if (kv->type == OBJ_HASH) { + uint64_t minExpiredField = hashTypeGetMinExpire(kv, 1); if (minExpiredField != EB_EXPIRE_TIME_INVALID) - hashTypeAddToExpires(c->db, dictGetKey(de), obj, minExpiredField); + hashTypeAddToExpires(c->db, kv, minExpiredField); } if (ttl) { @@ -259,7 +259,7 @@ void restoreCommand(client *c) { rewriteClientCommandArgument(c,c->argc,shared.absttl); } } - objectSetLRUOrLFU(obj,lfu_freq,lru_idle,lru_clock,1000); + objectSetLRUOrLFU(kv, lfu_freq, lru_idle, lru_clock, 1000); signalModifiedKey(c,c->db,key); notifyKeyspaceEvent(NOTIFY_GENERIC,"restore",key,c->db->id); addReply(c,shared.ok); @@ -388,8 +388,8 @@ void migrateCommand(client *c) { char *password = NULL; long timeout; long dbid; - robj **ov = NULL; /* Objects to migrate. */ - robj **kv = NULL; /* Key names. */ + robj **kvArray = NULL; /* Objects to migrate. */ + robj **keyArray = NULL; /* Key names. */ robj **newargv = NULL; /* Used to rewrite the command as DEL ... keys ... */ rio cmd, payload; int may_retry = 1; @@ -453,19 +453,19 @@ void migrateCommand(client *c) { * the caller there was nothing to migrate. We don't return an error in * this case, since often this is due to a normal condition like the key * expiring in the meantime. */ - ov = zrealloc(ov,sizeof(robj*)*num_keys); - kv = zrealloc(kv,sizeof(robj*)*num_keys); - int oi = 0; + kvArray = zrealloc(kvArray,sizeof(kvobj*)*num_keys); + keyArray = zrealloc(keyArray,sizeof(robj*)*num_keys); + int num_exists = 0; for (j = 0; j < num_keys; j++) { - if ((ov[oi] = lookupKeyRead(c->db,c->argv[first_key+j])) != NULL) { - kv[oi] = c->argv[first_key+j]; - oi++; + if ((kvArray[num_exists] = lookupKeyRead(c->db,c->argv[first_key+j])) != NULL) { + keyArray[num_exists] = c->argv[first_key+j]; + num_exists++; } } - num_keys = oi; + num_keys = num_exists; if (num_keys == 0) { - zfree(ov); zfree(kv); + zfree(kvArray); zfree(keyArray); addReplySds(c,sdsnew("+NOKEY\r\n")); return; } @@ -476,7 +476,7 @@ void migrateCommand(client *c) { /* Connect */ cs = migrateGetSocket(c,c->argv[1],c->argv[2],timeout); if (cs == NULL) { - zfree(ov); zfree(kv); + zfree(kvArray); zfree(keyArray); return; /* error sent to the client by migrateGetSocket() */ } @@ -511,7 +511,7 @@ void migrateCommand(client *c) { /* Create RESTORE payload and generate the protocol to call the command. */ for (j = 0; j < num_keys; j++) { long long ttl = 0; - long long expireat = getExpire(c->db,kv[j]); + long long expireat = kvobjGetExpire(kvArray[j]); if (expireat != -1) { ttl = expireat-commandTimeSnapshot(); @@ -524,8 +524,8 @@ void migrateCommand(client *c) { /* Relocate valid (non expired) keys and values into the array in successive * positions to remove holes created by the keys that were present * in the first lookup but are now expired after the second lookup. */ - ov[non_expired] = ov[j]; - kv[non_expired++] = kv[j]; + kvArray[non_expired] = kvArray[j]; + keyArray[non_expired++] = keyArray[j]; serverAssertWithInfo(c,NULL, rioWriteBulkCount(&cmd,'*',replace ? 5 : 4)); @@ -535,14 +535,14 @@ void migrateCommand(client *c) { rioWriteBulkString(&cmd,"RESTORE-ASKING",14)); else serverAssertWithInfo(c,NULL,rioWriteBulkString(&cmd,"RESTORE",7)); - serverAssertWithInfo(c,NULL,sdsEncodedObject(kv[j])); - serverAssertWithInfo(c,NULL,rioWriteBulkString(&cmd,kv[j]->ptr, - sdslen(kv[j]->ptr))); + serverAssertWithInfo(c,NULL,sdsEncodedObject(keyArray[j])); + serverAssertWithInfo(c,NULL,rioWriteBulkString(&cmd,keyArray[j]->ptr, + sdslen(keyArray[j]->ptr))); serverAssertWithInfo(c,NULL,rioWriteBulkLongLong(&cmd,ttl)); /* Emit the payload argument, that is the serialized object using * the DUMP format. */ - createDumpPayload(&payload,ov[j],kv[j],dbid); + createDumpPayload(&payload,kvArray[j],keyArray[j],dbid); serverAssertWithInfo(c,NULL, rioWriteBulkString(&cmd,payload.io.buffer.ptr, sdslen(payload.io.buffer.ptr))); @@ -622,14 +622,14 @@ void migrateCommand(client *c) { } else { if (!copy) { /* No COPY option: remove the local key, signal the change. */ - dbDelete(c->db,kv[j]); - signalModifiedKey(c,c->db,kv[j]); - notifyKeyspaceEvent(NOTIFY_GENERIC,"del",kv[j],c->db->id); + dbDelete(c->db,keyArray[j]); + signalModifiedKey(c,c->db,keyArray[j]); + notifyKeyspaceEvent(NOTIFY_GENERIC,"del",keyArray[j],c->db->id); server.dirty++; /* Populate the argument vector to replace the old one. */ - newargv[del_idx++] = kv[j]; - incrRefCount(kv[j]); + newargv[del_idx++] = keyArray[j]; + incrRefCount(keyArray[j]); } } } @@ -687,7 +687,7 @@ void migrateCommand(client *c) { } sdsfree(cmd.io.buffer.ptr); - zfree(ov); zfree(kv); zfree(newargv); + zfree(kvArray); zfree(keyArray); zfree(newargv); return; /* On socket errors we try to close the cached socket and try again. @@ -714,7 +714,7 @@ void migrateCommand(client *c) { } /* Cleanup we want to do if no retry is attempted. */ - zfree(ov); zfree(kv); + zfree(kvArray); zfree(keyArray); addReplyErrorSds(c, sdscatprintf(sdsempty(), "-IOERR error or timeout %s to target instance", write_error ? "writing" : "reading")); @@ -1011,7 +1011,7 @@ void clusterCommand(client *c) { for (unsigned int i = 0; i < numkeys; i++) { de = kvstoreDictIteratorNext(kvs_di); serverAssert(de != NULL); - sds sdskey = dictGetKey(de); + sds sdskey = kvobjGetKey(dictGetKV(de)); addReplyBulkCBuffer(c, sdskey, sdslen(sdskey)); } kvstoreReleaseDictIterator(kvs_di); diff --git a/src/cluster_legacy.c b/src/cluster_legacy.c index 88dc99691..54ddf02c6 100644 --- a/src/cluster_legacy.c +++ b/src/cluster_legacy.c @@ -5793,7 +5793,7 @@ unsigned int delKeysInSlot(unsigned int hashslot) { kvs_di = kvstoreGetDictSafeIterator(server.db->keys, hashslot); while((de = kvstoreDictIteratorNext(kvs_di)) != NULL) { enterExecutionUnit(1, 0); - sds sdskey = dictGetKey(de); + sds sdskey = kvobjGetKey(dictGetKV(de)); robj *key = createStringObject(sdskey, sdslen(sdskey)); dbDelete(&server.db[0], key); propagateDeletion(&server.db[0], key, server.lazyfree_lazy_server_del); diff --git a/src/db.c b/src/db.c index 0563f7eb6..f19505292 100644 --- a/src/db.c +++ b/src/db.c @@ -2,6 +2,9 @@ * Copyright (c) 2009-Present, Redis Ltd. * All rights reserved. * + * Copyright (c) 2024-present, Valkey contributors. + * All rights reserved. + * * Licensed under your choice of (a) the Redis Source Available License 2.0 * (RSALv2); or (b) the Server Side Public License v1 (SSPLv1); or (c) the * GNU Affero General Public License v3 (AGPLv3). @@ -36,12 +39,8 @@ typedef enum { KEY_DELETED /* The key was deleted now. */ } keyStatus; -static inline keyStatus expireIfNeededWithSlot(redisDb *db, robj *key, int flags, const int keySlot); -keyStatus expireIfNeeded(redisDb *db, robj *key, int flags); -int keyIsExpired(redisDb *db, robj *key); -static void dbSetValue(redisDb *db, robj *key, robj *val, int overwrite, dictEntry *de); -static inline dictEntry *dbFindWithKeySlot(redisDb *db, void *key, int keySlot); -static inline dictEntry *dbFindExpiresWithKeySlot(redisDb *db, void *key, int keySlot); +static keyStatus expireIfNeeded(redisDb *db, robj *key, kvobj *kv, int flags); +static void dbSetValue(redisDb *db, robj *key, robj **valref, int overwrite, dictEntryLink link); /* Update LFU when an object is accessed. * Firstly, decrement the counter if the decrement time is reached. @@ -92,7 +91,7 @@ void updateKeysizesHist(redisDb *db, int didx, uint32_t type, int64_t oldLen, in } else { /* here, oldLen can be either 0 or -1 */ if (oldLen == 0) { - /* Only strings can be empty. Yet, a command flow might temporarily + /* Only strings can be empty. Yet, a command flow might temporarily * dbAdd() empty collection, and only after add elements. */ if (dictMeta) { @@ -114,21 +113,22 @@ void updateKeysizesHist(redisDb *db, int didx, uint32_t type, int64_t oldLen, in } else { /* here, newLen can be either 0 or -1 */ if (newLen == 0) { - /* Only strings can be empty. Yet, a command flow might temporarily + /* Only strings can be empty. Yet, a command flow might temporarily * dbAdd() empty collection, and only after add elements. */ - + if (dictMeta) dictMeta->keysizes_hist[type][0]++; kvstoreMeta->keysizes_hist[type][0]++; } } } -/* Lookup a key for read or write operations, or return NULL if the key is not +/* Lookup a kvobj for read or write operations, or return NULL if the it is not * found in the specified DB. This function implements the functionality of * lookupKeyRead(), lookupKeyWrite() and their ...WithFlags() variants. * - * 'deref' is an optional output dictEntry reference argument, to get the - * associated dictEntry* of the key in case the key is found. + * link - If key found, return the link of the key. + * If key not found, return the bucket link, where the key should be added. + * Or NULL if dict wasn't allocated yet. * * Side-effects of calling this function: * @@ -153,12 +153,11 @@ void updateKeysizesHist(redisDb *db, int didx, uint32_t type, int64_t oldLen, in * Even if the key expiry is master-driven, we can correctly report a key is * expired on replicas even if the master is lagging expiring our key via DELs * in the replication link. */ -robj *lookupKey(redisDb *db, robj *key, int flags, dictEntry **deref) { - const int key_slot = getKeySlot(key->ptr); - dictEntry *de = dbFindWithKeySlot(db, key->ptr, key_slot); - robj *val = NULL; - if (de) { - val = dictGetVal(de); +kvobj *lookupKey(redisDb *db, robj *key, int flags, dictEntryLink *link) { + + kvobj *val = dbFindByLink(db, key->ptr, link); + + if (val) { /* Forcing deletion of expired keys on a replica makes the replica * inconsistent with the master. We forbid it on readonly replicas, but * we have to allow it on writable replicas to make write commands @@ -175,9 +174,10 @@ robj *lookupKey(redisDb *db, robj *key, int flags, dictEntry **deref) { expire_flags |= EXPIRE_AVOID_DELETE_EXPIRED; if (flags & LOOKUP_ACCESS_EXPIRED) expire_flags |= EXPIRE_ALLOW_ACCESS_EXPIRED; - if (expireIfNeededWithSlot(db, key, expire_flags, key_slot) != KEY_VALID) { + if (expireIfNeeded(db, key, val, expire_flags) != KEY_VALID) { /* The key is no longer valid. */ val = NULL; + if (link) *link = NULL; } } @@ -207,7 +207,6 @@ robj *lookupKey(redisDb *db, robj *key, int flags, dictEntry **deref) { /* TODO: Use separate misses stats and notify event for WRITE */ } - if (val && deref) *deref = de; return val; } @@ -220,14 +219,14 @@ robj *lookupKey(redisDb *db, robj *key, int flags, dictEntry **deref) { * This function is equivalent to lookupKey(). The point of using this function * rather than lookupKey() directly is to indicate that the purpose is to read * the key. */ -robj *lookupKeyReadWithFlags(redisDb *db, robj *key, int flags) { +kvobj *lookupKeyReadWithFlags(redisDb *db, robj *key, int flags) { serverAssert(!(flags & LOOKUP_WRITE)); return lookupKey(db, key, flags, NULL); } /* Like lookupKeyReadWithFlags(), but does not use any flag, which is the * common case. */ -robj *lookupKeyRead(redisDb *db, robj *key) { +kvobj *lookupKeyRead(redisDb *db, robj *key) { return lookupKeyReadWithFlags(db,key,LOOKUP_NONE); } @@ -237,58 +236,69 @@ robj *lookupKeyRead(redisDb *db, robj *key) { * * Returns the linked value object if the key exists or NULL if the key * does not exist in the specified DB. */ -robj *lookupKeyWriteWithFlags(redisDb *db, robj *key, int flags) { +kvobj *lookupKeyWriteWithFlags(redisDb *db, robj *key, int flags) { return lookupKey(db, key, flags | LOOKUP_WRITE, NULL); } -robj *lookupKeyWrite(redisDb *db, robj *key) { +kvobj *lookupKeyWrite(redisDb *db, robj *key) { return lookupKeyWriteWithFlags(db, key, LOOKUP_NONE); } -/* Like lookupKeyWrite(), but accepts an optional dictEntry input, - * which can be used if we already have one, thus saving the dbFind call. - */ -robj *lookupKeyWriteWithDictEntry(redisDb *db, robj *key, dictEntry **deref) { - return lookupKey(db, key, LOOKUP_NONE | LOOKUP_WRITE, deref); -} - -robj *lookupKeyReadOrReply(client *c, robj *key, robj *reply) { - robj *o = lookupKeyRead(c->db, key); - if (!o) addReplyOrErrorObject(c, reply); - return o; -} - -robj *lookupKeyWriteOrReply(client *c, robj *key, robj *reply) { - robj *o = lookupKeyWrite(c->db, key); - if (!o) addReplyOrErrorObject(c, reply); - return o; -} - -/* Add the key to the DB. It's up to the caller to increment the reference - * counter of the value if needed. +/* Like lookupKeyWrite(), but accepts ref to optional `link` * - * If the update_if_existing argument is false, the program is aborted - * if the key already exists, otherwise, it can fall back to dbOverwrite. */ -static dictEntry *dbAddInternal(redisDb *db, robj *key, robj *val, int update_if_existing) { - dictEntry *existing; - int slot = getKeySlot(key->ptr); - dictEntry *de = kvstoreDictAddRaw(db->keys, slot, key->ptr, &existing); - if (update_if_existing && existing) { - dbSetValue(db, key, val, 1, existing); - return existing; - } - serverAssertWithInfo(NULL, key, de != NULL); - kvstoreDictSetKey(db->keys, slot, de, sdsdup(key->ptr)); - initObjectLRUOrLFU(val); - kvstoreDictSetVal(db->keys, slot, de, val); - signalKeyAsReady(db, key, val->type); - notifyKeyspaceEvent(NOTIFY_NEW,"new",key,db->id); - updateKeysizesHist(db, slot, val->type, -1, getObjectLength(val)); /* add hist */ - return de; + * link - If key found, updated to link the key. + * If key not found, updated to the bucket where the key should be added. + * If key not found and dict is empty, it is set to NULL + */ +kvobj *lookupKeyWriteWithLink(redisDb *db, robj *key, dictEntryLink *link) { + return lookupKey(db, key, LOOKUP_NONE | LOOKUP_WRITE, link); } -dictEntry *dbAdd(redisDb *db, robj *key, robj *val) { - return dbAddInternal(db, key, val, 0); +kvobj *lookupKeyReadOrReply(client *c, robj *key, robj *reply) { + kvobj *kv = lookupKeyRead(c->db, key); + if (!kv) addReplyOrErrorObject(c, reply); + return kv; +} + +kvobj *lookupKeyWriteOrReply(client *c, robj *key, robj *reply) { + kvobj *kv = lookupKeyWrite(c->db, key); + if (!kv) addReplyOrErrorObject(c, reply); + return kv; +} + +/* Add a key-value entry to the DB. + * + * A copy of 'key' is stored in the database. The caller must ensure the + * `key` is properly freed by calling decrRefcount(key). + * + * The value may (if its reference counter == 1) be reallocated and become + * invalid after a call to this function. The (possibly reallocated) value is + * stored in the database and the 'valref' pointer is updated to point to the + * new allocation. + * + * The reference counter of the value pointed to by valref is not incremented, + * so the caller should not free the value using decrRefcount after calling this + * function. + * + * link - Optional link to bucket where the key should be added. + * On return, get updated, by need, to the inserted key. + */ +kvobj *dbAddByLink(redisDb *db, robj *key, robj **valref, dictEntryLink *link) { + int slot = getKeySlot(key->ptr); + robj *val = *valref; + kvobj *kv = kvobjSet(key->ptr, val, -1); + initObjectLRUOrLFU(kv); + kvstoreDictSetAtLink(db->keys, slot, kv, link, 1); + signalKeyAsReady(db, key, kv->type); + notifyKeyspaceEvent(NOTIFY_NEW,"new",key,db->id); + updateKeysizesHist(db, slot, kv->type, -1, getObjectLength(kv)); /* add hist */ + *valref = kv; + return kv; +} + +/* Read dbAddByLink() comment */ +kvobj *dbAdd(redisDb *db, robj *key, robj **valref) { + return dbAddByLink(db, key, valref, NULL); } /* Returns key's hash slot when cluster mode is enabled, or 0 when disabled. @@ -316,46 +326,73 @@ int getKeySlot(sds key) { /* This is a special version of dbAdd() that is used only when loading * keys from the RDB file: the key is passed as an SDS string that is - * retained by the function (and not freed by the caller). + * copied by the function and freed by the caller. * * Moreover this function will not abort if the key is already busy, to * give more control to the caller, nor will signal the key as ready * since it is not useful in this context. * - * The function returns 1 if the key was added to the database, taking - * ownership of the SDS string, otherwise 0 is returned, and is up to the - * caller to free the SDS string. */ -int dbAddRDBLoad(redisDb *db, sds key, robj *val) { + * If added to db, returns pointer to the object, Otherwise NULL is returned. + */ +kvobj *dbAddRDBLoad(redisDb *db, sds key, robj **valref, long long expire) { + /* Add new kvobj to the db. */ int slot = getKeySlot(key); - dictEntry *de = kvstoreDictAddRaw(db->keys, slot, key, NULL); - if (de == NULL) return 0; - updateKeysizesHist(db, slot, val->type, -1, getObjectLength(val)); /* add hist */ - initObjectLRUOrLFU(val); - kvstoreDictSetVal(db->keys, slot, de, val); - return 1; + + dictEntryLink link, bucket; + link = kvstoreDictFindLink(db->keys, slot, key, &bucket); + + /* If already exists, return NULL */ + if (link != NULL) + return NULL; + + /* prepare kvobj for insertion. Pass expire to reserve space for it */ + kvobj *kv = kvobjSet(key, *valref, -1); + initObjectLRUOrLFU(kv); + kvstoreDictSetAtLink(db->keys, slot, kv, &bucket, 1); + + /* Set the expire time if needed */ + if (expire != -1) + kv = setExpireByLink(NULL, db, key, expire, bucket); + + updateKeysizesHist(db, slot, kv->type, -1, (int64_t) getObjectLength(kv)); + return *valref = kv; } -/* Overwrite an existing key with a new value. Incrementing the reference - * count of the new value is up to the caller. - * This function does not modify the expire time of the existing key. +/** + * Overwrite an existing key's value in db with a new value. * - * The 'overwrite' flag is an indication whether this is done as part of a - * complete replacement of their key, which can be thought as a deletion and - * replacement (in which case we need to emit deletion signals), or just an - * update of a value of an existing key (when false). - * - * The dictEntry input is optional, can be used if we already have one. - * - * The program is aborted if the key was not already present. */ -static void dbSetValue(redisDb *db, robj *key, robj *val, int overwrite, dictEntry *de) { + * - If the reference count of 'valref' is 1 the ownership of the value is + * transferred to this function. The value may be reallocated, potentially + * invalidating any external references to it. The (potentially reallocated) + * value is stored in the database, and the 'valref' pointer is updated to + * reflect the new allocation, if one occurs. + * - The reference counter of the value referenced by 'valref' is not incremented + * so the caller must refrain from releasing it using decrRefCount after this + * function is called. + * - This function does not modify the expire time of the existing key. + * - The 'overwrite' flag is an indication whether this is done as part of a + * complete replacement of their key, which can be thought as a deletion and + * replacement (in which case we need to emit deletion signals), or just an + * update of a value of an existing key (when false). + * - The `link` is optional, can save lookup, if provided. + */ +static void dbSetValue(redisDb *db, robj *key, robj **valref, int overwrite, dictEntryLink link) { + robj *val = *valref; int slot = getKeySlot(key->ptr); - if (!de) de = kvstoreDictFind(db->keys, slot, key->ptr); - serverAssertWithInfo(NULL,key,de != NULL); - robj *old = dictGetVal(de); + if (!link) { + link = kvstoreDictFindLink(db->keys, slot, key->ptr, NULL); + serverAssertWithInfo(NULL, key, link != NULL); /* expected to exist */ + } + kvobj *old = dictGetKV(*link); + kvobj *kvNew; int64_t oldlen = (int64_t) getObjectLength(old); + int oldtype = old->type; - val->lru = old->lru; + /* if hash with HFEs, take care to remove from global HFE DS before attempting + * to manipulate and maybe free kvOld object */ + if (old->type == OBJ_HASH) + hashTypeRemoveFromExpires(&db->hexpires, old); if (overwrite) { /* RM_StringDMA may call dbUnshareStringValue which may free val, so we @@ -369,49 +406,77 @@ static void dbSetValue(redisDb *db, robj *key, robj *val, int overwrite, dictEnt signalDeletedKeyAsReady(db,key,old->type); decrRefCount(old); /* Because of RM_StringDMA, old may be changed, so we need get old again */ - old = dictGetVal(de); + old = dictGetKV(*link); + } + + if ((old->refcount == 1 && old->encoding != OBJ_ENCODING_EMBSTR) && + (val->refcount == 1 && val->encoding != OBJ_ENCODING_EMBSTR)) { + /* Keep old object in the database. Just swap it's ptr, type and + * encoding with the content of val. */ + robj tmp = *old; + old->type = val->type; + old->encoding = val->encoding; + old->ptr = val->ptr; + val->type = tmp.type; + val->encoding = tmp.encoding; + val->ptr = tmp.ptr; + /* Set new to old to keep the old object. Set old to val to be freed below. */ + kvNew = old; + old = val; + } else { + /* Replace the old value at its location in the key space. */ + val->lru = old->lru; + /* Update expire reference if needed */ + long long expire = getExpire(db, key->ptr, old); + kvNew = kvobjSet(key->ptr, val, expire); + kvstoreDictSetAtLink(db->keys, slot, kvNew, &link, 0); + + /* Replace the old value at its location in the expire space. */ + if (expire >= 0) { + dictEntryLink exLink = kvstoreDictFindLink(db->expires, slot, key->ptr, NULL); + serverAssertWithInfo(NULL,key,exLink != NULL); + kvstoreDictSetAtLink(db->expires, slot, kvNew, &exLink, 0); + } } - kvstoreDictSetVal(db->keys, slot, de, val); /* Remove old key and add new key to KEYSIZES histogram */ - int64_t newlen = (int64_t) getObjectLength(val); + int64_t newlen = (int64_t) getObjectLength(kvNew); /* Save one call if old and new are the same type */ - if (old->type == val->type) { - updateKeysizesHist(db, slot, old->type, oldlen, newlen); + if (oldtype == kvNew->type) { + updateKeysizesHist(db, slot, oldtype, oldlen, newlen); } else { - updateKeysizesHist(db, slot, old->type, oldlen, -1); - updateKeysizesHist(db, slot, val->type, -1, newlen); + updateKeysizesHist(db, slot, oldtype, oldlen, -1); + updateKeysizesHist(db, slot, kvNew->type, -1, newlen); } - /* if hash with HFEs, take care to remove from global HFE DS */ - if (old->type == OBJ_HASH) - hashTypeRemoveFromExpires(&db->hexpires, old); - if (server.lazyfree_lazy_server_del) { - freeObjAsync(key,old,db->id); + freeObjAsync(key, old, db->id); } else { decrRefCount(old); } + *valref = kvNew; } /* Replace an existing key with a new value, we just replace value and don't * emit any events */ -void dbReplaceValue(redisDb *db, robj *key, robj *val) { - dbSetValue(db, key, val, 0, NULL); +void dbReplaceValue(redisDb *db, robj *key, robj **valref) { + dbSetValue(db, key, valref, 0, NULL); } -/* Replace an existing key with a new value, we just replace value and don't - * emit any events. - * The dictEntry input is optional, can be used if we already have one. +/* Replace an existing key with a new value (don't emit any events) + * + * parameter 'link' is optional. If provided, saves lookup. */ -void dbReplaceValueWithDictEntry(redisDb *db, robj *key, robj *val, dictEntry *de) { - dbSetValue(db, key, val, 0, de); +void dbReplaceValueWithLink(redisDb *db, robj *key, robj **val, dictEntryLink link) { + dbSetValue(db, key, val, 0, link); } /* High level Set operation. This function can be used in order to set * a key, whatever it was existing or not, to a new object. * - * 1) The ref count of the value object is incremented. + * 1) The value may be reallocated when adding it to the database. The value + * pointer 'valref' is updated to point to the reallocated object. The + * reference count of the value object is *not* incremented. * 2) clients WATCHing for the destination key notified. * 3) The expire time of the key is reset (the key is made persistent), * unless 'SETKEY_KEEPTTL' is enabled in flags. @@ -421,32 +486,45 @@ void dbReplaceValueWithDictEntry(redisDb *db, robj *key, robj *val, dictEntry *d * All the new keys in the database should be created via this interface. * The client 'c' argument may be set to NULL if the operation is performed * in a context where there is no clear client performing the operation. */ -void setKey(client *c, redisDb *db, robj *key, robj *val, int flags) { - setKeyWithDictEntry(c,db,key,val,flags,NULL); +void setKey(client *c, redisDb *db, robj *key, robj **valref, int flags) { + setKeyByLink(c, db, key, valref, flags, NULL); } -/* Like setKey(), but accepts an optional dictEntry input, - * which can be used if we already have one, thus saving the dictFind call. */ -void setKeyWithDictEntry(client *c, redisDb *db, robj *key, robj *val, int flags, dictEntry *de) { - int keyfound = 0; +/* Like setKey(), but accepts an optional link + * + * - If flags is set with SETKEY_ALREADY_EXIST, then `link` must be provided + * - If flags is set with SETKEY_DOESNT_EXIST, then `link` is optional. If + * provided, it will point to the bucket where the key should be added. + * - If flag is not set (0) then add or update key, and `link` must be NULL + * On return, link get updated, by need, to the inserted kvobj. + */ +void setKeyByLink(client *c, redisDb *db, robj *key, robj **valref, int flags, dictEntryLink *plink) { + dictEntryLink dummy = NULL, *link = plink ? plink : &dummy; + int exists; - if (flags & SETKEY_ALREADY_EXIST) - keyfound = 1; - else if (flags & SETKEY_ADD_OR_UPDATE) - keyfound = -1; - else if (!(flags & SETKEY_DOESNT_EXIST)) - keyfound = (lookupKeyWrite(db,key) != NULL); - - if (!keyfound) { - dbAdd(db,key,val); - } else if (keyfound<0) { - dbAddInternal(db,key,val,1); + if (flags & SETKEY_ALREADY_EXIST) { + exists = 1; + debugServerAssert((*link) != NULL); + } else if (flags & SETKEY_DOESNT_EXIST) { + /* link is optional */ + exists = 0; } else { - dbSetValue(db,key,val,1,de); + /* Add or update key */ + exists = (lookupKeyWriteWithLink(db, key, link)) != NULL; } - incrRefCount(val); - if (!(flags & SETKEY_KEEPTTL)) removeExpire(db,key); - if (!(flags & SETKEY_NO_SIGNAL)) signalModifiedKey(c,db,key); + + if (exists) { + /* Update the value of an existing key */ + dbSetValue(db, key, valref, 1, *link); + if ((-1 != kvobjGetExpire(*valref)) && (!(flags & SETKEY_KEEPTTL))) + removeExpire(db,key); + } else { + /* Add the new key to the database */ + dbAddByLink(db, key, valref, link); + } + + if (!(flags & SETKEY_NO_SIGNAL)) + signalModifiedKey(c,db,key); } /* Return a random key, in form of a Redis object. @@ -459,13 +537,13 @@ robj *dbRandomKey(redisDb *db) { int allvolatile = kvstoreSize(db->keys) == kvstoreSize(db->expires); while(1) { - sds key; robj *keyobj; int randomSlot = kvstoreGetFairRandomDictIndex(db->keys); de = kvstoreDictGetFairRandomKey(db->keys, randomSlot); if (de == NULL) return NULL; - key = dictGetKey(de); + kvobj *kv = dictGetKV(de); + sds key = kvobjGetKey(kv); keyobj = createStringObject(key,sdslen(key)); if (allvolatile && (server.masterhost || isPausedActions(PAUSE_ACTION_EXPIRE)) && --maxtries == 0) { /* If the DB is composed only of keys with an expire set, @@ -478,7 +556,7 @@ robj *dbRandomKey(redisDb *db) { * return a key name that may be already expired. */ return keyobj; } - if (expireIfNeededWithSlot(db,keyobj,0,randomSlot) != KEY_VALID) { + if (expireIfNeeded(db, keyobj, kv, 0) != KEY_VALID) { decrRefCount(keyobj); continue; /* search for another key. This expired. */ } @@ -489,15 +567,16 @@ robj *dbRandomKey(redisDb *db) { /* Helper for sync and async delete. */ int dbGenericDelete(redisDb *db, robj *key, int async, int flags) { - dictEntry **plink; + dictEntryLink link; int table; int slot = getKeySlot(key->ptr); - dictEntry *de = kvstoreDictTwoPhaseUnlinkFind(db->keys, slot, key->ptr, &plink, &table); - if (de) { - robj *val = dictGetVal(de); + link = kvstoreDictTwoPhaseUnlinkFind(db->keys, slot, key->ptr, &table); + + if (link) { + kvobj *val = dictGetKV(*link); int64_t oldlen = (int64_t) getObjectLength(val); - int type = val->type; + int type = val->type; /* If hash object with expiry on fields, remove it from HFE DS of DB */ if (type == OBJ_HASH) @@ -505,7 +584,7 @@ int dbGenericDelete(redisDb *db, robj *key, int async, int flags) { /* RM_StringDMA may call dbUnshareStringValue which may free val, so we * need to incr to retain val */ - incrRefCount(val); + incrRefCount(val); /* refcnt=1->2 */ /* Tells the module that the key has been unlinked from the database. */ moduleNotifyKeyUnlink(key,val,db->id,flags); /* We want to try to unblock any module clients or clients using a blocking XREADGROUP */ @@ -513,21 +592,21 @@ int dbGenericDelete(redisDb *db, robj *key, int async, int flags) { /* We should call decr before freeObjAsync. If not, the refcount may be * greater than 1, so freeObjAsync doesn't work */ decrRefCount(val); - if (async) { - /* Because of dbUnshareStringValue, the val in de may change. */ - freeObjAsync(key, dictGetVal(de), db->id); - kvstoreDictSetVal(db->keys, slot, de, NULL); - } - /* Deleting an entry from the expires dict will not free the sds of - * the key, because it is shared with the main dictionary. */ + + /* Delete an entry from the expires dict is not decrRefCount of kvobj */ kvstoreDictDelete(db->expires, slot, key->ptr); - kvstoreDictTwoPhaseUnlinkFree(db->keys, slot, de, plink, table); + if (async) { + /* Because of dbUnshareStringValue, the val in db may change. */ + freeObjAsync(key, dictGetKV(*link), db->id); + /* Set the key to NULL in the main dictionary. */ + kvstoreDictSetAtLink(db->keys, slot, NULL, &link, 0); + } + kvstoreDictTwoPhaseUnlinkFree(db->keys, slot, link, table); /* remove key from histogram */ if(!(flags & DB_FLAG_NO_UPDATE_KEYSIZES)) updateKeysizesHist(db, slot, type, oldlen, -1); - return 1; } else { return 0; @@ -589,19 +668,19 @@ int dbDeleteSkipKeysizesUpdate(redisDb *db, robj *key) { * At this point the caller is ready to modify the object, for example * using an sdscat() call to append some data, or anything else. */ -robj *dbUnshareStringValue(redisDb *db, robj *key, robj *o) { - return dbUnshareStringValueWithDictEntry(db,key,o,NULL); +kvobj *dbUnshareStringValue(redisDb *db, robj *key, kvobj *kv) { + return dbUnshareStringValueByLink(db,key,kv,NULL); } -/* Like dbUnshareStringValue(), but accepts a optional dictEntry, +/* Like dbUnshareStringValue(), but accepts a optional link, * which can be used if we already have one, thus saving the dbFind call. */ -robj *dbUnshareStringValueWithDictEntry(redisDb *db, robj *key, robj *o, dictEntry *de) { +kvobj *dbUnshareStringValueByLink(redisDb *db, robj *key, kvobj *o, dictEntryLink link) { serverAssert(o->type == OBJ_STRING); if (o->refcount != 1 || o->encoding != OBJ_ENCODING_RAW) { robj *decoded = getDecodedObject(o); o = createRawStringObject(decoded->ptr, sdslen(decoded->ptr)); decrRefCount(decoded); - dbReplaceValueWithDictEntry(db,key,o,de); + dbReplaceValueWithLink(db, key, &o, link); } return o; } @@ -710,7 +789,7 @@ redisDb *initTempDb(void) { redisDb *tempDb = zcalloc(sizeof(redisDb)*server.dbnum); for (int i=0; iargc; j++) { - if (expireIfNeeded(c->db,c->argv[j],0) == KEY_DELETED) + if (expireIfNeeded(c->db, c->argv[j], NULL, 0) == KEY_DELETED) continue; int deleted = lazy ? dbAsyncDelete(c->db,c->argv[j]) : dbSyncDelete(c->db,c->argv[j]); @@ -1069,13 +1148,13 @@ void keysCommand(client *c) { } else { kvs_it = kvstoreIteratorInit(c->db->keys); } - robj keyobj; + while ((de = kvs_di ? kvstoreDictIteratorNext(kvs_di) : kvstoreIteratorNext(kvs_it)) != NULL) { - sds key = dictGetKey(de); + kvobj *kv = dictGetKV(de); + sds key = kvobjGetKey(kv); if (allkeys || stringmatchlen(pattern,plen,key,sdslen(key),0)) { - initStaticStringObject(keyobj, key); - if (!keyIsExpired(c->db, &keyobj)) { + if (!keyIsExpired(c->db, NULL, kv)) { addReplyBulkCBuffer(c, key, sdslen(key)); numkeys++; } @@ -1118,26 +1197,31 @@ int objectTypeCompare(robj *o, long long target) { } /* This callback is used by scanGenericCommand in order to collect elements * returned by the dictionary iterator into a list. */ -void scanCallback(void *privdata, const dictEntry *de) { +void scanCallback(void *privdata, const dictEntry *de, dictEntryLink plink) { + UNUSED(plink); scanData *data = (scanData *)privdata; list *keys = data->keys; robj *o = data->o; sds val = NULL; void *key = NULL; /* if OBJ_HASH then key is of type `hfield`. Otherwise, `sds` */ + void *keyStr; data->sampled++; /* o and typename can not have values at the same time. */ serverAssert(!((data->type != LLONG_MAX) && o)); - /* Filter an element if it isn't the type we want. */ - /* TODO: uncomment in redis 8.0 - if (!o && data->type != LLONG_MAX) { - robj *rval = dictGetVal(de); - if (!objectTypeCompare(rval, data->type)) return; - }*/ + if (!o) { /* If scanning keyspace */ + kvobj *kv = dictGetKV(de); + /* scan filter an element if it isn't the type we want. */ + /* TODO: uncomment in redis 8.0 + if (data->type != LLONG_MAX) + if (!objectTypeCompare(kv, data->type)) return;*/ + keyStr = kvobjGetKey(kv); + } else { + keyStr = dictGetKey(de); + } /* Filter element if it does not match the pattern. */ - void *keyStr = dictGetKey(de); if (data->pattern) { if (!stringmatchlen(data->pattern, sdslen(data->pattern), keyStr, data->strlen(keyStr), 0)) { return; @@ -1532,13 +1616,13 @@ void scanGenericCommand(client *c, robj *o, unsigned long long cursor) { /* Filter an element if it isn't the type we want. */ /* TODO: remove this in redis 8.0 */ if (typename) { - robj* typecheck = lookupKeyReadWithFlags(c->db, &kobj, LOOKUP_NOTOUCH|LOOKUP_NONOTIFY); - if (!typecheck || !objectTypeCompare(typecheck, type)) { + kvobj* kv = lookupKeyReadWithFlags(c->db, &kobj, LOOKUP_NOTOUCH|LOOKUP_NONOTIFY); + if (!kv || !objectTypeCompare(kv, type)) { listDelNode(keys, ln); } continue; } - if (expireIfNeeded(c->db, &kobj, 0) != KEY_VALID) { + if (expireIfNeeded(c->db, &kobj, NULL, 0) != KEY_VALID) { listDelNode(keys, ln); } } @@ -1578,9 +1662,8 @@ void lastsaveCommand(client *c) { } void typeCommand(client *c) { - robj *o; - o = lookupKeyReadWithFlags(c->db,c->argv[1],LOOKUP_NOTOUCH); - addReplyStatus(c, getObjectTypeName(o)); + kvobj *kv = lookupKeyReadWithFlags(c->db,c->argv[1],LOOKUP_NOTOUCH); + addReplyStatus(c, getObjectTypeName(kv)); } void shutdownCommand(client *c) { @@ -1645,7 +1728,7 @@ void shutdownCommand(client *c) { } void renameGenericCommand(client *c, int nx) { - robj *o; + kvobj *o; long long expire; int samekey = 0; uint64_t minHashExpireTime = EB_EXPIRE_TIME_INVALID; @@ -1663,7 +1746,7 @@ void renameGenericCommand(client *c, int nx) { } incrRefCount(o); - expire = getExpire(c->db,c->argv[1]); + expire = kvobjGetExpire(o); if (lookupKeyWrite(c->db,c->argv[2]) != NULL) { if (nx) { decrRefCount(o); @@ -1674,8 +1757,6 @@ void renameGenericCommand(client *c, int nx) { * with the same name. */ dbDelete(c->db,c->argv[2]); } - dictEntry *de = dbAdd(c->db, c->argv[2], o); - if (expire != -1) setExpire(c,c->db,c->argv[2],expire); /* If hash with expiration on fields then remove it from global HFE DS and * keep next expiration time. Otherwise, dbDelete() will remove it from the @@ -1684,10 +1765,12 @@ void renameGenericCommand(client *c, int nx) { minHashExpireTime = hashTypeRemoveFromExpires(&c->db->hexpires, o); dbDelete(c->db,c->argv[1]); + dbAdd(c->db, c->argv[2], &o); + if (expire != -1) setExpire(c, c->db, c->argv[2], expire); /* If hash with HFEs, register in db->hexpires */ if (minHashExpireTime != EB_EXPIRE_TIME_INVALID) - hashTypeAddToExpires(c->db, dictGetKey(de), o, minHashExpireTime); + hashTypeAddToExpires(c->db, o, minHashExpireTime); signalModifiedKey(c,c->db,c->argv[1]); signalModifiedKey(c,c->db,c->argv[2]); @@ -1708,7 +1791,6 @@ void renamenxCommand(client *c) { } void moveCommand(client *c) { - robj *o; redisDb *src, *dst; int srcid, dbid; long long expire; @@ -1744,36 +1826,37 @@ void moveCommand(client *c) { server.stat_cluster_incompatible_ops++; /* Check if the element exists and get a reference */ - o = lookupKeyWrite(c->db,c->argv[1]); - if (!o) { + kvobj *kv = lookupKeyWrite(c->db,c->argv[1]); + if (!kv) { addReply(c,shared.czero); return; } - expire = getExpire(c->db,c->argv[1]); + expire = kvobjGetExpire(kv); /* Return zero if the key already exists in the target DB */ - if (lookupKeyWrite(dst,c->argv[1]) != NULL) { + dictEntryLink dstBucket; + if (lookupKey(dst, c->argv[1], LOOKUP_WRITE, &dstBucket) != NULL) { addReply(c,shared.czero); return; } - dictEntry *dstDictEntry = dbAdd(dst,c->argv[1],o); - if (expire != -1) setExpire(c,dst,c->argv[1],expire); /* If hash with expiration on fields, remove it from global HFE DS and keep - * aside registered expiration time. Must be before deletion of the object. - * hexpires (ebuckets) embed in stored items its structure. */ - if (o->type == OBJ_HASH) - hashExpireTime = hashTypeRemoveFromExpires(&src->hexpires, o); + * aside registered expiration time. Must be before addition/deletion of the + * object. hexpires (ebuckets) embed in stored items its structure. */ + if (kv->type == OBJ_HASH) + hashExpireTime = hashTypeRemoveFromExpires(&src->hexpires, kv); - incrRefCount(o); + incrRefCount(kv); /* ref counter = 1->2 */ + dbDelete(src,c->argv[1]); /* ref counter = 2->1 */ - /* OK! key moved, free the entry in the source DB */ - dbDelete(src,c->argv[1]); + dbAddByLink(dst, c->argv[1], &kv, &dstBucket); + if (expire != -1) + setExpireByLink(c, dst, c->argv[1]->ptr, expire, dstBucket); /* If object of type hash with expiration on fields. Taken care to add the * hash to hexpires of `dst` only after dbDelete(). */ if (hashExpireTime != EB_EXPIRE_TIME_INVALID) - hashTypeAddToExpires(dst, dictGetKey(dstDictEntry), o, hashExpireTime); + hashTypeAddToExpires(dst, kv, hashExpireTime); signalModifiedKey(c,src,c->argv[1]); signalModifiedKey(c,dst,c->argv[1]); @@ -1787,7 +1870,7 @@ void moveCommand(client *c) { } void copyCommand(client *c) { - robj *o; + kvobj *o; redisDb *src, *dst; int srcid, dbid; long long expire; @@ -1846,7 +1929,7 @@ void copyCommand(client *c) { addReply(c,shared.czero); return; } - expire = getExpire(c->db,key); + expire = kvobjGetExpire(o); /* Return zero if the key already exists in the target DB. * If REPLACE option is selected, delete newkey from targetDB. */ @@ -1867,7 +1950,7 @@ void copyCommand(client *c) { case OBJ_LIST: newobj = listTypeDup(o); break; case OBJ_SET: newobj = setTypeDup(o); break; case OBJ_ZSET: newobj = zsetDup(o); break; - case OBJ_HASH: newobj = hashTypeDup(o, newkey->ptr, &minHashExpire); break; + case OBJ_HASH: newobj = hashTypeDup(o, &minHashExpire); break; case OBJ_STREAM: newobj = streamDup(o); break; case OBJ_MODULE: newobj = moduleTypeDupOrReply(c, key, newkey, dst->id, o); @@ -1882,7 +1965,7 @@ void copyCommand(client *c) { dbDelete(dst,newkey); } - dictEntry *deCopy = dbAdd(dst,newkey,newobj); + kvobj *kvCopy = dbAdd(dst, newkey, &newobj); /* if key with expiration then set it */ if (expire != -1) @@ -1891,7 +1974,7 @@ void copyCommand(client *c) { /* If minExpiredField was set, then the object is hash with expiration * on fields and need to register it in global HFE DS */ if (minHashExpire != EB_EXPIRE_TIME_INVALID) - hashTypeAddToExpires(dst, dictGetKey(deCopy), newobj, minHashExpire); + hashTypeAddToExpires(dst, kvCopy, minHashExpire); /* OK! key copied */ signalModifiedKey(c,dst,c->argv[2]); @@ -1910,11 +1993,9 @@ void scanDatabaseForReadyKeys(redisDb *db) { dictIterator *di = dictGetSafeIterator(db->blocking_keys); while((de = dictNext(di)) != NULL) { robj *key = dictGetKey(de); - dictEntry *kde = dbFind(db, key->ptr); - if (kde) { - robj *value = dictGetVal(kde); - signalKeyAsReady(db, key, value->type); - } + kvobj *kv = dbFind(db, key->ptr); + if (kv) + signalKeyAsReady(db, key, kv->type); } dictReleaseIterator(di); } @@ -1930,18 +2011,16 @@ void scanDatabaseForDeletedKeys(redisDb *emptied, redisDb *replaced_with) { int existed = 0, exists = 0; int original_type = -1, curr_type = -1; - dictEntry *kde = dbFind(emptied, key->ptr); - if (kde) { - robj *value = dictGetVal(kde); - original_type = value->type; + kvobj *kv = dbFind(emptied, key->ptr); + if (kv) { + original_type = kv->type; existed = 1; } if (replaced_with) { - kde = dbFind(replaced_with, key->ptr); - if (kde) { - robj *value = dictGetVal(kde); - curr_type = value->type; + kv = dbFind(replaced_with, key->ptr); + if (kv) { + curr_type = kv->type; exists = 1; } } @@ -2087,8 +2166,22 @@ void swapdbCommand(client *c) { * Expires API *----------------------------------------------------------------------------*/ +/* Remove expiry from key + * + * Remove the object from db->expires and set to -1 attached TTL to KV + */ int removeExpire(redisDb *db, robj *key) { - return kvstoreDictDelete(db->expires, getKeySlot(key->ptr), key->ptr) == DICT_OK; + int table; + int slot = getKeySlot(key->ptr); + dictEntryLink link = kvstoreDictTwoPhaseUnlinkFind(db->expires, slot, key->ptr, &table); + + if (link == NULL) return 0; + dictEntry *de = *link; + kvobj *kv = dictGetKV(de); + kvobj *newkv = kvobjSetExpire(kv, -1); + serverAssert(newkv == kv); + kvstoreDictTwoPhaseUnlinkFree(db->expires, slot, link, table); + return 1; } @@ -2096,54 +2189,63 @@ int removeExpire(redisDb *db, robj *key) { * of an user calling a command 'c' is the client, otherwise 'c' is set * to NULL. The 'when' parameter is the absolute unix time in milliseconds * after which the key will no longer be considered valid. */ -void setExpire(client *c, redisDb *db, robj *key, long long when) { - setExpireWithDictEntry(c,db,key,when,NULL); +kvobj *setExpire(client *c, redisDb *db, robj *key, long long when) { + return setExpireByLink(c,db,key->ptr,when,NULL); } -/* Like setExpire(), but accepts an optional dictEntry input, - * which can be used if we already have one, thus saving the kvstoreDictFind call. */ -void setExpireWithDictEntry(client *c, redisDb *db, robj *key, long long when, dictEntry *kde) { - dictEntry *de, *existing; - +/* Like setExpire(), but accepts an optional `keyLink` to save lookup */ +kvobj *setExpireByLink(client *c, redisDb *db, sds key, long long when, dictEntryLink keyLink) { /* Reuse the sds from the main dict in the expire dict */ - int slot = getKeySlot(key->ptr); - if (!kde) kde = kvstoreDictFind(db->keys, slot, key->ptr); - serverAssertWithInfo(NULL,key,kde != NULL); - de = kvstoreDictAddRaw(db->expires, slot, dictGetKey(kde), &existing); - if (existing) { - dictSetSignedIntegerVal(existing, when); - } else { - dictSetSignedIntegerVal(de, when); + int slot = getKeySlot(key); + if (!keyLink) { + keyLink = kvstoreDictFindLink(db->keys, slot, key, NULL); + serverAssert(keyLink != NULL); + } + kvobj *kv = dictGetKV(*keyLink); + long long old_when = kvobjGetExpire(kv); + + if (old_when != -1) { /* old expire */ + kvobj *kvnew = kvobjSetExpire(kv, when); /* release kv if reallocated */ + /* Val already had an expire field, so it was not reallocated. */ + serverAssert(kv == kvnew); + } else { /* No old expire */ + uint64_t hexpire = EB_EXPIRE_TIME_INVALID; + /* If hash with HFEs, take care to remove from global HFE DS before attempting + * to manipulate and maybe free kv object */ + if (kv->type == OBJ_HASH) + hexpire = hashTypeRemoveFromExpires(&db->hexpires, kv); + + kvobj *kvnew = kvobjSetExpire(kv, when); /* release kv if reallocated */ + /* if kvobj was reallocated, update dict */ + if (kv != kvnew) { + kvstoreDictSetAtLink(db->keys, slot, kvnew, &keyLink, 0); + kv = kvnew; + } + /* Now add to expires */ + dictEntry *de = kvstoreDictAddRaw(db->expires, slot, kv, NULL); + serverAssert(de != NULL); + + if (hexpire != EB_EXPIRE_TIME_INVALID) + hashTypeAddToExpires(db, kv, hexpire); } int writable_slave = server.masterhost && server.repl_slave_ro == 0; if (c && writable_slave && !(c->flags & CLIENT_MASTER)) rememberSlaveKeyWithExpire(db,key); + return kv; } -/* Return the expire time of the specified key, or -1 if no expire - * is associated with this key (i.e. the key is non volatile) */ -static inline long long getExpireWithSlot(redisDb *db, robj *key, int keySlot) { - dictEntry *de; - - if ((de = dbFindExpiresWithKeySlot(db, key->ptr, keySlot)) == NULL) - return -1; - - return dictGetSignedIntegerVal(de); +/* Retrieve the expiration time for the specified key. + * Returns -1 if the key has no expiration set or doesn't exists + * + * To avoid lookup, pass key-value object (`kv`) instead of `key`. + */ +long long getExpire(redisDb *db, sds key, kvobj *kv) { + if (kv == NULL) kv = dbFindExpires(db, key); + if (kv == NULL) return -1; + return kvobjGetExpire(kv); } -/* Return the expire time of the specified key, or -1 if no expire - * is associated with this key (i.e. the key is non volatile) */ -long long getExpire(redisDb *db, robj *key) { - dictEntry *de; - - if ((de = dbFindExpires(db, key->ptr)) == NULL) - return -1; - - return dictGetSignedIntegerVal(de); -} - - /* Delete the specified expired or evicted key and propagate to replicas. * Currently notify_type can only be NOTIFY_EXPIRED or NOTIFY_EVICTED, * and it affects other aspects like the latency monitor event name and, @@ -2249,10 +2351,14 @@ void propagateDeletion(redisDb *db, robj *key, int lazy) { decrRefCount(argv[1]); } -/* Internal Check if the key is expired based upon mstime_t. */ -static inline int keyIsExpiredInternal(mstime_t when) { +/* Check if the key is expired + * + * Provide either the key name for a lookup or KV object (to save lookup) + */ +int keyIsExpired(redisDb *db, sds key, kvobj *kv) { /* Don't expire anything while loading. It will be done later. */ if (server.loading) return 0; + mstime_t when = getExpire(db, key, kv); if (when < 0) return 0; /* No expire for this key */ const mstime_t now = commandTimeSnapshot(); /* The key expired if the current (virtual or real) time is greater @@ -2260,16 +2366,6 @@ static inline int keyIsExpiredInternal(mstime_t when) { return now > when; } -/* Check if the key is expired. */ -static inline int keyIsExpiredWithSlot(redisDb *db, robj *key, int keySlot) { - return keyIsExpiredInternal(getExpireWithSlot(db,key,keySlot)); -} - -/* Check if the key is expired. */ -int keyIsExpired(redisDb *db, robj *key) { - return keyIsExpiredInternal(getExpire(db,key)); -} - /* This function is called when we are going to perform some operation * in a given key, but such key may be already logically expired even if * it still exists in the database. The main way this function is called @@ -2300,15 +2396,16 @@ int keyIsExpired(redisDb *db, robj *key) { * * The return value of the function is KEY_VALID if the key is still valid. * The function returns KEY_EXPIRED if the key is expired BUT not deleted, - * or returns KEY_DELETED if the key is expired and deleted. */ -keyStatus expireIfNeeded(redisDb *db, robj *key, int flags) { - return expireIfNeededWithSlot(db,key,flags,getKeySlot(key->ptr)); -} - -static inline keyStatus expireIfNeededWithSlot(redisDb *db, robj *key, int flags, const int keySlot) { + * or returns KEY_DELETED if the key is expired and deleted. + * + * You can optionally pass `kv` to save a lookup. + */ +keyStatus expireIfNeeded(redisDb *db, robj *key, kvobj *kv, int flags) { + serverAssert(key != NULL); + sds keyname = key->ptr; if ((server.allow_access_expired) || (flags & EXPIRE_ALLOW_ACCESS_EXPIRED) || - (!keyIsExpiredWithSlot(db,key,keySlot))) + (!keyIsExpired(db, keyname, kv))) return KEY_VALID; /* If we are running in the context of a replica, instead of @@ -2386,27 +2483,36 @@ int dbExpandExpires(redisDb *db, uint64_t db_size, int try_expand) { return dbExpandGeneric(db->expires, db_size, try_expand); } -static inline dictEntry *dbFindGenericWithKeySlot(kvstore *kvs, void *key, int keySlot) { - return kvstoreDictFind(kvs, keySlot, key); +static kvobj *dbFindGeneric(kvstore *kvs, sds key) { + dictEntry *res = kvstoreDictFind(kvs, getKeySlot(key), key); + return (res) ? dictGetKey(res) : NULL; } -static dictEntry *dbFindGeneric(kvstore *kvs, void *key) { - return kvstoreDictFind(kvs, getKeySlot(key), key); -} - -dictEntry *dbFind(redisDb *db, void *key) { +kvobj *dbFind(redisDb *db, sds key) { return dbFindGeneric(db->keys, key); } -static inline dictEntry *dbFindWithKeySlot(redisDb *db, void *key, int keySlot) { - return dbFindGenericWithKeySlot(db->keys, key, keySlot); +/* Find a KV in the main db. Return also link to it. + * + * plink - If found, set to the link of the key in the dict. + * If not found, set to the bucket where the key should be added. + * If set to NULL, then HT of dict not allocated yet. + */ +kvobj *dbFindByLink(redisDb *db, sds key, dictEntryLink *plink) { + int slot = getKeySlot(key); + dictEntryLink link, bucket; + + link = kvstoreDictFindLink(db->keys, slot, key, &bucket); + if (link == NULL) { + if (plink) *plink = bucket; + return NULL; + } else { + if (plink) *plink = link; + return dictGetKV(*link); + } } -static inline dictEntry *dbFindExpiresWithKeySlot(redisDb *db, void *key, int keySlot) { - return dbFindGenericWithKeySlot(db->expires, key, keySlot); -} - -dictEntry *dbFindExpires(redisDb *db, void *key) { +kvobj *dbFindExpires(redisDb *db, sds key) { return dbFindGeneric(db->expires, key); } diff --git a/src/debug.c b/src/debug.c index 30df823f8..7f49ab3c3 100644 --- a/src/debug.c +++ b/src/debug.c @@ -131,7 +131,7 @@ void mixStringObjectDigest(unsigned char *digest, robj *o) { void xorObjectDigest(redisDb *db, robj *keyobj, unsigned char *digest, robj *o) { uint32_t aux = htonl(o->type); mixDigest(digest,&aux,sizeof(aux)); - long long expiretime = getExpire(db,keyobj); + long long expiretime = getExpire(db, keyobj->ptr, NULL); char buf[128]; /* Save the key and associated value */ @@ -290,17 +290,16 @@ void computeDatasetDigest(unsigned char *final) { /* Iterate this DB writing every entry */ while((de = kvstoreIteratorNext(kvs_it)) != NULL) { - sds key; - robj *keyobj, *o; + robj *keyobj; memset(digest,0,20); /* This key-val digest */ - key = dictGetKey(de); + kvobj *kv = dictGetKV(de); + sds key = kvobjGetKey(kv); keyobj = createStringObject(key,sdslen(key)); mixDigest(digest,key,sdslen(key)); - o = dictGetVal(de); - xorObjectDigest(db,keyobj,digest,o); + xorObjectDigest(db, keyobj, digest, kv); /* We can finally xor the key-val digest to the final digest */ xorDigest(final,digest,20); @@ -605,22 +604,21 @@ NULL server.cluster_drop_packet_filter = packet_type; addReply(c,shared.ok); } else if (!strcasecmp(c->argv[1]->ptr,"object") && c->argc == 3) { - dictEntry *de; - robj *val; + kvobj *kv; char *strenc; - if ((de = dbFind(c->db, c->argv[2]->ptr)) == NULL) { + if ((kv = dbFind(c->db, c->argv[2]->ptr)) == NULL) { addReplyErrorObject(c,shared.nokeyerr); return; } - val = dictGetVal(de); - strenc = strEncoding(val->encoding); + + strenc = strEncoding(kv->encoding); char extra[138] = {0}; - if (val->encoding == OBJ_ENCODING_QUICKLIST) { + if (kv->encoding == OBJ_ENCODING_QUICKLIST) { char *nextra = extra; int remaining = sizeof(extra); - quicklist *ql = val->ptr; + quicklist *ql = kv->ptr; /* Add number of quicklist nodes */ int used = snprintf(nextra, remaining, " ql_nodes:%lu", ql->len); nextra += used; @@ -653,38 +651,42 @@ NULL "Value at:%p refcount:%d " "encoding:%s serializedlength:%zu " "lru:%d lru_seconds_idle:%llu%s", - (void*)val, val->refcount, - strenc, rdbSavedObjectLen(val, c->argv[2], c->db->id), - val->lru, estimateObjectIdleTime(val)/1000, extra); + (void*)kv, kv->refcount, + strenc, rdbSavedObjectLen(kv, c->argv[2], c->db->id), + kv->lru, estimateObjectIdleTime(kv)/1000, extra); } else if (!strcasecmp(c->argv[1]->ptr,"sdslen") && c->argc == 3) { - dictEntry *de; robj *val; sds key; + kvobj *kv; - if ((de = dbFind(c->db, c->argv[2]->ptr)) == NULL) { + if ((kv = dbFind(c->db, c->argv[2]->ptr)) == NULL) { addReplyErrorObject(c,shared.nokeyerr); return; } - val = dictGetVal(de); - key = dictGetKey(de); - - if (val->type != OBJ_STRING || !sdsEncodedObject(val)) { + + val = kv; + key = kvobjGetKey(kv); + if (kv->type != OBJ_STRING || !sdsEncodedObject(val)) { addReplyError(c,"Not an sds encoded string."); } else { + /* The key's allocation size reflects the entire robj allocation. + * For embedded values, report an allocation size of 0. */ + size_t obj_alloc = zmalloc_usable_size(val); + size_t val_alloc = val->encoding == OBJ_ENCODING_RAW ? sdsAllocSize(val->ptr) : 0; addReplyStatusFormat(c, "key_sds_len:%lld, key_sds_avail:%lld, key_zmalloc: %lld, " "val_sds_len:%lld, val_sds_avail:%lld, val_zmalloc: %lld", (long long) sdslen(key), (long long) sdsavail(key), - (long long) sdsZmallocSize(key), + (long long) obj_alloc, (long long) sdslen(val->ptr), (long long) sdsavail(val->ptr), - (long long) getStringObjectSdsUsedMemory(val)); + (long long) val_alloc); } } else if (!strcasecmp(c->argv[1]->ptr,"listpack") && c->argc == 3) { - robj *o; + kvobj *o; - if ((o = objectCommandLookupOrReply(c,c->argv[2],shared.nokeyerr)) + if ((o = kvobjCommandLookupOrReply(c, c->argv[2], shared.nokeyerr)) == NULL) return; if (o->encoding != OBJ_ENCODING_LISTPACK && o->encoding != OBJ_ENCODING_LISTPACK_EX) { @@ -698,9 +700,9 @@ NULL addReplyStatus(c,"Listpack structure printed on stdout"); } } else if (!strcasecmp(c->argv[1]->ptr,"quicklist") && (c->argc == 3 || c->argc == 4)) { - robj *o; + kvobj *o; - if ((o = objectCommandLookupOrReply(c,c->argv[2],shared.nokeyerr)) + if ((o = kvobjCommandLookupOrReply(c, c->argv[2], shared.nokeyerr)) == NULL) return; int full = 0; @@ -750,7 +752,7 @@ NULL val = createStringObject(NULL,valsize); memcpy(val->ptr, buf, valsize<=buflen? valsize: buflen); } - dbAdd(c->db,key,val); + dbAdd(c->db, key, &val); signalModifiedKey(c,c->db,key); decrRefCount(key); } @@ -782,8 +784,7 @@ NULL /* We don't use lookupKey because a debug command should * work on logically expired keys */ - dictEntry *de; - robj *o = ((de = dbFind(c->db, c->argv[j]->ptr)) == NULL) ? NULL : dictGetVal(de); + kvobj *o = dbFind(c->db, c->argv[j]->ptr); if (o) xorObjectDigest(c->db,c->argv[j],digest,o); sds d = sdsempty(); @@ -901,7 +902,7 @@ NULL sds sizes = sdsempty(); sizes = sdscatprintf(sizes,"bits:%d ",(sizeof(void*) == 8)?64:32); sizes = sdscatprintf(sizes,"robj:%d ",(int)sizeof(robj)); - sizes = sdscatprintf(sizes,"dictentry:%d ",(int)dictEntryMemUsage()); + sizes = sdscatprintf(sizes,"dictentry:%d ",(int)dictEntryMemUsage(0)); sizes = sdscatprintf(sizes,"sdshdr5:%d ",(int)sizeof(struct sdshdr5)); sizes = sdscatprintf(sizes,"sdshdr8:%d ",(int)sizeof(struct sdshdr8)); sizes = sdscatprintf(sizes,"sdshdr16:%d ",(int)sizeof(struct sdshdr16)); @@ -937,14 +938,14 @@ NULL addReplyVerbatim(c,stats,sdslen(stats),"txt"); sdsfree(stats); } else if (!strcasecmp(c->argv[1]->ptr,"htstats-key") && c->argc >= 3) { - robj *o; + kvobj *o; dict *ht = NULL; int full = 0; if (c->argc >= 4 && !strcasecmp(c->argv[3]->ptr,"full")) full = 1; - if ((o = objectCommandLookupOrReply(c,c->argv[2],shared.nokeyerr)) + if ((o = kvobjCommandLookupOrReply(c,c->argv[2],shared.nokeyerr)) == NULL) return; /* Get the hash table reference from the object, if possible. */ @@ -2190,15 +2191,11 @@ void logCurrentClient(client *cc, const char *title) { /* Check if the first argument, usually a key, is found inside the * selected DB, and if so print info about the associated object. */ if (cc->argc > 1) { - robj *val, *key; - dictEntry *de; - - key = getDecodedObject(cc->argv[1]); - de = dbFind(cc->db, key->ptr); - if (de) { - val = dictGetVal(de); + robj *key = getDecodedObject(cc->argv[1]); + kvobj *kv = dbFind(cc->db, key->ptr); + if (kv) { serverLog(LL_WARNING,"key '%s' found in DB containing the following object:", (char*)key->ptr); - serverLogObjectDebugInfo(val); + serverLogObjectDebugInfo(kv); } decrRefCount(key); } diff --git a/src/defrag.c b/src/defrag.c index 56c348c33..55d67af0c 100644 --- a/src/defrag.c +++ b/src/defrag.c @@ -375,12 +375,14 @@ void activeDefragZsetEntry(zset *zs, dictEntry *de) { #define DEFRAG_SDS_DICT_VAL_VOID_PTR 3 #define DEFRAG_SDS_DICT_VAL_LUA_SCRIPT 4 -void activeDefragSdsDictCallback(void *privdata, const dictEntry *de) { +void activeDefragSdsDictCallback(void *privdata, const dictEntry *de, dictEntryLink plink) { + UNUSED(plink); UNUSED(privdata); UNUSED(de); } -void activeDefragHfieldDictCallback(void *privdata, const dictEntry *de) { +void activeDefragHfieldDictCallback(void *privdata, const dictEntry *de, dictEntryLink plink) { + UNUSED(plink); dict *d = privdata; hfield newhf, hf = dictGetKey(de); @@ -392,17 +394,8 @@ void activeDefragHfieldDictCallback(void *privdata, const dictEntry *de) { ebuckets *eb = hashTypeGetDictMetaHFE(d); newhf = ebDefragItem(eb, &hashFieldExpireBucketsType, hf, (ebDefragFunction *)activeDefragHfield); } - if (newhf) { - /* We can't search in dict for that key after we've released - * the pointer it holds, since it won't be able to do the string - * compare, but we can find the entry using key hash and pointer. */ - dictUseStoredKeyApi(d, 1); - uint64_t hash = dictGetHash(d, newhf); - dictUseStoredKeyApi(d, 0); - dictEntry *de = dictFindByHashAndPtr(d, hf, hash); - serverAssert(de); - dictSetKey(d, de, newhf); - } + + if (newhf) dictSetKey(d, (dictEntry *) de, newhf); } /* Defrag a dict with sds key and optional value (either ptr, sds or robj string) */ @@ -467,13 +460,13 @@ void activeDefragQuickListNodes(quicklist *ql) { /* when the value has lots of elements, we want to handle it later and not as * part of the main dictionary scan. this is needed in order to prevent latency * spikes when handling large items */ -void defragLater(defragKeysCtx *ctx, dictEntry *kde) { +void defragLater(defragKeysCtx *ctx, kvobj *kv) { if (!ctx->defrag_later) { ctx->defrag_later = listCreate(); listSetFreeMethod(ctx->defrag_later, sdsfreegeneric); ctx->defrag_later_cursor = 0; } - sds key = sdsdup(dictGetKey(kde)); + sds key = sdsdup(kvobjGetKey(kv)); listAddNodeTail(ctx->defrag_later, key); } @@ -524,7 +517,8 @@ typedef struct { zset *zs; } scanLaterZsetData; -void scanLaterZsetCallback(void *privdata, const dictEntry *_de) { +void scanLaterZsetCallback(void *privdata, const dictEntry *_de, dictEntryLink plink) { + UNUSED(plink); dictEntry *de = (dictEntry*)_de; scanLaterZsetData *data = privdata; activeDefragZsetEntry(data->zs, de); @@ -541,7 +535,8 @@ void scanLaterZset(robj *ob, unsigned long *cursor) { } /* Used as scan callback when all the work is done in the dictDefragFunctions. */ -void scanCallbackCountScanned(void *privdata, const dictEntry *de) { +void scanCallbackCountScanned(void *privdata, const dictEntry *de, dictEntryLink plink) { + UNUSED(plink); UNUSED(privdata); UNUSED(de); server.stat_active_defrag_scanned++; @@ -568,20 +563,18 @@ void scanLaterHash(robj *ob, unsigned long *cursor) { *cursor = dictScanDefrag(d, *cursor, activeDefragHfieldDictCallback, &defragfns, d); } -void defragQuicklist(defragKeysCtx *ctx, dictEntry *kde) { - robj *ob = dictGetVal(kde); - quicklist *ql = ob->ptr, *newql; - serverAssert(ob->type == OBJ_LIST && ob->encoding == OBJ_ENCODING_QUICKLIST); +void defragQuicklist(defragKeysCtx *ctx, kvobj *kv) { + quicklist *ql = kv->ptr, *newql; + serverAssert(kv->type == OBJ_LIST && kv->encoding == OBJ_ENCODING_QUICKLIST); if ((newql = activeDefragAlloc(ql))) - ob->ptr = ql = newql; + kv->ptr = ql = newql; if (ql->len > server.active_defrag_max_scan_fields) - defragLater(ctx, kde); + defragLater(ctx, kv); else activeDefragQuickListNodes(ql); } -void defragZsetSkiplist(defragKeysCtx *ctx, dictEntry *kde) { - robj *ob = dictGetVal(kde); +void defragZsetSkiplist(defragKeysCtx *ctx, kvobj *ob) { zset *zs = (zset*)ob->ptr; zset *newzs; zskiplist *newzsl; @@ -596,7 +589,7 @@ void defragZsetSkiplist(defragKeysCtx *ctx, dictEntry *kde) { if ((newheader = activeDefragAlloc(zs->zsl->header))) zs->zsl->header = newheader; if (dictSize(zs->dict) > server.active_defrag_max_scan_fields) - defragLater(ctx, kde); + defragLater(ctx, ob); else { dictIterator *di = dictGetIterator(zs->dict); while((de = dictNext(di)) != NULL) { @@ -609,13 +602,12 @@ void defragZsetSkiplist(defragKeysCtx *ctx, dictEntry *kde) { zs->dict = newdict; } -void defragHash(defragKeysCtx *ctx, dictEntry *kde) { - robj *ob = dictGetVal(kde); +void defragHash(defragKeysCtx *ctx, kvobj *ob) { dict *d, *newd; serverAssert(ob->type == OBJ_HASH && ob->encoding == OBJ_ENCODING_HT); d = ob->ptr; if (dictSize(d) > server.active_defrag_max_scan_fields) - defragLater(ctx, kde); + defragLater(ctx, ob); else activeDefragHfieldDict(d); /* defrag the dict struct and tables */ @@ -623,13 +615,12 @@ void defragHash(defragKeysCtx *ctx, dictEntry *kde) { ob->ptr = newd; } -void defragSet(defragKeysCtx *ctx, dictEntry *kde) { - robj *ob = dictGetVal(kde); +void defragSet(defragKeysCtx *ctx, kvobj *ob) { dict *d, *newd; serverAssert(ob->type == OBJ_SET && ob->encoding == OBJ_ENCODING_HT); d = ob->ptr; if (dictSize(d) > server.active_defrag_max_scan_fields) - defragLater(ctx, kde); + defragLater(ctx, ob); else activeDefragSdsDict(d, DEFRAG_SDS_DICT_NO_VAL); /* defrag the dict struct and tables */ @@ -783,8 +774,7 @@ void* defragStreamConsumerGroup(raxIterator *ri, void *privdata) { return NULL; } -void defragStream(defragKeysCtx *ctx, dictEntry *kde) { - robj *ob = dictGetVal(kde); +void defragStream(defragKeysCtx *ctx, kvobj *ob) { serverAssert(ob->type == OBJ_STREAM && ob->encoding == OBJ_ENCODING_STREAM); stream *s = ob->ptr, *news; @@ -796,7 +786,7 @@ void defragStream(defragKeysCtx *ctx, dictEntry *kde) { rax *newrax = activeDefragAlloc(s->rax); if (newrax) s->rax = newrax; - defragLater(ctx, kde); + defragLater(ctx, ob); } else defragRadixTree(&s->rax, 1, NULL, NULL); @@ -807,62 +797,53 @@ void defragStream(defragKeysCtx *ctx, dictEntry *kde) { /* Defrag a module key. This is either done immediately or scheduled * for later. Returns then number of pointers defragged. */ -void defragModule(defragKeysCtx *ctx, redisDb *db, dictEntry *kde) { - robj *obj = dictGetVal(kde); - serverAssert(obj->type == OBJ_MODULE); +void defragModule(defragKeysCtx *ctx, redisDb *db, kvobj *kv) { + serverAssert(kv->type == OBJ_MODULE); robj keyobj; - initStaticStringObject(keyobj, dictGetKey(kde)); - if (!moduleDefragValue(&keyobj, obj, db->id)) - defragLater(ctx, kde); + initStaticStringObject(keyobj, kvobjGetKey(kv)); + if (!moduleDefragValue(&keyobj, kv, db->id)) + defragLater(ctx, kv); } /* for each key we scan in the main dict, this function will attempt to defrag * all the various pointers it has. */ -void defragKey(defragKeysCtx *ctx, dictEntry *de) { +void defragKey(defragKeysCtx *ctx, dictEntry *de, dictEntryLink link) { + dictEntryLink exlink = NULL; + kvobj *kvnew, *ob = dictGetKV(de); redisDb *db = &server.db[ctx->dbid]; int slot = ctx->kvstate.slot; - sds keysds = dictGetKey(de); - robj *newob, *ob = dictGetVal(de); unsigned char *newzl; - sds newsds; - - /* Try to defrag the key name. */ - newsds = activeDefragSds(keysds); - if (newsds) { - kvstoreDictSetKey(db->keys, slot, de, newsds); - if (kvstoreDictSize(db->expires, slot)) { - /* We can't search in db->expires for that key after we've released - * the pointer it holds, since it won't be able to do the string - * compare, but we can find the entry using key hash and pointer. */ - uint64_t hash = kvstoreGetHash(db->expires, newsds); - dictEntry *expire_de = kvstoreDictFindByHashAndPtr(db->expires, slot, keysds, hash); - if (expire_de) kvstoreDictSetKey(db->expires, slot, expire_de, newsds); - } - - /* Update the key's reference in the dict's metadata or the listpackEx. */ - if (unlikely(ob->type == OBJ_HASH)) - hashTypeUpdateKeyRef(ob, newsds); - } + + long long expire = kvobjGetExpire(ob); + /* We can't search in db->expires for that KV after we've released + * the pointer it holds, since it won't be able to do the string + * compare. Search it before, if needed. */ + if (expire != -1) { + exlink = kvstoreDictFindLink(db->expires, slot, kvobjGetKey(ob), NULL); + serverAssert(exlink != NULL); + } /* Try to defrag robj and / or string value. */ if (unlikely(ob->type == OBJ_HASH && hashTypeGetMinExpire(ob, 0) != EB_EXPIRE_TIME_INVALID)) { /* Update its reference in the ebucket while defragging it. */ - newob = ebDefragItem(&db->hexpires, &hashExpireBucketsType, ob, + kvnew = ebDefragItem(&db->hexpires, &hashExpireBucketsType, ob, (ebDefragFunction *)activeDefragStringOb); } else { /* If the dict doesn't have metadata, we directly defrag it. */ - newob = activeDefragStringOb(ob); + kvnew = activeDefragStringOb(ob); } - if (newob) { - kvstoreDictSetVal(db->keys, slot, de, newob); - ob = newob; + if (kvnew) { + kvstoreDictSetAtLink(db->keys, slot, kvnew, &link, 0); + if (expire != -1) + kvstoreDictSetAtLink(db->expires, slot, kvnew, &exlink, 0); + ob = kvnew; } if (ob->type == OBJ_STRING) { /* Already handled in activeDefragStringOb. */ } else if (ob->type == OBJ_LIST) { if (ob->encoding == OBJ_ENCODING_QUICKLIST) { - defragQuicklist(ctx, de); + defragQuicklist(ctx, ob); } else if (ob->encoding == OBJ_ENCODING_LISTPACK) { if ((newzl = activeDefragAlloc(ob->ptr))) ob->ptr = newzl; @@ -871,7 +852,7 @@ void defragKey(defragKeysCtx *ctx, dictEntry *de) { } } else if (ob->type == OBJ_SET) { if (ob->encoding == OBJ_ENCODING_HT) { - defragSet(ctx, de); + defragSet(ctx, ob); } else if (ob->encoding == OBJ_ENCODING_INTSET || ob->encoding == OBJ_ENCODING_LISTPACK) { @@ -886,7 +867,7 @@ void defragKey(defragKeysCtx *ctx, dictEntry *de) { if ((newzl = activeDefragAlloc(ob->ptr))) ob->ptr = newzl; } else if (ob->encoding == OBJ_ENCODING_SKIPLIST) { - defragZsetSkiplist(ctx, de); + defragZsetSkiplist(ctx, ob); } else { serverPanic("Unknown sorted set encoding"); } @@ -901,23 +882,23 @@ void defragKey(defragKeysCtx *ctx, dictEntry *de) { if ((newzl = activeDefragAlloc(lpt->lp))) lpt->lp = newzl; } else if (ob->encoding == OBJ_ENCODING_HT) { - defragHash(ctx, de); + defragHash(ctx, ob); } else { serverPanic("Unknown hash encoding"); } } else if (ob->type == OBJ_STREAM) { - defragStream(ctx, de); + defragStream(ctx, ob); } else if (ob->type == OBJ_MODULE) { - defragModule(ctx,db, de); + defragModule(ctx,db, ob); } else { serverPanic("Unknown object type"); } } /* Defrag scan callback for the main db dictionary. */ -static void dbKeysScanCallback(void *privdata, const dictEntry *de) { +static void dbKeysScanCallback(void *privdata, const dictEntry *de, dictEntryLink plink) { long long hits_before = server.stat_active_defrag_hits; - defragKey((defragKeysCtx *)privdata, (dictEntry *)de); + defragKey((defragKeysCtx *)privdata, (dictEntry *)de, plink); if (server.stat_active_defrag_hits != hits_before) server.stat_active_defrag_key_hits++; else @@ -960,7 +941,8 @@ float getAllocatorFragmentation(size_t *out_frag_bytes) { } /* Defrag scan callback for the pubsub dictionary. */ -void defragPubsubScanCallback(void *privdata, const dictEntry *de) { +void defragPubsubScanCallback(void *privdata, const dictEntry *de, dictEntryLink plink) { + UNUSED(plink); defragPubSubCtx *ctx = privdata; kvstore *pubsub_channels = ctx->kvstate.kvs; robj *newchannel, *channel = dictGetKey(de); @@ -996,9 +978,8 @@ void defragPubsubScanCallback(void *privdata, const dictEntry *de) { /* returns 0 more work may or may not be needed (see non-zero cursor), * and 1 if time is up and more work is needed. */ -int defragLaterItem(dictEntry *de, unsigned long *cursor, monotime endtime, int dbid) { - if (de) { - robj *ob = dictGetVal(de); +int defragLaterItem(kvobj *ob, unsigned long *cursor, monotime endtime, int dbid) { + if (ob) { if (ob->type == OBJ_LIST && ob->encoding == OBJ_ENCODING_QUICKLIST) { return scanLaterList(ob, cursor, endtime); } else if (ob->type == OBJ_SET && ob->encoding == OBJ_ENCODING_HT) { @@ -1011,7 +992,7 @@ int defragLaterItem(dictEntry *de, unsigned long *cursor, monotime endtime, int return scanLaterStreamListpacks(ob, cursor, endtime); } else if (ob->type == OBJ_MODULE) { robj keyobj; - initStaticStringObject(keyobj, dictGetKey(de)); + initStaticStringObject(keyobj, kvobjGetKey(ob)); return moduleLateDefrag(&keyobj, ob, cursor, endtime, dbid); } else { *cursor = 0; /* object type/encoding may have changed since we schedule it for later */ @@ -1038,9 +1019,10 @@ static doneStatus defragLaterStep(void *ctx, monotime endtime) { listNode *head = listFirst(defrag_keys_ctx->defrag_later); sds key = head->value; dictEntry *de = kvstoreDictFind(defrag_keys_ctx->kvstate.kvs, defrag_keys_ctx->kvstate.slot, key); + kvobj *kv = de ? dictGetKV(de) : NULL; long long key_defragged = server.stat_active_defrag_hits; - int timeout = (defragLaterItem(de, &defrag_keys_ctx->defrag_later_cursor, endtime, defrag_keys_ctx->dbid) == 1); + int timeout = (defragLaterItem(kv, &defrag_keys_ctx->defrag_later_cursor, endtime, defrag_keys_ctx->dbid) == 1); if (key_defragged != server.stat_active_defrag_hits) { server.stat_active_defrag_key_hits++; } else { diff --git a/src/dict.c b/src/dict.c index 1554ddeb9..25fcde711 100644 --- a/src/dict.c +++ b/src/dict.c @@ -61,20 +61,30 @@ typedef struct { /* -------------------------- private prototypes ---------------------------- */ -static void _dictExpandIfNeeded(dict *d); +static int _dictExpandIfNeeded(dict *d); static void _dictShrinkIfNeeded(dict *d); static void _dictRehashStepIfNeeded(dict *d, uint64_t visitedIdx); static signed char _dictNextExp(unsigned long size); static int _dictInit(dict *d, dictType *type); static dictEntry *dictGetNext(const dictEntry *de); -static dictEntry **dictGetNextRef(dictEntry *de); +static dictEntryLink dictGetNextLink(dictEntry *de); static void dictSetNext(dictEntry *de, dictEntry *next); -static int dictDefaultCompare(dict *d, const void *key1, const void *key2); +static int dictDefaultCompare(dictCmpCache *cache, const void *key1, const void *key2); +static dictEntryLink dictFindLinkInternal(dict *d, const void *key, dictEntryLink *bucket); +dictEntryLink dictFindLinkForInsert(dict *d, const void *key, dictEntry **existing); +static dictEntry *dictInsertKeyAtLink(dict *d, void *key, dictEntryLink link); + +/* -------------------------- unused --------------------------- */ +void dictSetSignedIntegerVal(dictEntry *de, int64_t val); +int64_t dictGetSignedIntegerVal(const dictEntry *de); +double dictIncrDoubleVal(dictEntry *de, double val); +void *dictEntryMetadata(dictEntry *de); +int64_t dictIncrSignedIntegerVal(dictEntry *de, int64_t val); /* -------------------------- misc inline functions -------------------------------- */ -typedef int (*keyCmpFunc)(dict *d, const void *key1, const void *key2); -static inline keyCmpFunc dictGetKeyCmpFunc(dict *d) { +typedef int (*keyCmpFunc)(dictCmpCache *cache, const void *key1, const void *key2); +static inline keyCmpFunc dictGetCmpFunc(dict *d) { if (d->useStoredKeyApi && d->type->storedKeyCompare) return d->type->storedKeyCompare; if (d->type->keyCompare) @@ -97,10 +107,6 @@ void dictSetHashFunctionSeed(uint8_t *seed) { memcpy(dict_hash_function_seed,seed,sizeof(dict_hash_function_seed)); } -uint8_t *dictGetHashFunctionSeed(void) { - return dict_hash_function_seed; -} - /* The default hashing function uses SipHash implementation * in siphash.c. */ @@ -514,54 +520,21 @@ int dictAdd(dict *d, void *key, void *val) dictEntry *dictAddRaw(dict *d, void *key, dictEntry **existing) { /* Get the position for the new key or NULL if the key already exists. */ - void *position = dictFindPositionForInsert(d, key, existing); + void *position = dictFindLinkForInsert(d, key, existing); if (!position) return NULL; /* Dup the key if necessary. */ if (d->type->keyDup) key = d->type->keyDup(d, key); - return dictInsertAtPosition(d, key, position); + return dictInsertKeyAtLink(d, key, position); } -/* Low-level add function for non-existing keys: - * This function adds a new entry to the dictionary, assuming the key does not - * already exist. - * Parameters: - * - `dict *d`: Pointer to the dictionary structure. - * - `void *key`: Pointer to the key being added. - * - `const uint64_t hash`: hash of the key being added. - * Guarantees: - * - The key is assumed to be non-existing. - * Note: - * Ensure that the key's uniqueness is managed externally before calling this function. */ -dictEntry *dictAddNonExistsByHash(dict *d, void *key, const uint64_t hash) { - /* Get the position for the new key, it should never be NULL. */ - unsigned long idx, table; - idx = hash & DICTHT_SIZE_MASK(d->ht_size_exp[0]); - - /* Rehash the hash table if needed */ - _dictRehashStepIfNeeded(d,idx); - - /* Expand the hash table if needed */ - _dictExpandIfNeeded(d); - - table = dictIsRehashing(d) ? 1 : 0; - idx = hash & DICTHT_SIZE_MASK(d->ht_size_exp[table]); - void *position = &d->ht_table[table][idx]; - assert(position!=NULL); - - /* Dup the key if necessary. */ - if (d->type->keyDup) key = d->type->keyDup(d, key); - - return dictInsertAtPosition(d, key, position); -} - -/* Adds a key in the dict's hashtable at the position returned by a preceding - * call to dictFindPositionForInsert. This is a low level function which allows +/* Adds a key in the dict's hashtable at the link returned by a preceding + * call to dictFindLinkForInsert(). This is a low level function which allows * splitting dictAddRaw in two parts. Normally, dictAddRaw or dictAdd should be - * used instead. */ -dictEntry *dictInsertAtPosition(dict *d, void *key, void *position) { - dictEntry **bucket = position; /* It's a bucket, but the API hides that. */ + * used instead. It assumes that dictExpandIfNeeded() was called before. */ +dictEntry *dictInsertKeyAtLink(dict *d, void *key, dictEntryLink link) { + dictEntryLink bucket = link; /* It's a bucket, but the API hides that. */ dictEntry *entry; /* If rehashing is ongoing, we insert in table 1, otherwise in table 0. * Assert that the provided bucket is the right table. */ @@ -646,6 +619,7 @@ dictEntry *dictAddOrFind(dict *d, void *key) { * dictDelete() and dictUnlink(), please check the top comment * of those functions. */ static dictEntry *dictGenericDelete(dict *d, const void *key, int nofree) { + dictCmpCache cmpCache = {0}; uint64_t h, idx; dictEntry *he, *prevHe; int table; @@ -659,7 +633,7 @@ static dictEntry *dictGenericDelete(dict *d, const void *key, int nofree) { /* Rehash the hash table if needed */ _dictRehashStepIfNeeded(d,idx); - keyCmpFunc cmpFunc = dictGetKeyCmpFunc(d); + keyCmpFunc cmpFunc = dictGetCmpFunc(d); for (table = 0; table <= 1; table++) { if (table == 0 && (long)idx < d->rehashidx) continue; @@ -668,7 +642,7 @@ static dictEntry *dictGenericDelete(dict *d, const void *key, int nofree) { prevHe = NULL; while(he) { void *he_key = dictGetKey(he); - if (key == he_key || cmpFunc(d, key, he_key)) { + if (key == he_key || cmpFunc(&cmpCache, key, he_key)) { /* Unlink the element from the list */ if (prevHe) dictSetNext(prevHe, dictGetNext(he)); @@ -777,55 +751,153 @@ void dictRelease(dict *d) zfree(d); } -dictEntry *dictFindByHash(dict *d, const void *key, const uint64_t hash) { - dictEntry *he; - uint64_t idx, table; - - if (dictSize(d) == 0) return NULL; /* dict is empty */ +/* Finds a given key. Like dictFindLink(), yet search bucket even if dict is empty. + * + * Returns dictEntryLink reference if found. Otherwise, return NULL. + * + * bucket - return pointer to bucket that the key was mapped. unless dict is empty. + */ +static dictEntryLink dictFindLinkInternal(dict *d, const void *key, dictEntryLink *bucket) { + dictCmpCache cmpCache = {0}; + dictEntryLink link; + uint64_t idx; + int table; + + if (bucket) { + *bucket = NULL; + } else { + /* If dict is empty and no need to find bucket, return NULL */ + if (dictSize(d) == 0) return NULL; + } + const uint64_t hash = dictHashKey(d, key, d->useStoredKeyApi); idx = hash & DICTHT_SIZE_MASK(d->ht_size_exp[0]); - keyCmpFunc cmpFunc = dictGetKeyCmpFunc(d); + keyCmpFunc cmpFunc = dictGetCmpFunc(d); /* Rehash the hash table if needed */ _dictRehashStepIfNeeded(d,idx); - /* Check if we can use the compare function with length to avoid recomputing length of key always */ - keyCmpFuncWithLen cmpFuncWithLen = d->type->keyCompareWithLen; - keyLenFunc keyLenFunc = d->type->keyLen; - const int has_len_fn = (keyLenFunc != NULL && cmpFuncWithLen != NULL); - const size_t key_len = has_len_fn ? keyLenFunc(d,key) : 0; - for (table = 0; table <= 1; table++) { + int tables = (dictIsRehashing(d)) ? 2 : 1; + for (table = 0; table < tables; table++) { if (table == 0 && (long)idx < d->rehashidx) continue; idx = hash & DICTHT_SIZE_MASK(d->ht_size_exp[table]); /* Prefetch the bucket at the calculated index */ redis_prefetch_read(&d->ht_table[table][idx]); - he = d->ht_table[table][idx]; - while(he) { - void *he_key = dictGetKey(he); + link = &(d->ht_table[table][idx]); + if (bucket) *bucket = link; + while(link && *link) { + void *visitedKey = dictGetKey(*link); /* Prefetch the next entry to improve cache efficiency */ - redis_prefetch_read(dictGetNext(he)); - if (key == he_key || (has_len_fn ? - cmpFuncWithLen(d, key, key_len, he_key, keyLenFunc(d,he_key)) : - cmpFunc(d, key, he_key))) - { - return he; - } - he = dictGetNext(he); + redis_prefetch_read(dictGetNext(*link)); + + if (key == visitedKey || cmpFunc( &cmpCache, key, visitedKey)) + return link; + + link = dictGetNextLink(*link); } - /* Use unlikely to optimize branch prediction for the common case */ - if (unlikely(!dictIsRehashing(d))) return NULL; } return NULL; } dictEntry *dictFind(dict *d, const void *key) { - if (dictSize(d) == 0) return NULL; /* dict is empty */ - const uint64_t hash = dictHashKey(d, key, d->useStoredKeyApi); - return dictFindByHash(d,key,hash); + dictEntryLink link = dictFindLink(d, key, NULL); + return (link) ? *link : NULL; +} + +/* Find a key and return its dictEntryLink reference. Otherwise, return NULL + * + * A dictEntryLink pointer being used to find preceding dictEntry of searched item. + * It is Useful for deletion, addition, unlinking and updating, especially for + * dict configured with 'no_value'. In such cases returning only `dictEntry` from + * a lookup may be insufficient since it might be opt-out to be the object itself. + * By locating preceding dictEntry (dictEntryLink) these ops can be properly handled. + * + * After calling link = dictFindLink(...), any necessary updates based on returned + * link or bucket must be performed immediately after by calling dictSetKeyAtLink() + * without any intervening operations on given dict. Otherwise, `dictEntryLink` may + * become invalid. Example with kvobj of replacing key with new key: + * + * link = dictFindLink(d, key, &bucket, 0); + * ... Do something, but don't modify the dict ... + * // assert(link != NULL); + * dictSetKeyAtLink(d, kv, &link, 0); + * + * To add new value (If no space for the new key, dict will be expanded by + * dictSetKeyAtLink() and bucket will be looked up again.): + * + * link = dictFindLink(d, key, &bucket); + * ... Do something, but don't modify the dict ... + * // assert(link == NULL); + * dictSetKeyAtLink(d, kv, &bucket, 1); + * + * bucket - return link to bucket that the key was mapped. unless dict is empty. + */ +dictEntryLink dictFindLink(dict *d, const void *key, dictEntryLink *bucket) { + if (bucket) *bucket = NULL; + if (unlikely(dictSize(d) == 0)) + return NULL; + + return dictFindLinkInternal(d, key, bucket); +} + +/* Set the key with link + * + * link: - When `newItem` is set, `link` points to the bucket of the key. + * - When `newItem` is not set, `link` points to the link of the key. + * - If *link is NULL, dictFindLink() will be called to locate the key. + * - On return, get updated, by need, to the inserted key. + * + * newItem: 1 = Add a key with a new dictEntry. + * 0 = Set a key to an existing dictEntry. + */ +void dictSetKeyAtLink(dict *d, void *key, dictEntryLink *link, int newItem) { + dictEntryLink dummy = NULL; + if (link == NULL) link = &dummy; + void *addedKey = (d->type->keyDup) ? d->type->keyDup(d, key) : key; + + if (newItem) { + signed char snap[2] = {d->ht_size_exp[0], d->ht_size_exp[1] }; + + /* Make room if needed for the new key */ + dictExpandIfNeeded(d); + + /* Lookup key's link if tables reallocated or if given link is set to NULL */ + if (snap[0] != d->ht_size_exp[0] || snap[1] != d->ht_size_exp[1] || *link == NULL) { + dictEntryLink bucket; + /* Bypass dictFindLink() to search bucket even if dict is empty!!! */ + dictUseStoredKeyApi(d, 1); + *link = dictFindLinkInternal(d, key, &bucket); + dictUseStoredKeyApi(d, 0); + assert(bucket != NULL); + assert(*link == NULL); + *link = bucket; /* On newItem the link should be the bucket */ + } + dictInsertKeyAtLink(d, addedKey, *link); + return; + } + + /* Setting key of existing dictEntry (newItem == 0)*/ + + if (*link == NULL) { + *link = dictFindLink(d, key, NULL); + assert(*link != NULL); + } + + dictEntry **de = *link; + /* is it regular dict entry of key and next */ + if (entryIsNoValue(*de)) { + decodeEntryNoValue(*de)->key = addedKey; + } else if (entryIsKey(*de)) { + /* `de` opt-out to be actually a key. Replace key but keep the lsb flags */ + int mask = ((uintptr_t) *de) & ENTRY_PTR_MASK; + *de = encodeMaskedPtr(addedKey, mask); + } else { + (*de)->key = addedKey; /* `de` is a normal key-value dict entry */ + } } void *dictFetchValue(dict *d, const void *key) { @@ -835,30 +907,31 @@ void *dictFetchValue(dict *d, const void *key) { return he ? dictGetVal(he) : NULL; } -/* Find an element from the table, also get the plink of the entry. The entry - * is returned if the element is found, and the user should later call - * `dictTwoPhaseUnlinkFree` with it in order to unlink and release it. Otherwise if - * the key is not found, NULL is returned. These two functions should be used in pair. +/* Find an element from the table. A link is returned if the element is found, and + * the user should later call `dictTwoPhaseUnlinkFree` with it in order to unlink + * and release it. Otherwise if the key is not found, NULL is returned. These two + * functions should be used in pair. * `dictTwoPhaseUnlinkFind` pauses rehash and `dictTwoPhaseUnlinkFree` resumes rehash. * * We can use like this: * - * dictEntry *de = dictTwoPhaseUnlinkFind(db->dict,key->ptr,&plink, &table); + * dictEntryLink link = dictTwoPhaseUnlinkFind(db->dict,key->ptr, &table); * // Do something, but we can't modify the dict - * dictTwoPhaseUnlinkFree(db->dict,de,plink,table); // We don't need to lookup again + * dictTwoPhaseUnlinkFree(db->dict, link, table); // We don't need to lookup again * * If we want to find an entry before delete this entry, this an optimization to avoid * dictFind followed by dictDelete. i.e. the first API is a find, and it gives some info * to the second one to avoid repeating the lookup */ -dictEntry *dictTwoPhaseUnlinkFind(dict *d, const void *key, dictEntry ***plink, int *table_index) { +dictEntryLink dictTwoPhaseUnlinkFind(dict *d, const void *key, int *table_index) { + dictCmpCache cmpCache = {0}; uint64_t h, idx, table; if (dictSize(d) == 0) return NULL; /* dict is empty */ if (dictIsRehashing(d)) _dictRehashStep(d); - h = dictHashKey(d, key, d->useStoredKeyApi); - keyCmpFunc cmpFunc = dictGetKeyCmpFunc(d); + h = dictHashKey(d, key, d->useStoredKeyApi); + keyCmpFunc cmpFunc = dictGetCmpFunc(d); for (table = 0; table <= 1; table++) { idx = h & DICTHT_SIZE_MASK(d->ht_size_exp[table]); @@ -866,26 +939,27 @@ dictEntry *dictTwoPhaseUnlinkFind(dict *d, const void *key, dictEntry ***plink, dictEntry **ref = &d->ht_table[table][idx]; while (ref && *ref) { void *de_key = dictGetKey(*ref); - if (key == de_key || cmpFunc(d, key, de_key)) { + if (key == de_key || cmpFunc(&cmpCache, key, de_key)) { *table_index = table; - *plink = ref; dictPauseRehashing(d); - return *ref; + return ref; } - ref = dictGetNextRef(*ref); + ref = dictGetNextLink(*ref); } if (!dictIsRehashing(d)) return NULL; } return NULL; } -void dictTwoPhaseUnlinkFree(dict *d, dictEntry *he, dictEntry **plink, int table_index) { - if (he == NULL) return; +void dictTwoPhaseUnlinkFree(dict *d, dictEntryLink plink, int table_index) { + if (plink == NULL || *plink == NULL) return; + dictEntry *de = *plink; d->ht_used[table_index]--; - *plink = dictGetNext(he); - dictFreeKey(d, he); - dictFreeVal(d, he); - if (!entryIsKey(he)) zfree(decodeMaskedPtr(he)); + + *plink = dictGetNext(de); + dictFreeKey(d, de); + dictFreeVal(d, de); + if (!entryIsKey(de)) zfree(decodeMaskedPtr(de)); _dictShrinkIfNeeded(d); dictResumeRehashing(d); } @@ -978,7 +1052,7 @@ static dictEntry *dictGetNext(const dictEntry *de) { /* Returns a pointer to the 'next' field in the entry or NULL if the entry * doesn't have a next field. */ -static dictEntry **dictGetNextRef(dictEntry *de) { +static dictEntryLink dictGetNextLink(dictEntry *de) { if (entryIsKey(de)) return NULL; if (entryIsNoValue(de)) return &decodeEntryNoValue(de)->next; return &de->next; @@ -1001,8 +1075,8 @@ size_t dictMemUsage(const dict *d) { dictBuckets(d) * sizeof(dictEntry*); } -size_t dictEntryMemUsage(void) { - return sizeof(dictEntry); +size_t dictEntryMemUsage(int noValueDict) { + return (noValueDict) ? sizeof(dictEntryNoValue) :sizeof(dictEntry); } /* A fingerprint is a 64 bit number that represents the state of the dictionary @@ -1284,7 +1358,6 @@ static void dictDefragBucket(dictEntry **bucketref, dictDefragFunctions *defragf void *newval = defragval ? defragval(dictGetVal(de)) : NULL; if (entryIsKey(de)) { if (newkey) *bucketref = newkey; - assert(entryIsKey(*bucketref)); } else if (entryIsNoValue(de)) { dictEntryNoValue *entry = decodeEntryNoValue(de), *newentry; if ((newentry = defragalloc(entry))) { @@ -1302,7 +1375,7 @@ static void dictDefragBucket(dictEntry **bucketref, dictDefragFunctions *defragf if (newde) { *bucketref = newde; } - bucketref = dictGetNextRef(*bucketref); + bucketref = dictGetNextLink(*bucketref); } } @@ -1434,6 +1507,32 @@ unsigned long dictScan(dict *d, return dictScanDefrag(d, v, fn, NULL, privdata); } +void dictScanDefragBucket(dictScanFunction *fn, + dictDefragFunctions *defragfns, + void *privdata, + dictEntry **bucketref) { + dictEntry **plink, *de, *next; + + /* Emit entries at bucket */ + if (defragfns) dictDefragBucket(bucketref, defragfns); + + de = *bucketref; + plink = bucketref; + while (de) { + next = dictGetNext(de); + fn(privdata, de, plink); + + if (!next) break; /* if last element, break */ + + /* if `*plink` still pointing to 'de', then it means that the + * visited item wasn't deleted by fn() */ + if (*plink == de) + plink = (entryIsNoValue(de)) ? &(decodeEntryNoValue(de)->next) : &(de->next); + + de = next; + } +} + /* Like dictScan, but additionally reallocates the memory used by the dict * entries using the provided allocation function. This feature was added for * the active defrag feature. @@ -1449,7 +1548,6 @@ unsigned long dictScanDefrag(dict *d, void *privdata) { int htidx0, htidx1; - const dictEntry *de, *next; unsigned long m0, m1; if (dictSize(d) == 0) return 0; @@ -1460,17 +1558,7 @@ unsigned long dictScanDefrag(dict *d, if (!dictIsRehashing(d)) { htidx0 = 0; m0 = DICTHT_SIZE_MASK(d->ht_size_exp[htidx0]); - - /* Emit entries at cursor */ - if (defragfns) { - dictDefragBucket(&d->ht_table[htidx0][v & m0], defragfns); - } - de = d->ht_table[htidx0][v & m0]; - while (de) { - next = dictGetNext(de); - fn(privdata, de); - de = next; - } + dictScanDefragBucket(fn, defragfns, privdata, &d->ht_table[htidx0][v & m0]); /* Set unmasked bits so incrementing the reversed cursor * operates on the masked bits */ @@ -1494,30 +1582,12 @@ unsigned long dictScanDefrag(dict *d, m0 = DICTHT_SIZE_MASK(d->ht_size_exp[htidx0]); m1 = DICTHT_SIZE_MASK(d->ht_size_exp[htidx1]); - /* Emit entries at cursor */ - if (defragfns) { - dictDefragBucket(&d->ht_table[htidx0][v & m0], defragfns); - } - de = d->ht_table[htidx0][v & m0]; - while (de) { - next = dictGetNext(de); - fn(privdata, de); - de = next; - } + dictScanDefragBucket(fn, defragfns, privdata, &d->ht_table[htidx0][v & m0]); /* Iterate over indices in larger table that are the expansion * of the index pointed to by the cursor in the smaller table */ do { - /* Emit entries at cursor */ - if (defragfns) { - dictDefragBucket(&d->ht_table[htidx1][v & m1], defragfns); - } - de = d->ht_table[htidx1][v & m1]; - while (de) { - next = dictGetNext(de); - fn(privdata, de); - de = next; - } + dictScanDefragBucket(fn, defragfns, privdata, &d->ht_table[htidx1][v & m1]); /* Increment the reverse cursor not covered by the smaller mask.*/ v |= ~m1; @@ -1575,12 +1645,12 @@ int dictExpandIfNeeded(dict *d) { return DICT_ERR; } -/* Expand the hash table if needed */ -static void _dictExpandIfNeeded(dict *d) { +/* Expand the hash table if needed (OK=Expanded, ERR=Not expanded) */ +static int _dictExpandIfNeeded(dict *d) { /* Automatic resizing is disallowed. Return */ - if (d->pauseAutoResize > 0) return; - - dictExpandIfNeeded(d); + if (d->pauseAutoResize > 0) return DICT_ERR; + + return dictExpandIfNeeded(d); } /* Returning DICT_OK indicates a successful shrinking or the dictionary is undergoing rehashing, @@ -1640,12 +1710,13 @@ static signed char _dictNextExp(unsigned long size) return 8*sizeof(long) - __builtin_clzl(size-1); } -/* Finds and returns the position within the dict where the provided key should - * be inserted using dictInsertAtPosition if the key does not already exist in +/* Finds and returns the link within the dict where the provided key should + * be inserted using dictInsertKeyAtLink() if the key does not already exist in * the dict. If the key exists in the dict, NULL is returned and the optional * 'existing' entry pointer is populated, if provided. */ -void *dictFindPositionForInsert(dict *d, const void *key, dictEntry **existing) { +dictEntryLink dictFindLinkForInsert(dict *d, const void *key, dictEntry **existing) { unsigned long idx, table; + dictCmpCache cmpCache = {0}; dictEntry *he; uint64_t hash = dictHashKey(d, key, d->useStoredKeyApi); if (existing) *existing = NULL; @@ -1656,7 +1727,7 @@ void *dictFindPositionForInsert(dict *d, const void *key, dictEntry **existing) /* Expand the hash table if needed */ _dictExpandIfNeeded(d); - keyCmpFunc cmpFunc = dictGetKeyCmpFunc(d); + keyCmpFunc cmpFunc = dictGetCmpFunc(d); for (table = 0; table <= 1; table++) { if (table == 0 && (long)idx < d->rehashidx) continue; @@ -1665,7 +1736,7 @@ void *dictFindPositionForInsert(dict *d, const void *key, dictEntry **existing) he = d->ht_table[table][idx]; while(he) { void *he_key = dictGetKey(he); - if (key == he_key || cmpFunc(d, key, he_key)) { + if (key == he_key || cmpFunc(&cmpCache, key, he_key)) { if (existing) *existing = he; return NULL; } @@ -1706,30 +1777,6 @@ uint64_t dictGetHash(dict *d, const void *key) { return dictHashKey(d, key, d->useStoredKeyApi); } -/* Finds the dictEntry using pointer and pre-calculated hash. - * oldkey is a dead pointer and should not be accessed. - * the hash value should be provided using dictGetHash. - * no string / key comparison is performed. - * return value is a pointer to the dictEntry if found, or NULL if not found. */ -dictEntry *dictFindByHashAndPtr(dict *d, const void *oldptr, const uint64_t hash) { - dictEntry *he; - unsigned long idx, table; - - if (dictSize(d) == 0) return NULL; /* dict is empty */ - for (table = 0; table <= 1; table++) { - idx = hash & DICTHT_SIZE_MASK(d->ht_size_exp[table]); - if (table == 0 && (long)idx < d->rehashidx) continue; - he = d->ht_table[table][idx]; - while(he) { - if (oldptr == dictGetKey(he)) - return he; - he = dictGetNext(he); - } - if (!dictIsRehashing(d)) return NULL; - } - return NULL; -} - /* Provides the old and new ht size for a given dictionary during rehashing. This method * should only be invoked during initialization/rehashing. */ void dictRehashingInfo(dict *d, unsigned long long *from_size, unsigned long long *to_size) { @@ -1848,8 +1895,8 @@ void dictGetStats(char *buf, size_t bufsize, dict *d, int full) { orig_buf[orig_bufsize-1] = '\0'; } -static int dictDefaultCompare(dict *d, const void *key1, const void *key2) { - (void)(d); /*unused*/ +static int dictDefaultCompare(dictCmpCache *cache, const void *key1, const void *key2) { + (void)(cache); /*unused*/ return key1 == key2; } @@ -1865,9 +1912,9 @@ uint64_t hashCallback(const void *key) { return dictGenHashFunction((unsigned char*)key, strlen((char*)key)); } -int compareCallback(dict *d, const void *key1, const void *key2) { +int compareCallback(dictCmpCache *cache, const void *key1, const void *key2) { int l1,l2; - UNUSED(d); + UNUSED(cache); l1 = strlen((char*)key1); l2 = strlen((char*)key2); @@ -2126,57 +2173,6 @@ int dictTest(int argc, char **argv, int flags) { end_benchmark("Inserting via dictAddRaw() existing (no insertion)"); assert((long)dictSize(d) == count); - dictEmpty(d, NULL); - - start_benchmark(); - for (j = 0; j < count; j++) { - void *key = stringFromLongLong(j); - const uint64_t hash = dictGetHash(d, key); - de = dictAddNonExistsByHash(d,key,hash); - assert(de != NULL); - } - end_benchmark("Inserting via dictAddNonExistsByHash() non existing"); - assert((long)dictSize(d) == count); - - /* Wait for rehashing. */ - while (dictIsRehashing(d)) { - dictRehashMicroseconds(d,100*1000); - } - - dictEmpty(d, NULL); - - start_benchmark(); - for (j = 0; j < count; j++) { - /* Create a key */ - void *key = stringFromLongLong(j); - - /* Check if the key exists */ - dictEntry *entry = dictFind(d, key); - assert(entry == NULL); - - /* Add the key */ - dictEntry *de = dictAddRaw(d, key, NULL); - assert(de != NULL); - } - end_benchmark("Find() and inserting via dictFind()+dictAddRaw() non existing"); - - dictEmpty(d, NULL); - - start_benchmark(); - for (j = 0; j < count; j++) { - /* Create a key */ - void *key = stringFromLongLong(j); - uint64_t hash = dictGetHash(d, key); - - /* Check if the key exists */ - dictEntry *entry = dictFindByHash(d, key, hash); - assert(entry == NULL); - de = dictAddNonExistsByHash(d, key, hash); - assert(de != NULL); - } - end_benchmark("Find() and inserting via dictGetHash()+dictFindByHash()+dictAddNonExistsByHash() non existing"); - assert((long)dictSize(d) == count); - /* Wait for rehashing. */ while (dictIsRehashing(d)) { dictRehashMicroseconds(d,100*1000); @@ -2275,6 +2271,53 @@ int dictTest(int argc, char **argv, int flags) { zfree(lookupKeys); } + TEST("Test dictFindLink() functionality") { + dictType dt = BenchmarkDictType; + dict *d = dictCreate(&dt); + + /* find in empty dict */ + dictEntryLink link = dictFindLink(d, "key", NULL); + assert(link == NULL); + + /* Add keys to dict and test */ + for (j = 0; j < 10; j++) { + /* Add another key to dict */ + char *key = stringFromLongLong(j); + retval = dictAdd(d, key, (void*)j); + assert(retval == DICT_OK); + /* find existing keys with dictFindLink() */ + dictEntryLink link = dictFindLink(d, key, NULL); + assert(link != NULL); + assert(*link != NULL); + assert(dictGetKey(*link) != NULL); + + /* Test that the key found is the correct one */ + void *foundKey = dictGetKey(*link); + assert(compareCallback( NULL, foundKey, key)); + + /* Test finding a non-existing key */ + char *nonExistingKey = stringFromLongLong(j + 10); + link = dictFindLink(d, nonExistingKey, NULL); + assert(link == NULL); + + /* Test with bucket parameter */ + dictEntryLink bucket = NULL; + link = dictFindLink(d, key, &bucket); + assert(link != NULL); + assert(bucket != NULL); + + /* Test bucket parameter with non-existing key */ + link = dictFindLink(d, nonExistingKey, &bucket); + assert(link == NULL); + assert(bucket != NULL); /* Bucket should still be set even for non-existing keys */ + + /* Clean up */ + zfree(nonExistingKey); + } + + dictRelease(d); + } + return 0; } #endif diff --git a/src/dict.h b/src/dict.h index 7af501110..288d95b97 100644 --- a/src/dict.h +++ b/src/dict.h @@ -29,15 +29,33 @@ typedef struct dictEntry dictEntry; /* opaque */ typedef struct dict dict; -typedef size_t (*keyLenFunc)(dict *d, const void *key1); -typedef int (*keyCmpFuncWithLen)(dict *d, const void *key1, const size_t key1_len, const void *key2, const size_t key2_len); +typedef dictEntry **dictEntryLink; /* See description of dictFindLink() */ + +/* Searching for a key in a dict may involve few comparisons. + * If extracting the looked-up key is expensive (e.g., sdslen(), kvobjGetKey()), + * caching can be used to reduce those repetitive computations. + * + * This struct, passed to the comparison function as temporary caching, if + * needed by the function across comparison of a given lookup. + * for the looked-up key and resets before each new lookup. */ +typedef struct dictCmpCache { + int useCache; + + union { + uint64_t u64; + int64_t i64; + int i; + size_t sz; + void *p; + } data[2]; +} dictCmpCache; typedef struct dictType { /* Callbacks */ uint64_t (*hashFunction)(const void *key); void *(*keyDup)(dict *d, const void *key); void *(*valDup)(dict *d, const void *obj); - int (*keyCompare)(dict *d, const void *key1, const void *key2); + int (*keyCompare)(dictCmpCache *cache, const void *key1, const void *key2); void (*keyDestructor)(dict *d, void *key); void (*valDestructor)(dict *d, void *obj); int (*resizeAllowed)(size_t moreMem, double usedRatio); @@ -66,8 +84,6 @@ typedef struct dictType { /* This flag is required for `no_value` optimization since the optimization * reuses LSB bits as metadata */ unsigned int keys_are_odd:1; - /* TODO: Add a 'keys_are_even' flag and use a similar optimization if that - * flag is set. */ /* Ensures that the entire hash table is rehashed at once if set. */ unsigned int force_full_rehash:1; @@ -94,14 +110,10 @@ typedef struct dictType { * * Set to NULL both functions, if you don't want to support this feature. */ uint64_t (*storedHashFunction)(const void *key); - int (*storedKeyCompare)(dict *d, const void *key1, const void *key2); + int (*storedKeyCompare)(dictCmpCache *cache, const void *key1, const void *key2); /* Optional callback called when the dict is destroyed. */ void (*onDictRelease)(dict *d); - - /* Optional keylen to avoid duplication computation of key lengths. */ - keyLenFunc keyLen; - keyCmpFuncWithLen keyCompareWithLen; } dictType; #define DICTHT_SIZE(exp) ((exp) == -1 ? 0 : (unsigned long)1<<(exp)) @@ -147,7 +159,7 @@ typedef struct dictStats { unsigned long *clvector; } dictStats; -typedef void (dictScanFunction)(void *privdata, const dictEntry *de); +typedef void (dictScanFunction)(void *privdata, const dictEntry *de, dictEntry **plink); typedef void *(dictDefragAllocFunction)(void *ptr); typedef struct { dictDefragAllocFunction *defragAlloc; /* Used for entries etc. */ @@ -210,40 +222,20 @@ int dictTryExpand(dict *d, unsigned long size); int dictShrink(dict *d, unsigned long size); int dictAdd(dict *d, void *key, void *val); dictEntry *dictAddRaw(dict *d, void *key, dictEntry **existing); -dictEntry *dictAddNonExistsByHash(dict *d, void *key, const uint64_t hash); -void *dictFindPositionForInsert(dict *d, const void *key, dictEntry **existing); -dictEntry *dictInsertAtPosition(dict *d, void *key, void *position); dictEntry *dictAddOrFind(dict *d, void *key); int dictReplace(dict *d, void *key, void *val); int dictDelete(dict *d, const void *key); dictEntry *dictUnlink(dict *d, const void *key); void dictFreeUnlinkedEntry(dict *d, dictEntry *he); -dictEntry *dictTwoPhaseUnlinkFind(dict *d, const void *key, dictEntry ***plink, int *table_index); -void dictTwoPhaseUnlinkFree(dict *d, dictEntry *he, dictEntry **plink, int table_index); +dictEntryLink dictTwoPhaseUnlinkFind(dict *d, const void *key, int *table_index); +void dictTwoPhaseUnlinkFree(dict *d, dictEntryLink llink, int table_index); void dictRelease(dict *d); dictEntry * dictFind(dict *d, const void *key); -dictEntry *dictFindByHash(dict *d, const void *key, const uint64_t hash); -dictEntry *dictFindByHashAndPtr(dict *d, const void *oldptr, const uint64_t hash); -void *dictFetchValue(dict *d, const void *key); int dictShrinkIfNeeded(dict *d); int dictExpandIfNeeded(dict *d); -void dictSetKey(dict *d, dictEntry* de, void *key); -void dictSetVal(dict *d, dictEntry *de, void *val); -void dictSetSignedIntegerVal(dictEntry *de, int64_t val); -void dictSetUnsignedIntegerVal(dictEntry *de, uint64_t val); -void dictSetDoubleVal(dictEntry *de, double val); -int64_t dictIncrSignedIntegerVal(dictEntry *de, int64_t val); -uint64_t dictIncrUnsignedIntegerVal(dictEntry *de, uint64_t val); -double dictIncrDoubleVal(dictEntry *de, double val); -void *dictEntryMetadata(dictEntry *de); void *dictGetKey(const dictEntry *de); -void *dictGetVal(const dictEntry *de); -int64_t dictGetSignedIntegerVal(const dictEntry *de); -uint64_t dictGetUnsignedIntegerVal(const dictEntry *de); -double dictGetDoubleVal(const dictEntry *de); -double *dictGetDoubleValPtr(dictEntry *de); size_t dictMemUsage(const dict *d); -size_t dictEntryMemUsage(void); +size_t dictEntryMemUsage(int noValueDict); dictIterator *dictGetIterator(dict *d); dictIterator *dictGetSafeIterator(dict *d); void dictInitIterator(dictIterator *iter, dict *d); @@ -262,7 +254,6 @@ void dictSetResizeEnabled(dictResizeEnable enable); int dictRehash(dict *d, int n); int dictRehashMicroseconds(dict *d, uint64_t us); void dictSetHashFunctionSeed(uint8_t *seed); -uint8_t *dictGetHashFunctionSeed(void); unsigned long dictScan(dict *d, unsigned long v, dictScanFunction *fn, void *privdata); unsigned long dictScanDefrag(dict *d, unsigned long v, dictScanFunction *fn, dictDefragFunctions *defragfns, void *privdata); uint64_t dictGetHash(dict *d, const void *key); @@ -273,6 +264,21 @@ dictStats* dictGetStatsHt(dict *d, int htidx, int full); void dictCombineStats(dictStats *from, dictStats *into); void dictFreeStats(dictStats *stats); +dictEntryLink dictFindLink(dict *d, const void *key, dictEntryLink *bucket); +void dictSetKeyAtLink(dict *d, void *key, dictEntryLink *link, int newItem); + +/* API relevant only when dict is used as a hash-map (no_value=0) */ +void dictSetKey(dict *d, dictEntry* de, void *key); +void dictSetVal(dict *d, dictEntry *de, void *val); +void *dictGetVal(const dictEntry *de); +void dictSetDoubleVal(dictEntry *de, double val); +double dictGetDoubleVal(const dictEntry *de); +double *dictGetDoubleValPtr(dictEntry *de); +void *dictFetchValue(dict *d, const void *key); +void dictSetUnsignedIntegerVal(dictEntry *de, uint64_t val); +uint64_t dictIncrUnsignedIntegerVal(dictEntry *de, uint64_t val); +uint64_t dictGetUnsignedIntegerVal(const dictEntry *de); + #define dictForEach(d, ty, m, ...) do { \ dictIterator *di = dictGetIterator(d); \ dictEntry *de; \ diff --git a/src/evict.c b/src/evict.c index 19e4d59f4..52c66fbf1 100644 --- a/src/evict.c +++ b/src/evict.c @@ -131,27 +131,16 @@ int evictionPoolPopulate(redisDb *db, kvstore *samplekvs, struct evictionPoolEnt count = kvstoreDictGetSomeKeys(samplekvs,slot,samples,server.maxmemory_samples); for (j = 0; j < count; j++) { unsigned long long idle; - sds key; - robj *o; - dictEntry *de; - - de = samples[j]; - key = dictGetKey(de); - - /* If the dictionary we are sampling from is not the main - * dictionary (but the expires one) we need to lookup the key - * again in the key dictionary to obtain the value object. */ - if (server.maxmemory_policy != MAXMEMORY_VOLATILE_TTL) { - if (samplekvs != db->keys) - de = kvstoreDictFind(db->keys, slot, key); - o = dictGetVal(de); - } - + + dictEntry *de = samples[j]; + kvobj *kv = dictGetKV(de); + sds key = kvobjGetKey(kv); + /* Calculate the idle time according to the policy. This is called * idle just because the code initially handled LRU, but is in fact * just a score where a higher score means better candidate. */ if (server.maxmemory_policy & MAXMEMORY_FLAG_LRU) { - idle = estimateObjectIdleTime(o); + idle = estimateObjectIdleTime(kv); } else if (server.maxmemory_policy & MAXMEMORY_FLAG_LFU) { /* When we use an LRU policy, we sort the keys by idle time * so that we expire keys starting from greater idle time. @@ -160,10 +149,10 @@ int evictionPoolPopulate(redisDb *db, kvstore *samplekvs, struct evictionPoolEnt * first. So inside the pool we put objects using the inverted * frequency subtracting the actual frequency to the maximum * frequency of 255. */ - idle = 255-LFUDecrAndReturn(o); + idle = 255-LFUDecrAndReturn(kv); } else if (server.maxmemory_policy == MAXMEMORY_VOLATILE_TTL) { /* In this case the sooner the expire the better. */ - idle = ULLONG_MAX - dictGetSignedIntegerVal(de); + idle = ULLONG_MAX - kvobjGetExpire(kv); } else { serverPanic("Unknown eviction policy in evictionPoolPopulate()"); } @@ -622,7 +611,7 @@ int performEvictions(void) { /* If the key exists, is our pick. Otherwise it is * a ghost and we need to try the next element. */ if (de) { - bestkey = dictGetKey(de); + bestkey = kvobjGetKey(dictGetKV(de)); break; } else { /* Ghost... Iterate again. */ @@ -650,7 +639,8 @@ int performEvictions(void) { int slot = kvstoreGetFairRandomDictIndex(kvs); de = kvstoreDictGetRandomKey(kvs, slot); if (de) { - bestkey = dictGetKey(de); + kvobj *kv = dictGetKV(de); + bestkey = kvobjGetKey(kv); bestdbid = j; break; } diff --git a/src/expire.c b/src/expire.c index 7a5cdce81..1b3492b09 100644 --- a/src/expire.c +++ b/src/expire.c @@ -25,8 +25,8 @@ static double avg_ttl_factor[16] = {0.98, 0.9604, 0.941192, 0.922368, 0.903921, 0.885842, 0.868126, 0.850763, 0.833748, 0.817073, 0.800731, 0.784717, 0.769022, 0.753642, 0.738569, 0.723798}; /* Helper function for the activeExpireCycle() function. - * This function will try to expire the key that is stored in the hash table - * entry 'de' of the 'expires' hash table of a Redis database. + * This function will try to expire the key-value entry that is stored in the + * hash table entry 'de' of the 'expires' hash table of a Redis database. * * If the key is found to be expired, it is removed from the database and * 1 is returned. Otherwise no operation is performed and 0 is returned. @@ -35,13 +35,12 @@ static double avg_ttl_factor[16] = {0.98, 0.9604, 0.941192, 0.922368, 0.903921, * * The parameter 'now' is the current time in milliseconds as is passed * to the function to avoid too many gettimeofday() syscalls. */ -int activeExpireCycleTryExpire(redisDb *db, dictEntry *de, long long now) { - long long t = dictGetSignedIntegerVal(de); - if (now < t) +int activeExpireCycleTryExpire(redisDb *db, kvobj *kv, long long now) { + if (now < kvobjGetExpire(kv)) return 0; enterExecutionUnit(1, 0); - sds key = dictGetKey(de); + sds key = kvobjGetKey(kv); robj *keyobj = createStringObject(key,sdslen(key)); deleteExpiredKeyAndPropagate(db,keyobj); decrRefCount(keyobj); @@ -109,11 +108,12 @@ typedef struct { int ttl_samples; /* num keys with ttl not yet expired */ } expireScanData; -void expireScanCallback(void *privdata, const dictEntry *const_de) { - dictEntry *de = (dictEntry *)const_de; +void expireScanCallback(void *privdata, const dictEntry *de, dictEntryLink plink) { + UNUSED(plink); + kvobj *kv = dictGetKV(de); expireScanData *data = privdata; - long long ttl = dictGetSignedIntegerVal(de) - data->now; - if (activeExpireCycleTryExpire(data->db, de, data->now)) { + long long ttl = kvobjGetExpire(kv) - data->now; + if (activeExpireCycleTryExpire(data->db, kv, data->now)) { data->expired++; } if (ttl > 0) { @@ -464,14 +464,14 @@ void expireSlaveKeys(void) { while(dbids && dbid < server.dbnum) { if ((dbids & 1) != 0) { redisDb *db = server.db+dbid; - dictEntry *expire = dbFindExpires(db, keyname); - int expired = expire && activeExpireCycleTryExpire(server.db+dbid,expire,start); + kvobj *kv = dbFindExpires(db, keyname); + int expired = kv && activeExpireCycleTryExpire(server.db+dbid, kv, start); /* If the key was not expired in this DB, we need to set the * corresponding bit in the new bitmap we set as value. * At the end of the loop if the bitmap is zero, it means we * no longer need to keep track of this key. */ - if (expire && !expired) { + if (kv && !expired) { noexpire++; new_dbids |= (uint64_t)1 << dbid; } @@ -499,7 +499,7 @@ void expireSlaveKeys(void) { /* Track keys that received an EXPIRE or similar command in the context * of a writable slave. */ -void rememberSlaveKeyWithExpire(redisDb *db, robj *key) { +void rememberSlaveKeyWithExpire(redisDb *db, sds key) { if (slaveKeysWithExpire == NULL) { static dictType dt = { dictSdsHash, /* hash function */ @@ -514,13 +514,13 @@ void rememberSlaveKeyWithExpire(redisDb *db, robj *key) { } if (db->id > 63) return; - dictEntry *de = dictAddOrFind(slaveKeysWithExpire,key->ptr); + dictEntry *de = dictAddOrFind(slaveKeysWithExpire, key); /* If the entry was just created, set it to a copy of the SDS string * representing the key: we don't want to need to take those keys * in sync with the main DB. The keys will be removed by expireSlaveKeys() * as it scans to find keys to remove. */ - if (dictGetKey(de) == key->ptr) { - dictSetKey(slaveKeysWithExpire, de, sdsdup(key->ptr)); + if (dictGetKey(de) == key) { + dictSetKey(slaveKeysWithExpire, de, sdsdup(key)); dictSetUnsignedIntegerVal(de,0); } @@ -654,13 +654,14 @@ void expireGenericCommand(client *c, long long basetime, int unit) { when += basetime; /* No key, return zero. */ - if (lookupKeyWrite(c->db,key) == NULL) { + kvobj *kv = lookupKeyWrite(c->db,key); + if (kv == NULL) { addReply(c,shared.czero); return; } if (flag) { - current_expire = getExpire(c->db, key); + current_expire = kvobjGetExpire(kv); /* NX option is set, check current expiry */ if (flag & EXPIRE_NX) { @@ -765,14 +766,15 @@ void ttlGenericCommand(client *c, int output_ms, int output_abs) { long long expire, ttl = -1; /* If the key does not exist at all, return -2 */ - if (lookupKeyReadWithFlags(c->db,c->argv[1],LOOKUP_NOTOUCH) == NULL) { + kvobj *kv = lookupKeyReadWithFlags(c->db,c->argv[1],LOOKUP_NOTOUCH); + if (kv == NULL) { addReplyLongLong(c,-2); return; } /* The key exists. Return -1 if it has no expire, or the actual * TTL value otherwise. */ - expire = getExpire(c->db,c->argv[1]); + expire = kvobjGetExpire(kv); if (expire != -1) { ttl = output_abs ? expire : expire-commandTimeSnapshot(); if (ttl < 0) ttl = 0; diff --git a/src/geo.c b/src/geo.c index cbb96cc25..1f9e5d20f 100644 --- a/src/geo.c +++ b/src/geo.c @@ -525,7 +525,7 @@ void georadiusGeneric(client *c, int srcKeyIndex, int flags) { int storedist = 0; /* 0 for STORE, 1 for STOREDIST. */ /* Look up the requested zset */ - robj *zobj = lookupKeyRead(c->db, c->argv[srcKeyIndex]); + kvobj *zobj = lookupKeyRead(c->db, c->argv[srcKeyIndex]); if (checkType(c, zobj, OBJ_ZSET)) return; /* Find long/lat to use for radius or box search based on inquiry type */ @@ -828,8 +828,7 @@ void georadiusGeneric(client *c, int srcKeyIndex, int flags) { if (returned_items) { zsetConvertToListpackIfNeeded(zobj,maxelelen,totelelen); - setKey(c,c->db,storekey,zobj,0); - decrRefCount(zobj); + setKey(c,c->db,storekey,&zobj,0); notifyKeyspaceEvent(NOTIFY_ZSET,flags & GEOSEARCH ? "geosearchstore" : "georadiusstore",storekey, c->db->id); server.dirty += returned_items; @@ -880,7 +879,7 @@ void geohashCommand(client *c) { int j; /* Look up the requested zset */ - robj *zobj = lookupKeyRead(c->db, c->argv[1]); + kvobj *zobj = lookupKeyRead(c->db, c->argv[1]); if (checkType(c, zobj, OBJ_ZSET)) return; /* Geohash elements one after the other, using a null bulk reply for @@ -983,7 +982,7 @@ void geodistCommand(client *c) { } /* Look up the requested zset */ - robj *zobj = NULL; + kvobj *zobj = NULL; if ((zobj = lookupKeyReadOrReply(c, c->argv[1], shared.null[c->resp])) == NULL || checkType(c, zobj, OBJ_ZSET)) return; diff --git a/src/hyperloglog.c b/src/hyperloglog.c index 3088ade9a..f873897d8 100644 --- a/src/hyperloglog.c +++ b/src/hyperloglog.c @@ -1414,29 +1414,29 @@ invalid: /* PFADD var ele ele ele ... ele => :0 or :1 */ void pfaddCommand(client *c) { - robj *o = lookupKeyWrite(c->db,c->argv[1]); + uint64_t oldlen; + dictEntryLink link; + kvobj *kv = lookupKeyWriteWithLink(c->db,c->argv[1], &link); + struct hllhdr *hdr; int updated = 0, j; - if (o == NULL) { + if (kv == NULL) { /* Create the key with a string value of the exact length to * hold our HLL data structure. sdsnewlen() when NULL is passed * is guaranteed to return bytes initialized to zero. */ - o = createHLLObject(); - dbAdd(c->db,c->argv[1],o); + robj *o = createHLLObject(); + kv = dbAddByLink(c->db, c->argv[1], &o, &link); updated++; } else { - if (isHLLObjectOrReply(c,o) != C_OK) return; - o = dbUnshareStringValue(c->db,c->argv[1],o); + if (isHLLObjectOrReply(c,kv) != C_OK) return; + kv = dbUnshareStringValue(c->db,c->argv[1],kv); } - - /* HLL might change from sparse to dense. No way to predict KEYSIZES diff. - * Update as if the key is being removed. After the for-loop update it back */ - int64_t oldlen = stringObjectLen(o); + oldlen = stringObjectLen(kv); /* Perform the low level ADD operation for every element. */ for (j = 2; j < c->argc; j++) { - int retval = hllAdd(o, (unsigned char*)c->argv[j]->ptr, + int retval = hllAdd(kv, (unsigned char*)c->argv[j]->ptr, sdslen(c->argv[j]->ptr)); switch(retval) { case 1: @@ -1448,8 +1448,8 @@ void pfaddCommand(client *c) { } } - updateKeysizesHist(c->db, getKeySlot(c->argv[1]->ptr), OBJ_STRING, oldlen, stringObjectLen(o)); - hdr = o->ptr; + hdr = kv->ptr; + updateKeysizesHist(c->db, getKeySlot(c->argv[1]->ptr), OBJ_STRING, oldlen, stringObjectLen(kv)); if (updated) { HLL_INVALIDATE_CACHE(hdr); signalModifiedKey(c,c->db,c->argv[1]); @@ -1461,7 +1461,6 @@ void pfaddCommand(client *c) { /* PFCOUNT var -> approximated cardinality of set. */ void pfcountCommand(client *c) { - robj *o; struct hllhdr *hdr; uint64_t card; @@ -1480,7 +1479,7 @@ void pfcountCommand(client *c) { registers = max + HLL_HDR_SIZE; for (j = 1; j < c->argc; j++) { /* Check type and size. */ - robj *o = lookupKeyRead(c->db,c->argv[j]); + kvobj *o = lookupKeyRead(c->db,c->argv[j]); if (o == NULL) continue; /* Assume empty HLL for non existing var.*/ if (isHLLObjectOrReply(c,o) != C_OK) return; @@ -1508,7 +1507,7 @@ void pfcountCommand(client *c) { * logically expired key on a replica is deleted, while with lookupKeyRead * it isn't, but the lookup returns NULL either way if the key is logically * expired, which is what matters here. */ - o = lookupKeyRead(c->db,c->argv[1]); + kvobj *o = lookupKeyRead(c->db, c->argv[1]); if (o == NULL) { /* No key? Cardinality is zero since no element was added, otherwise * we would have a key as HLLADD creates it as a side effect. */ @@ -1568,9 +1567,9 @@ void pfmergeCommand(client *c) { memset(max,0,sizeof(max)); for (j = 1; j < c->argc; j++) { /* Check type and size. */ - robj *o = lookupKeyRead(c->db,c->argv[j]); + kvobj *o = lookupKeyRead(c->db, c->argv[j]); if (o == NULL) continue; /* Assume empty HLL for non existing var. */ - if (isHLLObjectOrReply(c,o) != C_OK) return; + if (isHLLObjectOrReply(c, o) != C_OK) return; /* If at least one involved HLL is dense, use the dense representation * as target ASAP to save time and avoid the conversion step. */ @@ -1586,25 +1585,26 @@ void pfmergeCommand(client *c) { } /* Create / unshare the destination key's value if needed. */ - robj *o = lookupKeyWrite(c->db,c->argv[1]); - if (o == NULL) { + dictEntryLink link; + kvobj *kv = lookupKeyWriteWithLink(c->db,c->argv[1],&link); + if (kv == NULL) { /* Create the key with a string value of the exact length to * hold our HLL data structure. sdsnewlen() when NULL is passed * is guaranteed to return bytes initialized to zero. */ - o = createHLLObject(); - dbAdd(c->db,c->argv[1],o); + robj *o = createHLLObject(); + kv = dbAddByLink(c->db, c->argv[1], &o, &link); } else { /* If key exists we are sure it's of the right type/size * since we checked when merging the different HLLs, so we * don't check again. */ - o = dbUnshareStringValue(c->db,c->argv[1],o); + kv = dbUnshareStringValue(c->db,c->argv[1],kv); } - uint64_t oldLen = stringObjectLen(o); + uint64_t oldLen = stringObjectLen(kv); /* Convert the destination object to dense representation if at least * one of the inputs was dense. */ - if (use_dense && hllSparseToDense(o) == C_ERR) { + if (use_dense && hllSparseToDense(kv) == C_ERR) { addReplyError(c,invalid_hll_err); return; } @@ -1612,19 +1612,19 @@ void pfmergeCommand(client *c) { /* Write the resulting HLL to the destination HLL registers and * invalidate the cached value. */ if (use_dense) { - hdr = o->ptr; + hdr = kv->ptr; hllDenseCompress(hdr->registers, max); } else { for (j = 0; j < HLL_REGISTERS; j++) { if (max[j] == 0) continue; - hdr = o->ptr; + hdr = kv->ptr; switch (hdr->encoding) { case HLL_DENSE: hllDenseSet(hdr->registers,j,max[j]); break; - case HLL_SPARSE: hllSparseSet(o,j,max[j]); break; + case HLL_SPARSE: hllSparseSet(kv,j,max[j]); break; } } } - hdr = o->ptr; /* o->ptr may be different now, as a side effect of + hdr = kv->ptr; /* o->ptr may be different now, as a side effect of last hllSparseSet() call. */ HLL_INVALIDATE_CACHE(hdr); @@ -1634,7 +1634,7 @@ void pfmergeCommand(client *c) { notifyKeyspaceEvent(NOTIFY_STRING,"pfadd",c->argv[1],c->db->id); updateKeysizesHist(c->db, getKeySlot(c->argv[1]->ptr), - OBJ_STRING, oldLen, stringObjectLen(o)); + OBJ_STRING, oldLen, stringObjectLen(kv)); server.dirty++; addReply(c,shared.ok); } @@ -1758,7 +1758,7 @@ cleanup: void pfdebugCommand(client *c) { char *cmd = c->argv[1]->ptr; struct hllhdr *hdr; - robj *o; + kvobj *o; int j; if (!strcasecmp(cmd, "simd")) { diff --git a/src/kvstore.c b/src/kvstore.c index 9ad4c7535..8aded921f 100644 --- a/src/kvstore.c +++ b/src/kvstore.c @@ -132,7 +132,9 @@ static int getAndClearDictIndexFromCursor(kvstore *kvs, unsigned long long *curs /* Updates binary index tree (also known as Fenwick tree), increasing key count for a given dict. * You can read more about this data structure here https://en.wikipedia.org/wiki/Fenwick_tree - * Time complexity is O(log(kvs->num_dicts)). */ + * Time complexity is O(log(kvs->num_dicts)). Take care to call it only after + * adding or removing keys from the kvstore. + */ static void cumulativeKeyCountAdd(kvstore *kvs, int didx, long delta) { kvs->key_count += delta; @@ -373,7 +375,7 @@ size_t kvstoreMemUsage(kvstore *kvs) { metaSize = sizeof(kvstoreDictMetaEx); unsigned long long keys_count = kvstoreSize(kvs); - mem += keys_count * dictEntryMemUsage() + + mem += keys_count * dictEntryMemUsage(kvs->dtype.no_value) + kvstoreBuckets(kvs) * sizeof(dictEntry*) + kvs->allocated_dicts * (sizeof(dict) + metaSize); @@ -775,14 +777,6 @@ dictEntry *kvstoreDictGetFairRandomKey(kvstore *kvs, int didx) return dictGetFairRandomKey(d); } -dictEntry *kvstoreDictFindByHashAndPtr(kvstore *kvs, int didx, const void *oldptr, uint64_t hash) -{ - dict *d = kvstoreGetDict(kvs, didx); - if (!d) - return NULL; - return dictFindByHashAndPtr(d, oldptr, hash); -} - unsigned int kvstoreDictGetSomeKeys(kvstore *kvs, int didx, dictEntry **des, unsigned int count) { dict *d = kvstoreGetDict(kvs, didx); @@ -836,16 +830,12 @@ unsigned long kvstoreDictLUTDefrag(kvstore *kvs, unsigned long cursor, kvstoreDi return 0; } -uint64_t kvstoreGetHash(kvstore *kvs, const void *key) -{ - return kvs->dtype.hashFunction(key); -} - void *kvstoreDictFetchValue(kvstore *kvs, int didx, const void *key) { dict *d = kvstoreGetDict(kvs, didx); if (!d) return NULL; + assert(d->type->no_value == 0); return dictFetchValue(d, key); } @@ -856,9 +846,67 @@ dictEntry *kvstoreDictFind(kvstore *kvs, int didx, void *key) { return dictFind(d, key); } +/* Find a link to a key in the specified kvstore. If not found return NULL. + * + * This function is a wrapper around dictFindLink(), used to locate a key in a dict + * from a kvstore. + * + * The caller may provide a bucket pointer to receive the reference to the bucket + * where the key is stored or need to be added. + * + * Returns: + * A reference to the dictEntry if found, otherwise NULL. + * + * Important: + * After calling kvstoreDictFindLink(), any necessary updates based on returned + * link or bucket must be made immediately after, commonly by kvstoreDictSetAtLink() + * without any operations in between that might modify the dict. Otherwise, + * the link or bucket may become invalid. Example usage: + * + * link = kvstoreDictFindLink(kvs, didx, key, &bucket); + * ... Do something, but don't modify kvs->dicts[didx] ... + * if (link) + * kvstoreDictSetAtLink(kvs, didx, kv, &link, 0); // Update existing entry + * else + * kvstoreDictSetAtLink(kvs, didx, kv, &bucket, 1); // Insert new entry + */ +dictEntryLink kvstoreDictFindLink(kvstore *kvs, int didx, void *key, dictEntryLink *bucket) { + if (bucket) *bucket = NULL; + dict *d = kvstoreGetDict(kvs, didx); + if (!d) return NULL; + return dictFindLink(d, key, bucket); +} + +/* Set a key (or key-value) in the specified kvstore. + * + * This function inserts a new key or updates an existing one, depending on + * the `newItem` flag. + * + * Parameters: + * link: - When `newItem` is set, `link` points to the bucket of the key. + * - When `newItem` is not set, `link` points to the link of the key. + * - If link is NULL, dictFindLink() will be called to locate the link. + * + * newItem: - If set, add a new key with a new dictEntry. + * - If not set, update the key of an existing dictEntry. + */ +void kvstoreDictSetAtLink(kvstore *kvs, int didx, void *kv, dictEntryLink *link, int newItem) { + dict *d; + if (newItem) { + d = createDictIfNeeded(kvs, didx); + dictSetKeyAtLink(d, kv, link, newItem); + cumulativeKeyCountAdd(kvs, didx, 1); /* must be called only after updating dict */ + } else { + d = kvstoreGetDict(kvs, didx); + dictSetKeyAtLink(d, kv, link, newItem); + } +} + dictEntry *kvstoreDictAddRaw(kvstore *kvs, int didx, void *key, dictEntry **existing) { dict *d = createDictIfNeeded(kvs, didx); + dictUseStoredKeyApi(d, 1); dictEntry *ret = dictAddRaw(d, key, existing); + dictUseStoredKeyApi(d, 0); if (ret) cumulativeKeyCountAdd(kvs, didx, 1); return ret; @@ -871,19 +919,20 @@ void kvstoreDictSetKey(kvstore *kvs, int didx, dictEntry* de, void *key) { void kvstoreDictSetVal(kvstore *kvs, int didx, dictEntry *de, void *val) { dict *d = kvstoreGetDict(kvs, didx); + assert(d->type->no_value == 0); dictSetVal(d, de, val); } -dictEntry *kvstoreDictTwoPhaseUnlinkFind(kvstore *kvs, int didx, const void *key, dictEntry ***plink, int *table_index) { +dictEntryLink kvstoreDictTwoPhaseUnlinkFind(kvstore *kvs, int didx, const void *key, int *table_index) { dict *d = kvstoreGetDict(kvs, didx); if (!d) return NULL; - return dictTwoPhaseUnlinkFind(kvstoreGetDict(kvs, didx), key, plink, table_index); + return dictTwoPhaseUnlinkFind(kvstoreGetDict(kvs, didx), key, table_index); } -void kvstoreDictTwoPhaseUnlinkFree(kvstore *kvs, int didx, dictEntry *he, dictEntry **plink, int table_index) { +void kvstoreDictTwoPhaseUnlinkFree(kvstore *kvs, int didx, dictEntryLink link, int table_index) { dict *d = kvstoreGetDict(kvs, didx); - dictTwoPhaseUnlinkFree(d, he, plink, table_index); + dictTwoPhaseUnlinkFree(d, link, table_index); cumulativeKeyCountAdd(kvs, didx, -1); freeDictIfNeeded(kvs, didx); } @@ -981,10 +1030,14 @@ int kvstoreTest(int argc, char **argv, int flags) { kvstoreIterator *kvs_it; kvstoreDictIterator *kvs_di; + /* Test also dictType with no_value=1 */ + dictType KvstoreDictNovalTestType = KvstoreDictTestType; + KvstoreDictNovalTestType.no_value = 1; + int didx = 0; int curr_slot = 0; kvstore *kvs1 = kvstoreCreate(&KvstoreDictTestType, 0, KVSTORE_ALLOCATE_DICTS_ON_DEMAND); - kvstore *kvs2 = kvstoreCreate(&KvstoreDictTestType, 0, KVSTORE_ALLOCATE_DICTS_ON_DEMAND | KVSTORE_FREE_EMPTY_DICTS); + kvstore *kvs2 = kvstoreCreate(&KvstoreDictNovalTestType, 0, KVSTORE_ALLOCATE_DICTS_ON_DEMAND | KVSTORE_FREE_EMPTY_DICTS); TEST("Add 16 keys") { for (i = 0; i < 16; i++) { diff --git a/src/kvstore.h b/src/kvstore.h index d634f9755..3a1295fd4 100644 --- a/src/kvstore.h +++ b/src/kvstore.h @@ -63,7 +63,6 @@ int kvstoreGetNextNonEmptyDictIndex(kvstore *kvs, int didx); int kvstoreNumNonEmptyDicts(kvstore *kvs); int kvstoreNumAllocatedDicts(kvstore *kvs); int kvstoreNumDicts(kvstore *kvs); -uint64_t kvstoreGetHash(kvstore *kvs, const void *key); /* kvstore iterator specific functions */ kvstoreIterator *kvstoreIteratorInit(kvstore *kvs); @@ -87,7 +86,6 @@ void kvstoreReleaseDictIterator(kvstoreDictIterator *kvs_id); dictEntry *kvstoreDictIteratorNext(kvstoreDictIterator *kvs_di); dictEntry *kvstoreDictGetRandomKey(kvstore *kvs, int didx); dictEntry *kvstoreDictGetFairRandomKey(kvstore *kvs, int didx); -dictEntry *kvstoreDictFindByHashAndPtr(kvstore *kvs, int didx, const void *oldptr, uint64_t hash); unsigned int kvstoreDictGetSomeKeys(kvstore *kvs, int didx, dictEntry **des, unsigned int count); int kvstoreDictExpand(kvstore *kvs, int didx, unsigned long size); unsigned long kvstoreDictScanDefrag(kvstore *kvs, int didx, unsigned long v, dictScanFunction *fn, dictDefragFunctions *defragfns, void *privdata); @@ -96,14 +94,20 @@ unsigned long kvstoreDictLUTDefrag(kvstore *kvs, unsigned long cursor, kvstoreDi void *kvstoreDictFetchValue(kvstore *kvs, int didx, const void *key); dictEntry *kvstoreDictFind(kvstore *kvs, int didx, void *key); dictEntry *kvstoreDictAddRaw(kvstore *kvs, int didx, void *key, dictEntry **existing); -void kvstoreDictSetKey(kvstore *kvs, int didx, dictEntry* de, void *key); -void kvstoreDictSetVal(kvstore *kvs, int didx, dictEntry *de, void *val); -dictEntry *kvstoreDictTwoPhaseUnlinkFind(kvstore *kvs, int didx, const void *key, dictEntry ***plink, int *table_index); -void kvstoreDictTwoPhaseUnlinkFree(kvstore *kvs, int didx, dictEntry *he, dictEntry **plink, int table_index); +dictEntryLink kvstoreDictTwoPhaseUnlinkFind(kvstore *kvs, int didx, const void *key, int *table_index); +void kvstoreDictTwoPhaseUnlinkFree(kvstore *kvs, int didx, dictEntryLink plink, int table_index); int kvstoreDictDelete(kvstore *kvs, int didx, const void *key); kvstoreDictMetadata *kvstoreGetDictMetadata(kvstore *kvs, int didx); kvstoreMetadata *kvstoreGetMetadata(kvstore *kvs); +dictEntryLink kvstoreDictFindLink(kvstore *kvs, int didx, void *key, dictEntryLink *bucket); +void kvstoreDictSetAtLink(kvstore *kvs, int didx, void *kv, dictEntryLink *link, int newItem); + +/* dict with distinct key & value (no_value=1) currently is used only by pubsub. */ +void kvstoreDictSetKey(kvstore *kvs, int didx, dictEntry* de, void *key); +void kvstoreDictSetVal(kvstore *kvs, int didx, dictEntry *de, void *val); +void *kvstoreDictFetchValue(kvstore *kvs, int didx, const void *key); + #ifdef REDIS_TEST int kvstoreTest(int argc, char *argv[], int flags); #endif diff --git a/src/latency.c b/src/latency.c index 09639bd50..1cb82b5da 100644 --- a/src/latency.c +++ b/src/latency.c @@ -17,8 +17,8 @@ #include "hdr_histogram.h" /* Dictionary type for latency events. */ -int dictStringKeyCompare(dict *d, const void *key1, const void *key2) { - UNUSED(d); +int dictStringKeyCompare(dictCmpCache *cache, const void *key1, const void *key2) { + UNUSED(cache); return strcmp(key1,key2) == 0; } diff --git a/src/module.c b/src/module.c index ed28cc071..aea3694f9 100644 --- a/src/module.c +++ b/src/module.c @@ -176,7 +176,7 @@ struct RedisModuleKey { RedisModuleCtx *ctx; redisDb *db; robj *key; /* Key name object. */ - robj *value; /* Value object, or NULL if the key was not found. */ + kvobj *kv; /* Key-Value object, or NULL if the key was not found. */ void *iter; /* Iterator. */ int mode; /* Opening mode. */ @@ -696,7 +696,7 @@ int moduleCreateEmptyKey(RedisModuleKey *key, int type) { robj *obj; /* The key must be open for writing and non existing to proceed. */ - if (!(key->mode & REDISMODULE_WRITE) || key->value) + if (!(key->mode & REDISMODULE_WRITE) || key->kv) return REDISMODULE_ERR; switch(type) { @@ -714,8 +714,8 @@ int moduleCreateEmptyKey(RedisModuleKey *key, int type) { break; default: return REDISMODULE_ERR; } - dbAdd(key->db,key->key,obj); - key->value = obj; + + key->kv = dbAdd(key->db, key->key, &obj); moduleInitKeyTypeSpecific(key); return REDISMODULE_OK; } @@ -723,7 +723,7 @@ int moduleCreateEmptyKey(RedisModuleKey *key, int type) { /* Frees key->iter and sets it to NULL. */ static void moduleFreeKeyIterator(RedisModuleKey *key) { serverAssert(key->iter != NULL); - switch (key->value->type) { + switch (key->kv->type) { case OBJ_LIST: listTypeReleaseIterator(key->iter); break; case OBJ_STREAM: streamIteratorStop(key->iter); @@ -738,7 +738,7 @@ static void moduleFreeKeyIterator(RedisModuleKey *key) { * Frees list iterator and sets it to NULL. */ static void moduleFreeListIterator(void *data) { RedisModuleKey *key = (RedisModuleKey*)data; - serverAssert(key->value->type == OBJ_LIST); + serverAssert(key->kv->type == OBJ_LIST); if (key->iter) moduleFreeKeyIterator(key); } @@ -753,9 +753,9 @@ static void moduleFreeListIterator(void *data) { * The function returns 1 if the key value object is found empty and is * deleted, otherwise 0 is returned. */ int moduleDelKeyIfEmpty(RedisModuleKey *key) { - if (!(key->mode & REDISMODULE_WRITE) || key->value == NULL) return 0; + if (!(key->mode & REDISMODULE_WRITE) || key->kv == NULL) return 0; int isempty; - robj *o = key->value; + robj *o = key->kv; switch(o->type) { case OBJ_LIST: isempty = listTypeLength(o) == 0; break; @@ -769,7 +769,7 @@ int moduleDelKeyIfEmpty(RedisModuleKey *key) { if (isempty) { if (key->iter) moduleFreeKeyIterator(key); dbDelete(key->db,key->key); - key->value = NULL; + key->kv = NULL; return 1; } else { return 0; @@ -4069,25 +4069,25 @@ int RM_SelectDb(RedisModuleCtx *ctx, int newid) { * calling RM_CloseKey on the opened key. */ int RM_KeyExists(RedisModuleCtx *ctx, robj *keyname) { - robj *value = lookupKeyReadWithFlags(ctx->client->db, keyname, LOOKUP_NOTOUCH); - return (value != NULL); + kvobj *kv = lookupKeyReadWithFlags(ctx->client->db, keyname, LOOKUP_NOTOUCH); + return (kv != NULL); } /* Initialize a RedisModuleKey struct */ -static void moduleInitKey(RedisModuleKey *kp, RedisModuleCtx *ctx, robj *keyname, robj *value, int mode){ +static void moduleInitKey(RedisModuleKey *kp, RedisModuleCtx *ctx, robj *keyname, kvobj *kv, int mode){ kp->ctx = ctx; kp->db = ctx->client->db; kp->key = keyname; incrRefCount(keyname); - kp->value = value; + kp->kv = kv; kp->iter = NULL; kp->mode = mode; - if (kp->value) moduleInitKeyTypeSpecific(kp); + if (kp->kv) moduleInitKeyTypeSpecific(kp); } /* Initialize the type-specific part of the key. Only when key has a value. */ static void moduleInitKeyTypeSpecific(RedisModuleKey *key) { - switch (key->value->type) { + switch (key->kv->type) { case OBJ_ZSET: zsetKeyReset(key); break; case OBJ_STREAM: key->u.stream.signalready = 0; break; } @@ -4117,7 +4117,7 @@ static void moduleInitKeyTypeSpecific(RedisModuleKey *key) { * * REDISMODULE_OPEN_KEY_ACCESS_EXPIRED - Access expired keys that have not yet been deleted */ RedisModuleKey *RM_OpenKey(RedisModuleCtx *ctx, robj *keyname, int mode) { RedisModuleKey *kp; - robj *value; + kvobj *kv; int flags = 0; flags |= (mode & REDISMODULE_OPEN_KEY_NOTOUCH? LOOKUP_NOTOUCH: 0); flags |= (mode & REDISMODULE_OPEN_KEY_NONOTIFY? LOOKUP_NONOTIFY: 0); @@ -4127,17 +4127,17 @@ RedisModuleKey *RM_OpenKey(RedisModuleCtx *ctx, robj *keyname, int mode) { flags |= (mode & REDISMODULE_OPEN_KEY_ACCESS_EXPIRED ? (LOOKUP_ACCESS_EXPIRED) : 0); if (mode & REDISMODULE_WRITE) { - value = lookupKeyWriteWithFlags(ctx->client->db,keyname, flags); + kv = lookupKeyWriteWithFlags(ctx->client->db,keyname, flags); } else { - value = lookupKeyReadWithFlags(ctx->client->db,keyname, flags); - if (value == NULL) { + kv = lookupKeyReadWithFlags(ctx->client->db,keyname, flags); + if (kv == NULL) { return NULL; } } /* Setup the key handle. */ kp = zmalloc(sizeof(*kp)); - moduleInitKey(kp, ctx, keyname, value, mode); + moduleInitKey(kp, ctx, keyname, kv, mode); autoMemoryAdd(ctx,REDISMODULE_AM_KEY,kp); return kp; } @@ -4164,9 +4164,9 @@ static void moduleCloseKey(RedisModuleKey *key) { int signal = SHOULD_SIGNAL_MODIFIED_KEYS(key->ctx); if ((key->mode & REDISMODULE_WRITE) && signal) signalModifiedKey(key->ctx->client,key->db,key->key); - if (key->value) { + if (key->kv) { if (key->iter) moduleFreeKeyIterator(key); - switch (key->value->type) { + switch (key->kv->type) { case OBJ_ZSET: RM_ZsetRangeStop(key); break; @@ -4192,10 +4192,10 @@ void RM_CloseKey(RedisModuleKey *key) { /* Return the type of the key. If the key pointer is NULL then * REDISMODULE_KEYTYPE_EMPTY is returned. */ int RM_KeyType(RedisModuleKey *key) { - if (key == NULL || key->value == NULL) return REDISMODULE_KEYTYPE_EMPTY; + if (key == NULL || key->kv == NULL) return REDISMODULE_KEYTYPE_EMPTY; /* We map between defines so that we are free to change the internal * defines as desired. */ - switch(key->value->type) { + switch(key->kv->type) { case OBJ_STRING: return REDISMODULE_KEYTYPE_STRING; case OBJ_LIST: return REDISMODULE_KEYTYPE_LIST; case OBJ_SET: return REDISMODULE_KEYTYPE_SET; @@ -4213,8 +4213,8 @@ int RM_KeyType(RedisModuleKey *key) { * * If the key pointer is NULL or the key is empty, zero is returned. */ size_t RM_ValueLength(RedisModuleKey *key) { - if (key == NULL || key->value == NULL) return 0; - return getObjectLength(key->value); + if (key == NULL || key->kv == NULL) return 0; + return getObjectLength(key->kv); } /* If the key is open for writing, remove it, and setup the key to @@ -4223,9 +4223,9 @@ size_t RM_ValueLength(RedisModuleKey *key) { * writing REDISMODULE_ERR is returned. */ int RM_DeleteKey(RedisModuleKey *key) { if (!(key->mode & REDISMODULE_WRITE)) return REDISMODULE_ERR; - if (key->value) { + if (key->kv) { dbDelete(key->db,key->key); - key->value = NULL; + key->kv = NULL; } return REDISMODULE_OK; } @@ -4237,9 +4237,9 @@ int RM_DeleteKey(RedisModuleKey *key) { * writing REDISMODULE_ERR is returned. */ int RM_UnlinkKey(RedisModuleKey *key) { if (!(key->mode & REDISMODULE_WRITE)) return REDISMODULE_ERR; - if (key->value) { + if (key->kv) { dbAsyncDelete(key->db,key->key); - key->value = NULL; + key->kv = NULL; } return REDISMODULE_OK; } @@ -4248,8 +4248,8 @@ int RM_UnlinkKey(RedisModuleKey *key) { * If no TTL is associated with the key or if the key is empty, * REDISMODULE_NO_EXPIRE is returned. */ mstime_t RM_GetExpire(RedisModuleKey *key) { - mstime_t expire = getExpire(key->db,key->key); - if (expire == -1 || key->value == NULL) + mstime_t expire = kvobjGetExpire(key->kv); + if (expire == -1 || key->kv == NULL) return REDISMODULE_NO_EXPIRE; expire -= commandTimeSnapshot(); return expire >= 0 ? expire : 0; @@ -4265,7 +4265,7 @@ mstime_t RM_GetExpire(RedisModuleKey *key) { * The function returns REDISMODULE_OK on success or REDISMODULE_ERR if * the key was not open for writing or is an empty key. */ int RM_SetExpire(RedisModuleKey *key, mstime_t expire) { - if (!(key->mode & REDISMODULE_WRITE) || key->value == NULL || (expire < 0 && expire != REDISMODULE_NO_EXPIRE)) + if (!(key->mode & REDISMODULE_WRITE) || key->kv == NULL || (expire < 0 && expire != REDISMODULE_NO_EXPIRE)) return REDISMODULE_ERR; if (expire != REDISMODULE_NO_EXPIRE) { expire += commandTimeSnapshot(); @@ -4280,8 +4280,8 @@ int RM_SetExpire(RedisModuleKey *key, mstime_t expire) { * If no TTL is associated with the key or if the key is empty, * REDISMODULE_NO_EXPIRE is returned. */ mstime_t RM_GetAbsExpire(RedisModuleKey *key) { - mstime_t expire = getExpire(key->db,key->key); - if (expire == -1 || key->value == NULL) + mstime_t expire = kvobjGetExpire(key->kv); + if (expire == -1 || key->kv == NULL) return REDISMODULE_NO_EXPIRE; return expire; } @@ -4296,10 +4296,10 @@ mstime_t RM_GetAbsExpire(RedisModuleKey *key) { * The function returns REDISMODULE_OK on success or REDISMODULE_ERR if * the key was not open for writing or is an empty key. */ int RM_SetAbsExpire(RedisModuleKey *key, mstime_t expire) { - if (!(key->mode & REDISMODULE_WRITE) || key->value == NULL || (expire < 0 && expire != REDISMODULE_NO_EXPIRE)) + if (!(key->mode & REDISMODULE_WRITE) || key->kv == NULL || (expire < 0 && expire != REDISMODULE_NO_EXPIRE)) return REDISMODULE_ERR; if (expire != REDISMODULE_NO_EXPIRE) { - setExpire(key->ctx->client,key->db,key->key,expire); + key->kv = setExpire(key->ctx->client,key->db,key->key,expire); } else { removeExpire(key->db,key->key); } @@ -4360,8 +4360,10 @@ int RM_GetToDbIdFromOptCtx(RedisModuleKeyOptCtx *ctx) { int RM_StringSet(RedisModuleKey *key, RedisModuleString *str) { if (!(key->mode & REDISMODULE_WRITE) || key->iter) return REDISMODULE_ERR; RM_DeleteKey(key); - setKey(key->ctx->client,key->db,key->key,str,SETKEY_NO_SIGNAL); - key->value = str; + /* Retain str so setKey copies it to db rather than reallocating it. */ + incrRefCount(str); + setKey(key->ctx->client,key->db,key->key,&str,SETKEY_NO_SIGNAL); + key->kv = str; return REDISMODULE_OK; } @@ -4400,20 +4402,20 @@ char *RM_StringDMA(RedisModuleKey *key, size_t *len, int mode) { * a read only memory page, so the module will segfault if a write * attempt is performed. */ char *emptystring = ""; - if (key->value == NULL) { + if (key->kv == NULL) { *len = 0; return emptystring; } - if (key->value->type != OBJ_STRING) return NULL; + if (key->kv->type != OBJ_STRING) return NULL; /* For write access, and even for read access if the object is encoded, * we unshare the string (that has the side effect of decoding it). */ - if ((mode & REDISMODULE_WRITE) || key->value->encoding != OBJ_ENCODING_RAW) - key->value = dbUnshareStringValue(key->db, key->key, key->value); + if ((mode & REDISMODULE_WRITE) || key->kv->encoding != OBJ_ENCODING_RAW) + key->kv = dbUnshareStringValue(key->db, key->key, key->kv); - *len = sdslen(key->value->ptr); - return key->value->ptr; + *len = sdslen(key->kv->ptr); + return key->kv->ptr; } /* If the key is open for writing and is of string type, resize it, padding @@ -4430,30 +4432,29 @@ char *RM_StringDMA(RedisModuleKey *key, size_t *len, int mode) { * unless the new length value requested is zero. */ int RM_StringTruncate(RedisModuleKey *key, size_t newlen) { if (!(key->mode & REDISMODULE_WRITE)) return REDISMODULE_ERR; - if (key->value && key->value->type != OBJ_STRING) return REDISMODULE_ERR; + if (key->kv && key->kv->type != OBJ_STRING) return REDISMODULE_ERR; if (newlen > 512*1024*1024) return REDISMODULE_ERR; /* Empty key and new len set to 0. Just return REDISMODULE_OK without * doing anything. */ - if (key->value == NULL && newlen == 0) return REDISMODULE_OK; + if (key->kv == NULL && newlen == 0) return REDISMODULE_OK; - if (key->value == NULL) { + if (key->kv == NULL) { /* Empty key: create it with the new size. */ robj *o = createObject(OBJ_STRING,sdsnewlen(NULL, newlen)); - setKey(key->ctx->client,key->db,key->key,o,SETKEY_NO_SIGNAL); - key->value = o; - decrRefCount(o); + setKey(key->ctx->client, key->db, key->key, &o, SETKEY_NO_SIGNAL); + key->kv = o; } else { /* Unshare and resize. */ - key->value = dbUnshareStringValue(key->db, key->key, key->value); - size_t curlen = sdslen(key->value->ptr); + key->kv = dbUnshareStringValue(key->db, key->key, key->kv); + size_t curlen = sdslen(key->kv->ptr); if (newlen > curlen) { - key->value->ptr = sdsgrowzero(key->value->ptr,newlen); + key->kv->ptr = sdsgrowzero(key->kv->ptr,newlen); } else if (newlen < curlen) { - sdssubstr(key->value->ptr,0,newlen); + sdssubstr(key->kv->ptr,0,newlen); /* If the string is too wasteful, reallocate it. */ - if (sdslen(key->value->ptr) < sdsavail(key->value->ptr)) - key->value->ptr = sdsRemoveFreeSpace(key->value->ptr, 0); + if (sdslen(key->kv->ptr) < sdsavail(key->kv->ptr)) + key->kv->ptr = sdsRemoveFreeSpace(key->kv->ptr, 0); } } return REDISMODULE_OK; @@ -4495,7 +4496,7 @@ int moduleListIteratorSeek(RedisModuleKey *key, long index, int mode) { if (!key) { errno = EINVAL; return 0; - } else if (!key->value || key->value->type != OBJ_LIST) { + } else if (!key->kv || key->kv->type != OBJ_LIST) { errno = ENOTSUP; return 0; } if (!(key->mode & mode)) { @@ -4503,7 +4504,7 @@ int moduleListIteratorSeek(RedisModuleKey *key, long index, int mode) { return 0; } - long length = listTypeLength(key->value); + long length = listTypeLength(key->kv); if (index < -length || index >= length) { errno = EDOM; /* Invalid index */ return 0; @@ -4511,7 +4512,7 @@ int moduleListIteratorSeek(RedisModuleKey *key, long index, int mode) { if (key->iter == NULL) { /* No existing iterator. Create one. */ - key->iter = listTypeInitIterator(key->value, index, LIST_TAIL); + key->iter = listTypeInitIterator(key->kv, index, LIST_TAIL); serverAssert(key->iter != NULL); serverAssert(listTypeNext(key->iter, &key->u.list.entry)); key->u.list.index = index; @@ -4550,7 +4551,7 @@ int RM_ListPush(RedisModuleKey *key, int where, RedisModuleString *ele) { if (!key || !ele) { errno = EINVAL; return REDISMODULE_ERR; - } else if (key->value != NULL && key->value->type != OBJ_LIST) { + } else if (key->kv != NULL && key->kv->type != OBJ_LIST) { errno = ENOTSUP; return REDISMODULE_ERR; } if (!(key->mode & REDISMODULE_WRITE)) { @@ -4559,13 +4560,13 @@ int RM_ListPush(RedisModuleKey *key, int where, RedisModuleString *ele) { } if (!(key->mode & REDISMODULE_WRITE)) return REDISMODULE_ERR; - if (key->value && key->value->type != OBJ_LIST) return REDISMODULE_ERR; + if (key->kv && key->kv->type != OBJ_LIST) return REDISMODULE_ERR; if (key->iter) moduleFreeKeyIterator(key); - if (key->value == NULL) moduleCreateEmptyKey(key,REDISMODULE_KEYTYPE_LIST); - listTypeTryConversionAppend(key->value, &ele, 0, 0, moduleFreeListIterator, key); - listTypePush(key->value, ele, + if (key->kv == NULL) moduleCreateEmptyKey(key,REDISMODULE_KEYTYPE_LIST); + listTypeTryConversionAppend(key->kv, &ele, 0, 0, moduleFreeListIterator, key); + listTypePush(key->kv, ele, (where == REDISMODULE_LIST_HEAD) ? LIST_HEAD : LIST_TAIL); - int64_t l = listTypeLength(key->value); + int64_t l = listTypeLength(key->kv); updateKeysizesHist(key->db, getKeySlot(key->key->ptr), OBJ_LIST, l-1, l); return REDISMODULE_OK; } @@ -4586,7 +4587,7 @@ RedisModuleString *RM_ListPop(RedisModuleKey *key, int where) { if (!key) { errno = EINVAL; return NULL; - } else if (key->value == NULL || key->value->type != OBJ_LIST) { + } else if (key->kv == NULL || key->kv->type != OBJ_LIST) { errno = ENOTSUP; return NULL; } else if (!(key->mode & REDISMODULE_WRITE)) { @@ -4594,14 +4595,14 @@ RedisModuleString *RM_ListPop(RedisModuleKey *key, int where) { return NULL; } if (key->iter) moduleFreeKeyIterator(key); - robj *ele = listTypePop(key->value, + robj *ele = listTypePop(key->kv, (where == REDISMODULE_LIST_HEAD) ? LIST_HEAD : LIST_TAIL); robj *decoded = getDecodedObject(ele); decrRefCount(ele); - int64_t l = (int64_t) listTypeLength(key->value); + int64_t l = (int64_t) listTypeLength(key->kv); updateKeysizesHist(key->db, getKeySlot(key->key->ptr), OBJ_LIST, l+1, l); if (!moduleDelKeyIfEmpty(key)) - listTypeTryConversion(key->value, LIST_CONV_SHRINKING, moduleFreeListIterator, key); + listTypeTryConversion(key->kv, LIST_CONV_SHRINKING, moduleFreeListIterator, key); autoMemoryAdd(key->ctx,REDISMODULE_AM_STRING,decoded); return decoded; } @@ -4655,11 +4656,11 @@ int RM_ListSet(RedisModuleKey *key, long index, RedisModuleString *value) { errno = EINVAL; return REDISMODULE_ERR; } - if (!key->value || key->value->type != OBJ_LIST) { + if (!key->kv || key->kv->type != OBJ_LIST) { errno = ENOTSUP; return REDISMODULE_ERR; } - listTypeTryConversionAppend(key->value, &value, 0, 0, moduleFreeListIterator, key); + listTypeTryConversionAppend(key->kv, &value, 0, 0, moduleFreeListIterator, key); if (moduleListIteratorSeek(key, index, REDISMODULE_WRITE)) { listTypeReplace(&key->u.list.entry, value); /* A note in quicklist.c forbids use of iterator after insert, so @@ -4690,22 +4691,22 @@ int RM_ListInsert(RedisModuleKey *key, long index, RedisModuleString *value) { if (!value) { errno = EINVAL; return REDISMODULE_ERR; - } else if (key != NULL && key->value == NULL && + } else if (key != NULL && key->kv == NULL && (index == 0 || index == -1)) { /* Insert in empty key => push. */ return RM_ListPush(key, REDISMODULE_LIST_TAIL, value); - } else if (key != NULL && key->value != NULL && - key->value->type == OBJ_LIST && - (index == (long)listTypeLength(key->value) || index == -1)) { + } else if (key != NULL && key->kv != NULL && + key->kv->type == OBJ_LIST && + (index == (long)listTypeLength(key->kv) || index == -1)) { /* Insert after the last element => push tail. */ return RM_ListPush(key, REDISMODULE_LIST_TAIL, value); - } else if (key != NULL && key->value != NULL && - key->value->type == OBJ_LIST && - (index == 0 || index == -(long)listTypeLength(key->value) - 1)) { + } else if (key != NULL && key->kv != NULL && + key->kv->type == OBJ_LIST && + (index == 0 || index == -(long)listTypeLength(key->kv) - 1)) { /* Insert before the first element => push head. */ return RM_ListPush(key, REDISMODULE_LIST_HEAD, value); } - listTypeTryConversionAppend(key->value, &value, 0, 0, moduleFreeListIterator, key); + listTypeTryConversionAppend(key->kv, &value, 0, 0, moduleFreeListIterator, key); if (moduleListIteratorSeek(key, index, REDISMODULE_WRITE)) { int where = index < 0 ? LIST_TAIL : LIST_HEAD; listTypeInsert(&key->u.list.entry, value, where); @@ -4731,10 +4732,10 @@ int RM_ListInsert(RedisModuleKey *key, long index, RedisModuleString *value) { int RM_ListDelete(RedisModuleKey *key, long index) { if (moduleListIteratorSeek(key, index, REDISMODULE_WRITE)) { listTypeDelete(key->iter, &key->u.list.entry); - int64_t l = (int64_t) listTypeLength(key->value); - updateKeysizesHist(key->db, getKeySlot(key->value->ptr), OBJ_LIST, l+1, l); + int64_t l = (int64_t) listTypeLength(key->kv); + updateKeysizesHist(key->db, getKeySlot(key->kv->ptr), OBJ_LIST, l+1, l); if (moduleDelKeyIfEmpty(key)) return REDISMODULE_OK; - listTypeTryConversion(key->value, LIST_CONV_SHRINKING, moduleFreeListIterator, key); + listTypeTryConversion(key->kv, LIST_CONV_SHRINKING, moduleFreeListIterator, key); if (!key->iter) return REDISMODULE_OK; /* Return ASAP if iterator has been freed */ if (listTypeNext(key->iter, &key->u.list.entry)) { /* After delete entry at position 'index', we need to update @@ -4820,16 +4821,16 @@ int moduleZsetAddFlagsFromCoreFlags(int flags) { int RM_ZsetAdd(RedisModuleKey *key, double score, RedisModuleString *ele, int *flagsptr) { int in_flags = 0, out_flags = 0; if (!(key->mode & REDISMODULE_WRITE)) return REDISMODULE_ERR; - if (key->value && key->value->type != OBJ_ZSET) return REDISMODULE_ERR; - if (key->value == NULL) moduleCreateEmptyKey(key,REDISMODULE_KEYTYPE_ZSET); + if (key->kv && key->kv->type != OBJ_ZSET) return REDISMODULE_ERR; + if (key->kv == NULL) moduleCreateEmptyKey(key,REDISMODULE_KEYTYPE_ZSET); if (flagsptr) in_flags = moduleZsetAddFlagsToCoreFlags(*flagsptr); - if (zsetAdd(key->value,score,ele->ptr,in_flags,&out_flags,NULL) == 0) { + if (zsetAdd(key->kv,score,ele->ptr,in_flags,&out_flags,NULL) == 0) { if (flagsptr) *flagsptr = 0; moduleDelKeyIfEmpty(key); return REDISMODULE_ERR; } if (flagsptr) *flagsptr = moduleZsetAddFlagsFromCoreFlags(out_flags); - int64_t l = (int64_t) zsetLength(key->value); + int64_t l = (int64_t) zsetLength(key->kv); updateKeysizesHist(key->db, getKeySlot(key->key->ptr), OBJ_ZSET, l-1, l); return REDISMODULE_OK; } @@ -4850,18 +4851,18 @@ int RM_ZsetAdd(RedisModuleKey *key, double score, RedisModuleString *ele, int *f int RM_ZsetIncrby(RedisModuleKey *key, double score, RedisModuleString *ele, int *flagsptr, double *newscore) { int in_flags = 0, out_flags = 0; if (!(key->mode & REDISMODULE_WRITE)) return REDISMODULE_ERR; - if (key->value && key->value->type != OBJ_ZSET) return REDISMODULE_ERR; - if (key->value == NULL) moduleCreateEmptyKey(key,REDISMODULE_KEYTYPE_ZSET); + if (key->kv && key->kv->type != OBJ_ZSET) return REDISMODULE_ERR; + if (key->kv == NULL) moduleCreateEmptyKey(key,REDISMODULE_KEYTYPE_ZSET); if (flagsptr) in_flags = moduleZsetAddFlagsToCoreFlags(*flagsptr); in_flags |= ZADD_IN_INCR; - if (zsetAdd(key->value,score,ele->ptr,in_flags,&out_flags,newscore) == 0) { + if (zsetAdd(key->kv,score,ele->ptr,in_flags,&out_flags,newscore) == 0) { if (flagsptr) *flagsptr = 0; moduleDelKeyIfEmpty(key); return REDISMODULE_ERR; } if (out_flags & ZADD_OUT_ADDED) { - int64_t l = (int64_t) zsetLength(key->value); - updateKeysizesHist(key->db, getKeySlot(key->value->ptr), OBJ_ZSET, l-1, l); + int64_t l = (int64_t) zsetLength(key->kv); + updateKeysizesHist(key->db, getKeySlot(key->kv->ptr), OBJ_ZSET, l-1, l); } if (flagsptr) *flagsptr = moduleZsetAddFlagsFromCoreFlags(out_flags); return REDISMODULE_OK; @@ -4887,11 +4888,11 @@ int RM_ZsetIncrby(RedisModuleKey *key, double score, RedisModuleString *ele, int * Empty keys will be handled correctly by doing nothing. */ int RM_ZsetRem(RedisModuleKey *key, RedisModuleString *ele, int *deleted) { if (!(key->mode & REDISMODULE_WRITE)) return REDISMODULE_ERR; - if (key->value && key->value->type != OBJ_ZSET) return REDISMODULE_ERR; - if (key->value != NULL && zsetDel(key->value,ele->ptr)) { + if (key->kv && key->kv->type != OBJ_ZSET) return REDISMODULE_ERR; + if (key->kv != NULL && zsetDel(key->kv,ele->ptr)) { if (deleted) *deleted = 1; - int64_t l = (int64_t) zsetLength(key->value); - updateKeysizesHist(key->db, getKeySlot(key->value->ptr), OBJ_ZSET, l+1, l); + int64_t l = (int64_t) zsetLength(key->kv); + updateKeysizesHist(key->db, getKeySlot(key->kv->ptr), OBJ_ZSET, l+1, l); moduleDelKeyIfEmpty(key); } else { if (deleted) *deleted = 0; @@ -4908,9 +4909,9 @@ int RM_ZsetRem(RedisModuleKey *key, RedisModuleString *ele, int *deleted) { * * The key is an open empty key. */ int RM_ZsetScore(RedisModuleKey *key, RedisModuleString *ele, double *score) { - if (key->value == NULL) return REDISMODULE_ERR; - if (key->value->type != OBJ_ZSET) return REDISMODULE_ERR; - if (zsetScore(key->value,ele->ptr,score) == C_ERR) return REDISMODULE_ERR; + if (key->kv == NULL) return REDISMODULE_ERR; + if (key->kv->type != OBJ_ZSET) return REDISMODULE_ERR; + if (zsetScore(key->kv,ele->ptr,score) == C_ERR) return REDISMODULE_ERR; return REDISMODULE_OK; } @@ -4926,7 +4927,7 @@ void zsetKeyReset(RedisModuleKey *key) { /* Stop a sorted set iteration. */ void RM_ZsetRangeStop(RedisModuleKey *key) { - if (!key->value || key->value->type != OBJ_ZSET) return; + if (!key->kv || key->kv->type != OBJ_ZSET) return; /* Free resources if needed. */ if (key->u.zset.type == REDISMODULE_ZSET_RANGE_LEX) zslFreeLexRange(&key->u.zset.lrs); @@ -4938,7 +4939,7 @@ void RM_ZsetRangeStop(RedisModuleKey *key) { /* Return the "End of range" flag value to signal the end of the iteration. */ int RM_ZsetRangeEndReached(RedisModuleKey *key) { - if (!key->value || key->value->type != OBJ_ZSET) return 1; + if (!key->kv || key->kv->type != OBJ_ZSET) return 1; return key->u.zset.er; } @@ -4949,7 +4950,7 @@ int RM_ZsetRangeEndReached(RedisModuleKey *key) { * otherwise the last. Return REDISMODULE_OK on success otherwise * REDISMODULE_ERR. */ int zsetInitScoreRange(RedisModuleKey *key, double min, double max, int minex, int maxex, int first) { - if (!key->value || key->value->type != OBJ_ZSET) return REDISMODULE_ERR; + if (!key->kv || key->kv->type != OBJ_ZSET) return REDISMODULE_ERR; RM_ZsetRangeStop(key); key->u.zset.type = REDISMODULE_ZSET_RANGE_SCORE; @@ -4963,11 +4964,11 @@ int zsetInitScoreRange(RedisModuleKey *key, double min, double max, int minex, i zrs->minex = minex; zrs->maxex = maxex; - if (key->value->encoding == OBJ_ENCODING_LISTPACK) { - key->u.zset.current = first ? zzlFirstInRange(key->value->ptr,zrs) : - zzlLastInRange(key->value->ptr,zrs); - } else if (key->value->encoding == OBJ_ENCODING_SKIPLIST) { - zset *zs = key->value->ptr; + if (key->kv->encoding == OBJ_ENCODING_LISTPACK) { + key->u.zset.current = first ? zzlFirstInRange(key->kv->ptr,zrs) : + zzlLastInRange(key->kv->ptr,zrs); + } else if (key->kv->encoding == OBJ_ENCODING_SKIPLIST) { + zset *zs = key->kv->ptr; zskiplist *zsl = zs->zsl; key->u.zset.current = first ? zslNthInRange(zsl,zrs,0) : zslNthInRange(zsl,zrs,-1); @@ -5013,7 +5014,7 @@ int RM_ZsetLastInScoreRange(RedisModuleKey *key, double min, double max, int min * Note that this function takes 'min' and 'max' in the same form of the * Redis ZRANGEBYLEX command. */ int zsetInitLexRange(RedisModuleKey *key, RedisModuleString *min, RedisModuleString *max, int first) { - if (!key->value || key->value->type != OBJ_ZSET) return REDISMODULE_ERR; + if (!key->kv || key->kv->type != OBJ_ZSET) return REDISMODULE_ERR; RM_ZsetRangeStop(key); key->u.zset.er = 0; @@ -5027,11 +5028,11 @@ int zsetInitLexRange(RedisModuleKey *key, RedisModuleString *min, RedisModuleStr * otherwise we don't want the zlexrangespec to be freed. */ key->u.zset.type = REDISMODULE_ZSET_RANGE_LEX; - if (key->value->encoding == OBJ_ENCODING_LISTPACK) { - key->u.zset.current = first ? zzlFirstInLexRange(key->value->ptr,zlrs) : - zzlLastInLexRange(key->value->ptr,zlrs); - } else if (key->value->encoding == OBJ_ENCODING_SKIPLIST) { - zset *zs = key->value->ptr; + if (key->kv->encoding == OBJ_ENCODING_LISTPACK) { + key->u.zset.current = first ? zzlFirstInLexRange(key->kv->ptr,zlrs) : + zzlLastInLexRange(key->kv->ptr,zlrs); + } else if (key->kv->encoding == OBJ_ENCODING_SKIPLIST) { + zset *zs = key->kv->ptr; zskiplist *zsl = zs->zsl; key->u.zset.current = first ? zslNthInLexRange(zsl,zlrs,0) : zslNthInLexRange(zsl,zlrs,-1); @@ -5071,18 +5072,18 @@ int RM_ZsetLastInLexRange(RedisModuleKey *key, RedisModuleString *min, RedisModu RedisModuleString *RM_ZsetRangeCurrentElement(RedisModuleKey *key, double *score) { RedisModuleString *str; - if (!key->value || key->value->type != OBJ_ZSET) return NULL; + if (!key->kv || key->kv->type != OBJ_ZSET) return NULL; if (key->u.zset.current == NULL) return NULL; - if (key->value->encoding == OBJ_ENCODING_LISTPACK) { + if (key->kv->encoding == OBJ_ENCODING_LISTPACK) { unsigned char *eptr, *sptr; eptr = key->u.zset.current; sds ele = lpGetObject(eptr); if (score) { - sptr = lpNext(key->value->ptr,eptr); + sptr = lpNext(key->kv->ptr,eptr); *score = zzlGetScore(sptr); } str = createObject(OBJ_STRING,ele); - } else if (key->value->encoding == OBJ_ENCODING_SKIPLIST) { + } else if (key->kv->encoding == OBJ_ENCODING_SKIPLIST) { zskiplistNode *ln = key->u.zset.current; if (score) *score = ln->score; str = createStringObject(ln->ele,sdslen(ln->ele)); @@ -5097,11 +5098,11 @@ RedisModuleString *RM_ZsetRangeCurrentElement(RedisModuleKey *key, double *score * a next element, 0 if we are already at the latest element or the range * does not include any item at all. */ int RM_ZsetRangeNext(RedisModuleKey *key) { - if (!key->value || key->value->type != OBJ_ZSET) return 0; + if (!key->kv || key->kv->type != OBJ_ZSET) return 0; if (!key->u.zset.type || !key->u.zset.current) return 0; /* No active iterator. */ - if (key->value->encoding == OBJ_ENCODING_LISTPACK) { - unsigned char *zl = key->value->ptr; + if (key->kv->encoding == OBJ_ENCODING_LISTPACK) { + unsigned char *zl = key->kv->ptr; unsigned char *eptr = key->u.zset.current; unsigned char *next; next = lpNext(zl,eptr); /* Skip element. */ @@ -5131,7 +5132,7 @@ int RM_ZsetRangeNext(RedisModuleKey *key) { key->u.zset.current = next; return 1; } - } else if (key->value->encoding == OBJ_ENCODING_SKIPLIST) { + } else if (key->kv->encoding == OBJ_ENCODING_SKIPLIST) { zskiplistNode *ln = key->u.zset.current, *next = ln->level[0].forward; if (next == NULL) { key->u.zset.er = 1; @@ -5161,11 +5162,11 @@ int RM_ZsetRangeNext(RedisModuleKey *key) { * a previous element, 0 if we are already at the first element or the range * does not include any item at all. */ int RM_ZsetRangePrev(RedisModuleKey *key) { - if (!key->value || key->value->type != OBJ_ZSET) return 0; + if (!key->kv || key->kv->type != OBJ_ZSET) return 0; if (!key->u.zset.type || !key->u.zset.current) return 0; /* No active iterator. */ - if (key->value->encoding == OBJ_ENCODING_LISTPACK) { - unsigned char *zl = key->value->ptr; + if (key->kv->encoding == OBJ_ENCODING_LISTPACK) { + unsigned char *zl = key->kv->ptr; unsigned char *eptr = key->u.zset.current; unsigned char *prev; prev = lpPrev(zl,eptr); /* Go back to previous score. */ @@ -5195,7 +5196,7 @@ int RM_ZsetRangePrev(RedisModuleKey *key) { key->u.zset.current = prev; return 1; } - } else if (key->value->encoding == OBJ_ENCODING_SKIPLIST) { + } else if (key->kv->encoding == OBJ_ENCODING_SKIPLIST) { zskiplistNode *ln = key->u.zset.current, *prev = ln->backward; if (prev == NULL) { key->u.zset.er = 1; @@ -5302,16 +5303,16 @@ int RM_HashSet(RedisModuleKey *key, int flags, ...) { REDISMODULE_HASH_COUNT_ALL))) { errno = EINVAL; return 0; - } else if (key->value && key->value->type != OBJ_HASH) { + } else if (key->kv && key->kv->type != OBJ_HASH) { errno = ENOTSUP; return 0; } else if (!(key->mode & REDISMODULE_WRITE)) { errno = EBADF; return 0; } - if (key->value == NULL) moduleCreateEmptyKey(key,REDISMODULE_KEYTYPE_HASH); - - int64_t oldlen = (int64_t) getObjectLength(key->value); + if (key->kv == NULL) moduleCreateEmptyKey(key,REDISMODULE_KEYTYPE_HASH); + + int64_t oldlen = (int64_t) getObjectLength(key->kv); int count = 0; va_start(ap, flags); @@ -5330,7 +5331,7 @@ int RM_HashSet(RedisModuleKey *key, int flags, ...) { /* Handle XX and NX */ if (flags & (REDISMODULE_HASH_XX|REDISMODULE_HASH_NX)) { - int hfeFlags = HFE_LAZY_AVOID_HASH_DEL | HFE_LAZY_NO_UPDATE_KEYSIZES; + int hfeFlags = HFE_LAZY_AVOID_HASH_DEL | HFE_LAZY_NO_UPDATE_KEYSIZES; /* * The hash might contain expired fields. If we lazily delete expired @@ -5344,7 +5345,7 @@ int RM_HashSet(RedisModuleKey *key, int flags, ...) { if (flags & REDISMODULE_HASH_XX) hfeFlags |= HFE_LAZY_AVOID_FIELD_DEL; - int exists = hashTypeExists(key->db, key->value, field->ptr, hfeFlags, NULL); + int exists = hashTypeExists(key->db, key->kv, field->ptr, hfeFlags, NULL); if (((flags & REDISMODULE_HASH_XX) && !exists) || ((flags & REDISMODULE_HASH_NX) && exists)) { @@ -5355,7 +5356,7 @@ int RM_HashSet(RedisModuleKey *key, int flags, ...) { /* Handle deletion if value is REDISMODULE_HASH_DELETE. */ if (value == REDISMODULE_HASH_DELETE) { - count += hashTypeDelete(key->value, field->ptr, 1); + count += hashTypeDelete(key->kv, field->ptr, 1); if (flags & REDISMODULE_HASH_CFIELDS) decrRefCount(field); continue; } @@ -5368,8 +5369,8 @@ int RM_HashSet(RedisModuleKey *key, int flags, ...) { low_flags |= HASH_SET_TAKE_FIELD; robj *argv[2] = {field,value}; - hashTypeTryConversion(key->db,key->value,argv,0,1); - int updated = hashTypeSet(key->db, key->value, field->ptr, value->ptr, low_flags); + hashTypeTryConversion(key->db,key->kv,argv,0,1); + int updated = hashTypeSet(key->db, key->kv, field->ptr, value->ptr, low_flags); count += (flags & REDISMODULE_HASH_COUNT_ALL) ? 1 : updated; /* If CFIELDS is active, SDS string ownership is now of hashTypeSet(), @@ -5380,9 +5381,9 @@ int RM_HashSet(RedisModuleKey *key, int flags, ...) { } } va_end(ap); - updateKeysizesHist(key->db, getKeySlot(key->key->ptr), OBJ_HASH, oldlen, - (int64_t) hashTypeLength(key->value, 0)); - + updateKeysizesHist(key->db, getKeySlot(key->key->ptr), OBJ_HASH, oldlen, + (int64_t) hashTypeLength(key->kv, 0)); + moduleDelKeyIfEmpty(key); if (count == 0) errno = ENOENT; return count; @@ -5442,7 +5443,7 @@ int RM_HashSet(RedisModuleKey *key, int flags, ...) { int RM_HashGet(RedisModuleKey *key, int flags, ...) { int hfeFlags = HFE_LAZY_AVOID_FIELD_DEL | HFE_LAZY_AVOID_HASH_DEL | HFE_LAZY_NO_UPDATE_KEYSIZES; va_list ap; - if (key->value && key->value->type != OBJ_HASH) return REDISMODULE_ERR; + if (key->kv && key->kv->type != OBJ_HASH) return REDISMODULE_ERR; if (key->mode & REDISMODULE_OPEN_KEY_ACCESS_EXPIRED) hfeFlags = HFE_LAZY_ACCESS_EXPIRED; /* allow read also expired fields */ @@ -5468,26 +5469,26 @@ int RM_HashGet(RedisModuleKey *key, int flags, ...) { /* Query the hash for existence or value object. */ if (flags & REDISMODULE_HASH_EXISTS) { existsptr = va_arg(ap,int*); - if (key->value) { - *existsptr = hashTypeExists(key->db, key->value, field->ptr, hfeFlags, NULL); + if (key->kv) { + *existsptr = hashTypeExists(key->db, key->kv, field->ptr, hfeFlags, NULL); } else { *existsptr = 0; } } else if (flags & REDISMODULE_HASH_EXPIRE_TIME) { mstime_t *expireptr = va_arg(ap,mstime_t*); *expireptr = REDISMODULE_NO_EXPIRE; - if (key->value) { + if (key->kv) { uint64_t expireTime = 0; /* As an opt, avoid fetching value, only expire time */ - int res = hashTypeGetValueObject(key->db, key->value, field->ptr, + int res = hashTypeGetValueObject(key->db, key->kv, field->ptr, hfeFlags, NULL, &expireTime, NULL); /* If field has expiration time */ if (res && expireTime != 0) *expireptr = expireTime; } } else { valueptr = va_arg(ap,RedisModuleString**); - if (key->value) { - hashTypeGetValueObject(key->db, key->value, field->ptr, + if (key->kv) { + hashTypeGetValueObject(key->db, key->kv, field->ptr, hfeFlags, valueptr, NULL, NULL); if (*valueptr) { @@ -5519,10 +5520,10 @@ int RM_HashGet(RedisModuleKey *key, int flags, ...) { * is not a hash. */ mstime_t RM_HashFieldMinExpire(RedisModuleKey *key) { - if ((!key->value) || (key->value->type != OBJ_HASH)) + if ((!key->kv) || (key->kv->type != OBJ_HASH)) return REDISMODULE_NO_EXPIRE; - mstime_t min = hashTypeGetMinExpire(key->value, 1); + mstime_t min = hashTypeGetMinExpire(key->kv, 1); return (min == EB_EXPIRE_TIME_INVALID) ? REDISMODULE_NO_EXPIRE : min; } @@ -5574,7 +5575,7 @@ int RM_StreamAdd(RedisModuleKey *key, int flags, RedisModuleStreamID *id, RedisM (!(flags & REDISMODULE_STREAM_ADD_AUTOID) && !id)) { /* id required */ errno = EINVAL; return REDISMODULE_ERR; - } else if (key->value && key->value->type != OBJ_STREAM) { + } else if (key->kv && key->kv->type != OBJ_STREAM) { errno = ENOTSUP; /* wrong type */ return REDISMODULE_ERR; } else if (!(key->mode & REDISMODULE_WRITE)) { @@ -5588,12 +5589,12 @@ int RM_StreamAdd(RedisModuleKey *key, int flags, RedisModuleStreamID *id, RedisM /* Create key if necessary */ int created = 0; - if (key->value == NULL) { + if (key->kv == NULL) { moduleCreateEmptyKey(key, REDISMODULE_KEYTYPE_STREAM); created = 1; } - stream *s = key->value->ptr; + stream *s = key->kv->ptr; if (s->last_id.ms == UINT64_MAX && s->last_id.seq == UINT64_MAX) { /* The stream has reached the last possible ID */ errno = EFBIG; @@ -5650,7 +5651,7 @@ int RM_StreamDelete(RedisModuleKey *key, RedisModuleStreamID *id) { if (!key || !id) { errno = EINVAL; return REDISMODULE_ERR; - } else if (!key->value || key->value->type != OBJ_STREAM) { + } else if (!key->kv || key->kv->type != OBJ_STREAM) { errno = ENOTSUP; /* wrong type */ return REDISMODULE_ERR; } else if (!(key->mode & REDISMODULE_WRITE) || @@ -5658,7 +5659,7 @@ int RM_StreamDelete(RedisModuleKey *key, RedisModuleStreamID *id) { errno = EBADF; /* key not opened for writing or iterator started */ return REDISMODULE_ERR; } - stream *s = key->value->ptr; + stream *s = key->kv->ptr; streamID streamid = {id->ms, id->seq}; if (streamDeleteItem(s, &streamid)) { return REDISMODULE_OK; @@ -5724,7 +5725,7 @@ int RM_StreamIteratorStart(RedisModuleKey *key, int flags, RedisModuleStreamID * REDISMODULE_STREAM_ITERATOR_REVERSE))) { errno = EINVAL; /* key missing or invalid flags */ return REDISMODULE_ERR; - } else if (!key->value || key->value->type != OBJ_STREAM) { + } else if (!key->kv || key->kv->type != OBJ_STREAM) { errno = ENOTSUP; return REDISMODULE_ERR; /* not a stream */ } else if (key->iter) { @@ -5745,7 +5746,7 @@ int RM_StreamIteratorStart(RedisModuleKey *key, int flags, RedisModuleStreamID * } /* create iterator */ - stream *s = key->value->ptr; + stream *s = key->kv->ptr; int rev = flags & REDISMODULE_STREAM_ITERATOR_REVERSE; streamIterator *si = zmalloc(sizeof(*si)); streamIteratorStart(si, s, start ? &lower : NULL, end ? &upper : NULL, rev); @@ -5772,7 +5773,7 @@ int RM_StreamIteratorStop(RedisModuleKey *key) { if (!key) { errno = EINVAL; return REDISMODULE_ERR; - } else if (!key->value || key->value->type != OBJ_STREAM) { + } else if (!key->kv || key->kv->type != OBJ_STREAM) { errno = ENOTSUP; return REDISMODULE_ERR; } else if (!key->iter) { @@ -5814,7 +5815,7 @@ int RM_StreamIteratorNextID(RedisModuleKey *key, RedisModuleStreamID *id, long * if (!key) { errno = EINVAL; return REDISMODULE_ERR; - } else if (!key->value || key->value->type != OBJ_STREAM) { + } else if (!key->kv || key->kv->type != OBJ_STREAM) { errno = ENOTSUP; return REDISMODULE_ERR; } else if (!key->iter) { @@ -5870,7 +5871,7 @@ int RM_StreamIteratorNextField(RedisModuleKey *key, RedisModuleString **field_pt if (!key) { errno = EINVAL; return REDISMODULE_ERR; - } else if (!key->value || key->value->type != OBJ_STREAM) { + } else if (!key->kv || key->kv->type != OBJ_STREAM) { errno = ENOTSUP; return REDISMODULE_ERR; } else if (!key->iter) { @@ -5913,7 +5914,7 @@ int RM_StreamIteratorDelete(RedisModuleKey *key) { if (!key) { errno = EINVAL; return REDISMODULE_ERR; - } else if (!key->value || key->value->type != OBJ_STREAM) { + } else if (!key->kv || key->kv->type != OBJ_STREAM) { errno = ENOTSUP; return REDISMODULE_ERR; } else if (!(key->mode & REDISMODULE_WRITE) || !key->iter) { @@ -5951,7 +5952,7 @@ long long RM_StreamTrimByLength(RedisModuleKey *key, int flags, long long length if (!key || (flags & ~(REDISMODULE_STREAM_TRIM_APPROX)) || length < 0) { errno = EINVAL; return -1; - } else if (!key->value || key->value->type != OBJ_STREAM) { + } else if (!key->kv || key->kv->type != OBJ_STREAM) { errno = ENOTSUP; return -1; } else if (!(key->mode & REDISMODULE_WRITE)) { @@ -5959,7 +5960,7 @@ long long RM_StreamTrimByLength(RedisModuleKey *key, int flags, long long length return -1; } int approx = flags & REDISMODULE_STREAM_TRIM_APPROX ? 1 : 0; - return streamTrimByLength((stream *)key->value->ptr, length, approx); + return streamTrimByLength((stream *)key->kv->ptr, length, approx); } /* Trim a stream by ID, similar to XTRIM with MINID. @@ -5981,7 +5982,7 @@ long long RM_StreamTrimByID(RedisModuleKey *key, int flags, RedisModuleStreamID if (!key || (flags & ~(REDISMODULE_STREAM_TRIM_APPROX)) || !id) { errno = EINVAL; return -1; - } else if (!key->value || key->value->type != OBJ_STREAM) { + } else if (!key->kv || key->kv->type != OBJ_STREAM) { errno = ENOTSUP; return -1; } else if (!(key->mode & REDISMODULE_WRITE)) { @@ -5990,7 +5991,7 @@ long long RM_StreamTrimByID(RedisModuleKey *key, int flags, RedisModuleStreamID } int approx = flags & REDISMODULE_STREAM_TRIM_APPROX ? 1 : 0; streamID minid = (streamID){id->ms, id->seq}; - return streamTrimByID((stream *)key->value->ptr, minid, approx); + return streamTrimByID((stream *)key->kv->ptr, minid, approx); } /* -------------------------------------------------------------------------- @@ -7132,9 +7133,8 @@ int RM_ModuleTypeSetValue(RedisModuleKey *key, moduleType *mt, void *value) { if (!(key->mode & REDISMODULE_WRITE) || key->iter) return REDISMODULE_ERR; RM_DeleteKey(key); robj *o = createModuleObject(mt,value); - setKey(key->ctx->client,key->db,key->key,o,SETKEY_NO_SIGNAL); - decrRefCount(o); - key->value = o; + setKey(key->ctx->client,key->db,key->key, &o,SETKEY_NO_SIGNAL); + key->kv = o; return REDISMODULE_OK; } @@ -7145,9 +7145,9 @@ int RM_ModuleTypeSetValue(RedisModuleKey *key, moduleType *mt, void *value) { * then NULL is returned instead. */ moduleType *RM_ModuleTypeGetType(RedisModuleKey *key) { if (key == NULL || - key->value == NULL || + key->kv == NULL || RM_KeyType(key) != REDISMODULE_KEYTYPE_MODULE) return NULL; - moduleValue *mv = key->value->ptr; + moduleValue *mv = key->kv->ptr; return mv->type; } @@ -7159,9 +7159,9 @@ moduleType *RM_ModuleTypeGetType(RedisModuleKey *key) { * then NULL is returned instead. */ void *RM_ModuleTypeGetValue(RedisModuleKey *key) { if (key == NULL || - key->value == NULL || + key->kv == NULL || RM_KeyType(key) != REDISMODULE_KEYTYPE_MODULE) return NULL; - moduleValue *mv = key->value->ptr; + moduleValue *mv = key->kv->ptr; return mv->value; } @@ -11131,15 +11131,16 @@ typedef struct RedisModuleScanCursor{ int done; }RedisModuleScanCursor; -static void moduleScanCallback(void *privdata, const dictEntry *de) { +static void moduleScanCallback(void *privdata, const dictEntry *de, dictEntryLink plink) { + UNUSED(plink); ScanCBData *data = privdata; - sds key = dictGetKey(de); - robj* val = dictGetVal(de); + kvobj *keyvalObj = dictGetKey(de); + sds key = kvobjGetKey(keyvalObj); RedisModuleString *keyname = createObject(OBJ_STRING,sdsdup(key)); /* Setup the key handle. */ RedisModuleKey kp = {0}; - moduleInitKey(&kp, data->ctx, keyname, val, REDISMODULE_READ); + moduleInitKey(&kp, data->ctx, keyname, keyvalObj, REDISMODULE_READ); data->fn(data->ctx, keyname, &kp, data->user_data); @@ -11245,15 +11246,16 @@ typedef struct { RedisModuleScanKeyCB fn; } ScanKeyCBData; -static void moduleScanKeyCallback(void *privdata, const dictEntry *de) { +static void moduleScanKeyCallback(void *privdata, const dictEntry *de, dictEntryLink plink) { + UNUSED(plink); ScanKeyCBData *data = privdata; sds key = dictGetKey(de); - robj *o = data->key->value; + kvobj *kv = data->key->kv; robj *field = NULL; robj *value = NULL; - if (o->type == OBJ_SET) { + if (kv->type == OBJ_SET) { value = NULL; - } else if (o->type == OBJ_HASH) { + } else if (kv->type == OBJ_HASH) { sds val = dictGetVal(de); /* If field is expired and not indicated to access expired, then ignore */ @@ -11263,7 +11265,7 @@ static void moduleScanKeyCallback(void *privdata, const dictEntry *de) { field = createStringObject(key, hfieldlen(key)); value = createStringObject(val, sdslen(val)); - } else if (o->type == OBJ_ZSET) { + } else if (kv->type == OBJ_ZSET) { double *val = (double*)dictGetVal(de); value = createStringObjectFromLongDouble(*val, 0); } @@ -11325,21 +11327,21 @@ static void moduleScanKeyCallback(void *privdata, const dictEntry *de) { * deleting the current element of the data structure is safe, while removing * the key you are iterating is not safe. */ int RM_ScanKey(RedisModuleKey *key, RedisModuleScanCursor *cursor, RedisModuleScanKeyCB fn, void *privdata) { - if (key == NULL || key->value == NULL) { + if (key == NULL || key->kv == NULL) { errno = EINVAL; return 0; } dict *ht = NULL; - robj *o = key->value; - if (o->type == OBJ_SET) { - if (o->encoding == OBJ_ENCODING_HT) - ht = o->ptr; - } else if (o->type == OBJ_HASH) { - if (o->encoding == OBJ_ENCODING_HT) - ht = o->ptr; - } else if (o->type == OBJ_ZSET) { - if (o->encoding == OBJ_ENCODING_SKIPLIST) - ht = ((zset *)o->ptr)->dict; + kvobj *kv = key->kv; + if (kv->type == OBJ_SET) { + if (kv->encoding == OBJ_ENCODING_HT) + ht = kv->ptr; + } else if (kv->type == OBJ_HASH) { + if (kv->encoding == OBJ_ENCODING_HT) + ht = kv->ptr; + } else if (kv->type == OBJ_ZSET) { + if (kv->encoding == OBJ_ENCODING_SKIPLIST) + ht = ((zset *)kv->ptr)->dict; } else { errno = EINVAL; return 0; @@ -11356,8 +11358,8 @@ int RM_ScanKey(RedisModuleKey *key, RedisModuleScanCursor *cursor, RedisModuleSc cursor->done = 1; ret = 0; } - } else if (o->type == OBJ_SET) { - setTypeIterator *si = setTypeInitIterator(o); + } else if (kv->type == OBJ_SET) { + setTypeIterator *si = setTypeInitIterator(kv); sds sdsele; while ((sdsele = setTypeNextObject(si)) != NULL) { robj *field = createObject(OBJ_STRING, sdsele); @@ -11368,15 +11370,15 @@ int RM_ScanKey(RedisModuleKey *key, RedisModuleScanCursor *cursor, RedisModuleSc cursor->cursor = 1; cursor->done = 1; ret = 0; - } else if (o->type == OBJ_ZSET || o->type == OBJ_HASH) { + } else if (kv->type == OBJ_ZSET || kv->type == OBJ_HASH) { unsigned char *lp, *p; /* is hash with expiry on fields, then lp tuples are [field][value][expire] */ - int hfe = o->type == OBJ_HASH && o->encoding == OBJ_ENCODING_LISTPACK_EX; + int hfe = kv->type == OBJ_HASH && kv->encoding == OBJ_ENCODING_LISTPACK_EX; - if (o->type == OBJ_HASH) - lp = hashTypeListpackGetLp(o); + if (kv->type == OBJ_HASH) + lp = hashTypeListpackGetLp(kv); else - lp = o->ptr; + lp = kv->ptr; p = lpSeek(lp,0); while(p) { @@ -11395,7 +11397,7 @@ int RM_ScanKey(RedisModuleKey *key, RedisModuleScanCursor *cursor, RedisModuleSc /* Skip expired fields */ if ((!(key->mode & REDISMODULE_OPEN_KEY_ACCESS_EXPIRED)) && - (hashTypeIsExpired(o, vllExpire))) + (hashTypeIsExpired(kv, vllExpire))) continue; } @@ -11920,7 +11922,7 @@ int RM_IsSubEventSupported(RedisModuleEvent event, int64_t subevent) { typedef struct KeyInfo { int32_t dbnum; RedisModuleString *key; - robj *value; + kvobj *kv; /* key-value object */ int mode; } KeyInfo; @@ -11996,7 +11998,7 @@ void moduleFireServerEvent(uint64_t eid, int subid, void *data) { } else if (eid == REDISMODULE_EVENT_KEY) { KeyInfo *info = data; selectDb(ctx.client, info->dbnum); - moduleInitKey(&key, &ctx, info->key, info->value, info->mode); + moduleInitKey(&key, &ctx, info->key, info->kv, info->mode); moduledata = &ki; } @@ -12053,7 +12055,7 @@ void processModuleLoadingProgressEvent(int is_aof) { /* When a key is deleted (in dbAsyncDelete/dbSyncDelete/setKey), it * will be called to tell the module which key is about to be released. */ -void moduleNotifyKeyUnlink(robj *key, robj *val, int dbid, int flags) { +void moduleNotifyKeyUnlink(robj *key, kvobj *kv, int dbid, int flags) { server.allow_access_expired++; int subevent = REDISMODULE_SUBEVENT_KEY_DELETED; if (flags & DB_FLAG_KEY_EXPIRED) { @@ -12063,11 +12065,11 @@ void moduleNotifyKeyUnlink(robj *key, robj *val, int dbid, int flags) { } else if (flags & DB_FLAG_KEY_OVERWRITE) { subevent = REDISMODULE_SUBEVENT_KEY_OVERWRITTEN; } - KeyInfo info = {dbid, key, val, REDISMODULE_READ}; + KeyInfo info = {dbid, key, kv, REDISMODULE_READ}; moduleFireServerEvent(REDISMODULE_EVENT_KEY, subevent, &info); - if (val->type == OBJ_MODULE) { - moduleValue *mv = val->ptr; + if (kv->type == OBJ_MODULE) { + moduleValue *mv = kv->ptr; moduleType *mt = mv->type; /* We prefer to use the enhanced version. */ if (mt->unlink2 != NULL) { @@ -12126,8 +12128,8 @@ uint64_t dictCStringKeyHash(const void *key) { return dictGenHashFunction((unsigned char*)key, strlen((char*)key)); } -int dictCStringKeyCompare(dict *d, const void *key1, const void *key2) { - UNUSED(d); +int dictCStringKeyCompare(dictCmpCache *cache, const void *key1, const void *key2) { + UNUSED(cache); return strcmp(key1,key2) == 0; } @@ -13596,9 +13598,9 @@ size_t moduleCount(void) { * servers's maxmemory policy is LFU based. Value is idle time in milliseconds. * returns REDISMODULE_OK if the LRU was updated, REDISMODULE_ERR otherwise. */ int RM_SetLRU(RedisModuleKey *key, mstime_t lru_idle) { - if (!key->value) + if (!key->kv) return REDISMODULE_ERR; - if (objectSetLRUOrLFU(key->value, -1, lru_idle, lru_idle>=0 ? LRU_CLOCK() : 0, 1)) + if (objectSetLRUOrLFU(key->kv, -1, lru_idle, lru_idle>=0 ? LRU_CLOCK() : 0, 1)) return REDISMODULE_OK; return REDISMODULE_ERR; } @@ -13609,11 +13611,11 @@ int RM_SetLRU(RedisModuleKey *key, mstime_t lru_idle) { * returns REDISMODULE_OK if when key is valid. */ int RM_GetLRU(RedisModuleKey *key, mstime_t *lru_idle) { *lru_idle = -1; - if (!key->value) + if (!key->kv) return REDISMODULE_ERR; if (server.maxmemory_policy & MAXMEMORY_FLAG_LFU) return REDISMODULE_OK; - *lru_idle = estimateObjectIdleTime(key->value); + *lru_idle = estimateObjectIdleTime(key->kv); return REDISMODULE_OK; } @@ -13623,9 +13625,9 @@ int RM_GetLRU(RedisModuleKey *key, mstime_t *lru_idle) { * the access frequencyonly (must be <= 255). * returns REDISMODULE_OK if the LFU was updated, REDISMODULE_ERR otherwise. */ int RM_SetLFU(RedisModuleKey *key, long long lfu_freq) { - if (!key->value) + if (!key->kv) return REDISMODULE_ERR; - if (objectSetLRUOrLFU(key->value, lfu_freq, -1, 0, 1)) + if (objectSetLRUOrLFU(key->kv, lfu_freq, -1, 0, 1)) return REDISMODULE_OK; return REDISMODULE_ERR; } @@ -13635,10 +13637,10 @@ int RM_SetLFU(RedisModuleKey *key, long long lfu_freq) { * returns REDISMODULE_OK if when key is valid. */ int RM_GetLFU(RedisModuleKey *key, long long *lfu_freq) { *lfu_freq = -1; - if (!key->value) + if (!key->kv) return REDISMODULE_ERR; if (server.maxmemory_policy & MAXMEMORY_FLAG_LFU) - *lfu_freq = LFUDecrAndReturn(key->value); + *lfu_freq = LFUDecrAndReturn(key->kv); return REDISMODULE_OK; } @@ -13734,10 +13736,10 @@ int RM_GetTypeMethodVersion(void) { int RM_ModuleTypeReplaceValue(RedisModuleKey *key, moduleType *mt, void *new_value, void **old_value) { if (!(key->mode & REDISMODULE_WRITE) || key->iter) return REDISMODULE_ERR; - if (!key->value || key->value->type != OBJ_MODULE) + if (!key->kv || key->kv->type != OBJ_MODULE) return REDISMODULE_ERR; - moduleValue *mv = key->value->ptr; + moduleValue *mv = key->kv->ptr; if (mv->type != mt) return REDISMODULE_ERR; diff --git a/src/multi.c b/src/multi.c index c8082c297..ab2ee7010 100644 --- a/src/multi.c +++ b/src/multi.c @@ -304,7 +304,7 @@ void watchForKey(client *c, robj *key) { wk->key = key; wk->client = c; wk->db = c->db; - wk->expired = keyIsExpired(c->db, key); + wk->expired = keyIsExpired(c->db, key->ptr, NULL); incrRefCount(key); listAddNodeTail(c->watched_keys, wk); watchedKeyLinkToClients(clients, wk); @@ -349,7 +349,7 @@ int isWatchedKeyExpired(client *c) { while ((ln = listNext(&li))) { wk = listNodeValue(ln); if (wk->expired) continue; /* was expired when WATCH was called */ - if (keyIsExpired(wk->db, wk->key)) return 1; + if (keyIsExpired(wk->db, wk->key->ptr, NULL)) return 1; } return 0; @@ -436,11 +436,11 @@ void touchAllWatchedKeysInDb(redisDb *emptied, redisDb *replaced_with) { * flag. Deleted keys are not flagged as expired. */ wk->expired = 0; continue; - } else if (keyIsExpired(replaced_with, key)) { + } else if (keyIsExpired(replaced_with, key->ptr, NULL)) { /* Expired key remains expired. */ continue; } - } else if (!exists_in_emptied && keyIsExpired(replaced_with, key)) { + } else if (!exists_in_emptied && keyIsExpired(replaced_with, key->ptr, NULL)) { /* Non-existing key is replaced with an expired key. */ wk->expired = 1; continue; diff --git a/src/object.c b/src/object.c index 779a2b014..efa36a05e 100644 --- a/src/object.c +++ b/src/object.c @@ -2,6 +2,9 @@ * * Copyright (c) 2009-Present, Redis Ltd. * All rights reserved. + * + * Copyright (c) 2024-present, Valkey contributors. + * All rights reserved. * * Licensed under your choice of (a) the Redis Source Available License 2.0 * (RSALv2); or (b) the Server Side Public License v1 (SSPLv1); or (c) the @@ -18,8 +21,69 @@ #define strtold(a,b) ((long double)strtod((a),(b))) #endif +/* For objects with large embedded keys, we reserve space for an expire field, + * so if expire is set later, we don't need to reallocate the object. */ +#define KEY_SIZE_TO_INCLUDE_EXPIRE_THRESHOLD 128 + /* ===================== Creation and parsing of objects ==================== */ +/* Creates an object, with embedded key and expire fields. The key and expire + * fields can be omitted by passing NULL and -1, respectively. + * + * Example of kvobj "mykey" WITH expiry (16+8+1+7=32bytes): + * + * +-----------+------------+------------------+------------------------+ + * | robj (16) | expiry (8) | key-hdr-size (1) | sdshdr5 "mykey" \0 (7) | + * +-----------+------------+------------------+------------------------+ + */ +kvobj *kvobjCreate(int type, const sds key, void *ptr, long long expire) { + /* Determine embedded key and expiration flags */ + serverAssert(key != NULL); + int has_expire = ((expire != -1) || (sdslen(key) >= KEY_SIZE_TO_INCLUDE_EXPIRE_THRESHOLD)); + + /* Calculate embedded key size */ + size_t key_sds_len = sdslen(key); + char key_sds_type = sdsReqType(key_sds_len); + size_t key_sds_size = sdsReqSize(key_sds_len, key_sds_type); + + /* Compute the base object size */ + size_t min_size = sizeof(robj); + if (has_expire) min_size += sizeof(long long); + min_size += 1 + key_sds_size; /* 1 byte for SDS header size */ + + /* Allocate object memory */ + size_t bufsize = 0; + robj *o = zmalloc_usable(min_size, &bufsize); + o->type = type; + o->encoding = OBJ_ENCODING_RAW; + o->ptr = ptr; + o->refcount = 1; + o->lru = 0; + o->iskvobj = 1; + + /* If extra space allows, pre-allocate anyway expiration */ + if ((!has_expire) && (bufsize >= min_size + sizeof(long long))) { + has_expire = 1; + min_size += sizeof(long long); + } + o->expirable = has_expire; + + /* The memory after the struct where we embedded data. */ + char *data = (void *)(o + 1); + + /* Set the expire field. */ + if (o->expirable) { + *(long long *)data = expire; + data += sizeof(long long); + } + + /* Store embedded key. */ + *data++ = sdsHdrSize(key_sds_type); + sdsnewplacement(data, key_sds_size, key_sds_type, key, key_sds_len); + + return o; +} + robj *createObject(int type, void *ptr) { robj *o = zmalloc(sizeof(*o)); o->type = type; @@ -27,6 +91,8 @@ robj *createObject(int type, void *ptr) { o->ptr = ptr; o->refcount = 1; o->lru = 0; + o->iskvobj = 0; + o->expirable = 0; return o; } @@ -66,33 +132,191 @@ robj *createRawStringObject(const char *ptr, size_t len) { return createObject(OBJ_STRING, sdsnewlen(ptr,len)); } -/* Create a string object with encoding OBJ_ENCODING_EMBSTR, that is - * an object where the sds string is actually an unmodifiable string - * allocated in the same chunk as the object itself. */ -robj *createEmbeddedStringObject(const char *ptr, size_t len) { - robj *o = zmalloc(sizeof(robj)+sizeof(struct sdshdr8)+len+1); - struct sdshdr8 *sh = (void*)(o+1); +/* Creates a new embedded string object and copies the content of key, val and + * expire to the new object. LRU is set to 0. + * + * Example of kvobj "mykey" with embedded "myvalue" (16+1+7+11 = 35bytes): + * +-----------+------------------+------------------------+----------------------------+ + * | robj (16) | key-hdr-size (1) | sdshdr5 "mykey" \0 (7) | sdshdr8 "myvalue" \0 (11) | + * +-----------+------------------+------------------------+----------------------------+ + */ +static kvobj *kvobjCreateEmbedString(const char *val_ptr, size_t val_len, + const sds key, long long expire) + +{ + serverAssert(key != NULL); + /* Calculate sizes for embedded key */ + size_t key_sds_len = sdslen(key); + char key_sds_type = sdsReqType(key_sds_len); + size_t key_sds_size = sdsReqSize(key_sds_len, key_sds_type); + + /* Calculate size for embedded value (always SDS_TYPE_8) */ + size_t val_sds_size = sdsReqSize(val_len, SDS_TYPE_8); + + /* Compute base object size */ + size_t min_size = sizeof(robj) + val_sds_size; + if (expire != -1) min_size += sizeof(long long); + min_size += 1 + key_sds_size; /* 1 byte for SDS header size */ + + /* Allocate object memory */ + size_t bufsize = 0; + robj *o = zmalloc_usable(min_size, &bufsize); o->type = OBJ_STRING; o->encoding = OBJ_ENCODING_EMBSTR; - o->ptr = sh+1; o->refcount = 1; o->lru = 0; + o->expirable = (expire != -1); + o->iskvobj = 1; - sh->len = len; - sh->alloc = len; - sh->flags = SDS_TYPE_8; - if (ptr == SDS_NOINIT) - sh->buf[len] = '\0'; - else if (ptr) { - memcpy(sh->buf,ptr,len); - sh->buf[len] = '\0'; - } else { - memset(sh->buf,0,len+1); + /* If the allocation has enough space for an expire field, add it even if we + * don't need it now. Then we don't need to realloc if it's needed later. */ + if (!o->expirable && bufsize >= min_size + sizeof(long long)) { + o->expirable = 1; + min_size += sizeof(long long); } + + /* The memory after the struct where we embedded data. */ + char *data = (char *)(o + 1); + + /* Set the expire field. */ + if (o->expirable) { + *(long long *)data = expire; + data += sizeof(long long); + } + + /* Store embedded key */ + *data++ = sdsHdrSize(key_sds_type); + sdsnewplacement(data, key_sds_size, key_sds_type, key, key_sds_len); + data += key_sds_size; + + /* Copy embedded value (EMBSTR) always as SDS TYPE 8. Account for unused + * memory in the SDS alloc field. */ + size_t remaining_size = bufsize - (data - (char *)(void *)o); + o->ptr = sdsnewplacement(data, remaining_size, SDS_TYPE_8, val_ptr, val_len); return o; } +/* Create a string object with encoding OBJ_ENCODING_EMBSTR, that is + * an object where the sds string is actually an unmodifiable string + * allocated in the same chunk as the object itself. + * + * Example of robj with embedded "myvalue" (16+1+11 = 28 bytes): + * +-----------+------------------+----------------------------+ + * | robj (16) | key-hdr-size (1) | sdshdr8 "myvalue" \0 (11) | + * +-----------+------------------+----------------------------+ + */ +robj *createEmbeddedStringObject(const char *val_ptr, size_t val_len) { + /* Calculate size for embedded value (always SDS_TYPE_8) */ + size_t val_sds_size = sdsReqSize(val_len, SDS_TYPE_8); + + /* Allocate object memory */ + size_t bufsize = 0; + robj *o = zmalloc_usable(sizeof(robj) + val_sds_size, &bufsize); + o->type = OBJ_STRING; + o->encoding = OBJ_ENCODING_EMBSTR; + o->refcount = 1; + o->lru = 0; + o->expirable = 0; + o->iskvobj = 0; + + /* The memory after the struct where we embedded data. */ + char *data = (char *)(o + 1); + + /* Copy embedded value (EMBSTR) always as SDS TYPE 8. Account for unused + * memory in the SDS alloc field. */ + size_t remaining_size = bufsize - (data - (char *)(void *)o); + o->ptr = sdsnewplacement(data, remaining_size, SDS_TYPE_8, val_ptr, val_len); + return o; +} + +sds kvobjGetKey(const kvobj *kv) { + unsigned char *data = (void *)(kv + 1); + if (kv->expirable) { + /* Skip expire field */ + data += sizeof(long long); + } + if (kv->iskvobj) { + uint8_t hdr_size = *(uint8_t *)data; + data += 1 + hdr_size; + return (sds)data; + } + return NULL; +} + +long long kvobjGetExpire(const kvobj *kv) { + unsigned char *data = (void *)(kv + 1); + if (kv->expirable) { + return *(long long *)data; + } else { + return -1; + } +} + +/* This functions may reallocate the value. The new allocation is returned and + * the old object's reference counter is decremented and possibly freed. Use the + * returned object instead of 'val' after calling this function. */ +kvobj *kvobjSetExpire(kvobj *kv, long long expire) { + if (kv->expirable) { + /* Update existing expire field. */ + unsigned char *data = (void *)(kv + 1); + *(long long *)data = expire; + return kv; + } else if (expire == -1) { + return kv; + } else { + return kvobjSet(kvobjGetKey(kv), kv, expire); + } +} + +/* This functions may reallocate the value. The new allocation is returned and + * the old object's reference counter is decremented and possibly freed. Use the + * returned object instead of 'val' after calling this function. */ +kvobj *kvobjSet(sds key, robj *val, long long expire) { + if (val->type == OBJ_STRING && val->encoding == OBJ_ENCODING_EMBSTR) { + kvobj *kv; + size_t len = sdslen(val->ptr); + + /* Embed when the sum is up to 64 bytes. */ + size_t size = sizeof(kvobj); + size += (key != NULL) * (sdslen(key) + 3); /* hdr size (1) + hdr (1) + nullterm (1) */ + size += (expire != -1) * sizeof(long long); + size += 4 + len; /* embstr header (3) + nullterm (1) */ + if (size <= CACHE_LINE_SIZE) { + kv = kvobjCreateEmbedString(val->ptr, len, key, expire); + } else { + kv = kvobjCreate(OBJ_STRING, key, sdsnewlen(val->ptr, len), expire); + } + + kv->lru = val->lru; + decrRefCount(val); + return kv; + } + + /* Create a new object with embedded key. Reuse ptr if possible. */ + void *valptr; + if (val->refcount == 1) { + /* Reuse the ptr. There are no other references to val. */ + valptr = val->ptr; + val->ptr = NULL; + } else if (val->type == OBJ_STRING && val->encoding == OBJ_ENCODING_INT) { + /* The pointer is not allocated memory. We can just copy the pointer. */ + valptr = val->ptr; + } else if (val->type == OBJ_STRING && val->encoding == OBJ_ENCODING_RAW) { + /* Dup the string. */ + valptr = sdsdup(val->ptr); + } else { + /* There are multiple references to this non-string object. Most types + * can be duplicated, but for a module type is not always possible. */ + serverPanic("Not implemented"); + } + robj *new = kvobjCreate(val->type, key, valptr, expire); + new->encoding = val->encoding; + new->lru = val->lru; + decrRefCount(val); + return new; +} + /* Create a string object with EMBSTR encoding if it is smaller than * OBJ_ENCODING_EMBSTR_SIZE_LIMIT, otherwise the RAW encoding is * used. @@ -369,15 +593,17 @@ void decrRefCount(robj *o) { } if (--(o->refcount) == 0) { - switch(o->type) { - case OBJ_STRING: freeStringObject(o); break; - case OBJ_LIST: freeListObject(o); break; - case OBJ_SET: freeSetObject(o); break; - case OBJ_ZSET: freeZsetObject(o); break; - case OBJ_HASH: freeHashObject(o); break; - case OBJ_MODULE: freeModuleObject(o); break; - case OBJ_STREAM: freeStreamObject(o); break; - default: serverPanic("Unknown object type"); break; + if (o->ptr != NULL) { + switch(o->type) { + case OBJ_STRING: freeStringObject(o); break; + case OBJ_LIST: freeListObject(o); break; + case OBJ_SET: freeSetObject(o); break; + case OBJ_ZSET: freeZsetObject(o); break; + case OBJ_HASH: freeHashObject(o); break; + case OBJ_MODULE: freeModuleObject(o); break; + case OBJ_STREAM: freeStreamObject(o); break; + default: serverPanic("Unknown object type"); break; + } } zfree(o); } @@ -561,13 +787,6 @@ void dismissObject(robj *o, size_t size_hint) { #endif } -/* This variant of decrRefCount() gets its argument as void, and is useful - * as free method in data structures that expect a 'void free_object(void*)' - * prototype for the free method. */ -void decrRefCountVoid(void *o) { - decrRefCount(o); -} - int checkType(client *c, robj *o, int type) { /* A NULL is considered an empty key */ if (o && o->type != type) { @@ -636,27 +855,15 @@ robj *tryObjectEncodingEx(robj *o, int try_trim) { * representable as a 32 nor 64 bit integer. */ len = sdslen(s); if (len <= 20 && string2l(s,len,&value)) { - /* This object is encodable as a long. Try to use a shared object. - * Note that we avoid using shared integers when maxmemory is used - * because every object needs to have a private LRU field for the LRU - * algorithm to work well. */ - if ((server.maxmemory == 0 || - !(server.maxmemory_policy & MAXMEMORY_FLAG_NO_SHARED_INTEGERS)) && - value >= 0 && - value < OBJ_SHARED_INTEGERS) - { + /* This object is encodable as a long. */ + if (o->encoding == OBJ_ENCODING_RAW) { + sdsfree(o->ptr); + o->encoding = OBJ_ENCODING_INT; + o->ptr = (void*) value; + return o; + } else if (o->encoding == OBJ_ENCODING_EMBSTR) { decrRefCount(o); - return shared.integers[value]; - } else { - if (o->encoding == OBJ_ENCODING_RAW) { - sdsfree(o->ptr); - o->encoding = OBJ_ENCODING_INT; - o->ptr = (void*) value; - return o; - } else if (o->encoding == OBJ_ENCODING_EMBSTR) { - decrRefCount(o); - return createStringObjectFromLongLongForValue(value); - } + return createStringObjectFromLongLongForValue(value); } } @@ -1029,7 +1236,7 @@ size_t objectComputeSize(robj *key, robj *o, size_t sample_size, int dbid) { asize = sizeof(*o)+sizeof(dict)+(sizeof(struct dictEntry*)*dictBuckets(d)); while((de = dictNext(di)) != NULL && samples < sample_size) { sds ele = dictGetKey(de); - elesize += dictEntryMemUsage() + sdsZmallocSize(ele); + elesize += dictEntryMemUsage(0) + sdsZmallocSize(ele); samples++; } dictReleaseIterator(di); @@ -1053,7 +1260,7 @@ size_t objectComputeSize(robj *key, robj *o, size_t sample_size, int dbid) { zmalloc_size(zsl->header); while(znode != NULL && samples < sample_size) { elesize += sdsZmallocSize(znode->ele); - elesize += dictEntryMemUsage()+zmalloc_size(znode); + elesize += dictEntryMemUsage(1)+zmalloc_size(znode); samples++; znode = znode->level[0].forward; } @@ -1075,7 +1282,7 @@ size_t objectComputeSize(robj *key, robj *o, size_t sample_size, int dbid) { hfield ele = dictGetKey(de); sds ele2 = dictGetVal(de); elesize += hfieldZmallocSize(ele) + sdsZmallocSize(ele2); - elesize += dictEntryMemUsage(); + elesize += dictEntryMemUsage(0); samples++; } dictReleaseIterator(di); @@ -1453,20 +1660,20 @@ int objectSetLRUOrLFU(robj *val, long long lfu_freq, long long lru_idle, /* This is a helper function for the OBJECT command. We need to lookup keys * without any modification of LRU or other parameters. */ -robj *objectCommandLookup(client *c, robj *key) { +kvobj *kvobjCommandLookup(client *c, robj *key) { return lookupKeyReadWithFlags(c->db,key,LOOKUP_NOTOUCH|LOOKUP_NONOTIFY); } -robj *objectCommandLookupOrReply(client *c, robj *key, robj *reply) { - robj *o = objectCommandLookup(c,key); - if (!o) addReplyOrErrorObject(c, reply); - return o; +kvobj *kvobjCommandLookupOrReply(client *c, robj *key, robj *reply) { + kvobj *kv = kvobjCommandLookup(c,key); + if (!kv) addReplyOrErrorObject(c, reply); + return kv; } /* Object command allows to inspect the internals of a Redis Object. * Usage: OBJECT */ void objectCommand(client *c) { - robj *o; + kvobj *kv; if (c->argc == 2 && !strcasecmp(c->argv[1]->ptr,"help")) { const char *help[] = { @@ -1486,23 +1693,23 @@ NULL }; addReplyHelp(c, help); } else if (!strcasecmp(c->argv[1]->ptr,"refcount") && c->argc == 3) { - if ((o = objectCommandLookupOrReply(c,c->argv[2],shared.null[c->resp])) + if ((kv = kvobjCommandLookupOrReply(c, c->argv[2], shared.null[c->resp])) == NULL) return; - addReplyLongLong(c,o->refcount); + addReplyLongLong(c, kv->refcount); } else if (!strcasecmp(c->argv[1]->ptr,"encoding") && c->argc == 3) { - if ((o = objectCommandLookupOrReply(c,c->argv[2],shared.null[c->resp])) + if ((kv = kvobjCommandLookupOrReply(c, c->argv[2], shared.null[c->resp])) == NULL) return; - addReplyBulkCString(c,strEncoding(o->encoding)); + addReplyBulkCString(c,strEncoding(kv->encoding)); } else if (!strcasecmp(c->argv[1]->ptr,"idletime") && c->argc == 3) { - if ((o = objectCommandLookupOrReply(c,c->argv[2],shared.null[c->resp])) + if ((kv = kvobjCommandLookupOrReply(c, c->argv[2], shared.null[c->resp])) == NULL) return; if (server.maxmemory_policy & MAXMEMORY_FLAG_LFU) { addReplyError(c,"An LFU maxmemory policy is selected, idle time not tracked. Please note that when switching between policies at runtime LRU and LFU data will take some time to adjust."); return; } - addReplyLongLong(c,estimateObjectIdleTime(o)/1000); + addReplyLongLong(c, estimateObjectIdleTime(kv) / 1000); } else if (!strcasecmp(c->argv[1]->ptr,"freq") && c->argc == 3) { - if ((o = objectCommandLookupOrReply(c,c->argv[2],shared.null[c->resp])) + if ((kv = kvobjCommandLookupOrReply(c, c->argv[2], shared.null[c->resp])) == NULL) return; if (!(server.maxmemory_policy & MAXMEMORY_FLAG_LFU)) { addReplyError(c,"An LFU maxmemory policy is not selected, access frequency not tracked. Please note that when switching between policies at runtime LRU and LFU data will take some time to adjust."); @@ -1512,7 +1719,7 @@ NULL * in case of the key has not been accessed for a long time, * because we update the access time only * when the key is read or overwritten. */ - addReplyLongLong(c,LFUDecrAndReturn(o)); + addReplyLongLong(c,LFUDecrAndReturn(kv)); } else { addReplySubcommandSyntaxError(c); } @@ -1540,7 +1747,7 @@ NULL }; addReplyHelp(c, help); } else if (!strcasecmp(c->argv[1]->ptr,"usage") && c->argc >= 3) { - dictEntry *de; + kvobj *kv; long long samples = OBJ_COMPUTE_SIZE_DEF_SAMPLES; for (int j = 3; j < c->argc; j++) { if (!strcasecmp(c->argv[j]->ptr,"samples") && @@ -1559,13 +1766,11 @@ NULL return; } } - if ((de = dbFind(c->db, c->argv[2]->ptr)) == NULL) { + if ((kv = dbFind(c->db, c->argv[2]->ptr)) == NULL) { addReplyNull(c); return; } - size_t usage = objectComputeSize(c->argv[2],dictGetVal(de),samples,c->db->id); - usage += sdsZmallocSize(dictGetKey(de)); - usage += dictEntryMemUsage(); + size_t usage = objectComputeSize(c->argv[2], (robj *)kv, samples, c->db->id); addReplyLongLong(c,usage); } else if (!strcasecmp(c->argv[1]->ptr,"stats") && c->argc == 2) { struct redisMemOverhead *mh = getMemoryOverheadData(); diff --git a/src/pubsub.c b/src/pubsub.c index be82ced35..22c9c4d16 100644 --- a/src/pubsub.c +++ b/src/pubsub.c @@ -243,8 +243,9 @@ int pubsubSubscribeChannel(client *c, robj *channel, pubsubtype type) { unsigned int slot = 0; /* Add the channel to the client -> channels hash table */ - void *position = dictFindPositionForInsert(type.clientPubSubChannels(c),channel,NULL); - if (position) { /* Not yet subscribed to this channel */ + dictEntryLink bucket; + dictEntryLink link = dictFindLink(type.clientPubSubChannels(c),channel,&bucket); + if (link == NULL) { /* Not yet subscribed to this channel */ retval = 1; /* Add the client to the channel -> list of clients hash table */ if (server.cluster_enabled && type.shard) { @@ -263,7 +264,7 @@ int pubsubSubscribeChannel(client *c, robj *channel, pubsubtype type) { } serverAssert(dictAdd(clients, c, NULL) != DICT_ERR); - serverAssert(dictInsertAtPosition(type.clientPubSubChannels(c), channel, position)); + dictSetKeyAtLink(type.clientPubSubChannels(c), channel, &bucket, 1); incrRefCount(channel); } /* Notify the client */ diff --git a/src/rdb.c b/src/rdb.c index 3cee19788..89dd673da 100644 --- a/src/rdb.c +++ b/src/rdb.c @@ -1412,21 +1412,21 @@ ssize_t rdbSaveDb(rio *rdb, int dbid, int rdbflags, long *key_counter) { written += res; last_slot = curr_slot; } - sds keystr = dictGetKey(de); - robj key, *o = dictGetVal(de); + kvobj *kv = dictGetKV(de); + robj key; long long expire; size_t rdb_bytes_before_key = rdb->processed_bytes; - initStaticStringObject(key,keystr); - expire = getExpire(db,&key); - if ((res = rdbSaveKeyValuePair(rdb, &key, o, expire, dbid)) < 0) goto werr; + initStaticStringObject(key,kvobjGetKey(kv)); + expire = kvobjGetExpire(kv); + if ((res = rdbSaveKeyValuePair(rdb, &key, kv, expire, dbid)) < 0) goto werr; written += res; /* In fork child process, we can try to release memory back to the * OS and possibly avoid or decrease COW. We give the dismiss * mechanism a hint about an estimated size of the object we stored. */ size_t dump_size = rdb->processed_bytes - rdb_bytes_before_key; - if (server.in_fork_child) dismissObject(o, dump_size); + if (server.in_fork_child) dismissObject(kv, dump_size); /* Update child info every 1 second (approximately). * in order to avoid calling mstime() on each iteration, we will @@ -2300,7 +2300,7 @@ robj *rdbLoadObject(int rdbtype, rio *rdb, sds key, int dbid, int *error) if (len > server.hash_max_listpack_entries) { hashTypeConvert(o, OBJ_ENCODING_HT, NULL); dictTypeAddMeta((dict**)&o->ptr, &mstrHashDictTypeWithHFE); - initDictExpireMetadata(key, o); + initDictExpireMetadata(o); } else { hashTypeConvert(o, OBJ_ENCODING_LISTPACK_EX, NULL); if (deep_integrity_validation) { @@ -2749,7 +2749,6 @@ robj *rdbLoadObject(int rdbtype, rio *rdb, sds key, int dbid, int *error) (rdbtype == RDB_TYPE_HASH_LISTPACK_EX_PRE_GA) ) { listpackEx *lpt = listpackExCreate(); lpt->lp = encoded; - lpt->key = key; o->ptr = lpt; o->encoding = OBJ_ENCODING_LISTPACK_EX; } else @@ -3628,15 +3627,16 @@ int rdbLoadRioWithLoadingCtx(rio *rdb, int rdbflags, rdbSaveInfo *rsi, rdbLoadin initStaticStringObject(keyobj,key); /* Add the new object in the hash table */ - int added = dbAddRDBLoad(db,key,val); + kvobj *kv = dbAddRDBLoad(db, key, &val, expiretime); server.rdb_last_load_keys_loaded++; - if (!added) { + if (!kv) { if (rdbflags & RDBFLAGS_ALLOW_DUP) { /* This flag is useful for DEBUG RELOAD special modes. * When it's set we allow new keys to replace the current * keys with the same name. */ dbSyncDelete(db,&keyobj); - dbAddRDBLoad(db,key,val); + kv = dbAddRDBLoad(db, key, &val, expiretime); + serverAssert(kv != NULL); } else { serverLog(LL_WARNING, "RDB has duplicated key '%s' in DB %d",key,db->id); @@ -3646,15 +3646,10 @@ int rdbLoadRioWithLoadingCtx(rio *rdb, int rdbflags, rdbSaveInfo *rsi, rdbLoadin /* If minExpiredField was set, then the object is hash with expiration * on fields and need to register it in global HFE DS */ - if (val->type == OBJ_HASH) { - uint64_t minExpiredField = hashTypeGetMinExpire(val, 1); + if (kv->type == OBJ_HASH) { + uint64_t minExpiredField = hashTypeGetMinExpire(kv, 1); if (minExpiredField != EB_EXPIRE_TIME_INVALID) - hashTypeAddToExpires(db, key, val, minExpiredField); - } - - /* Set the expire time if needed */ - if (expiretime != -1) { - setExpire(NULL,db,&keyobj,expiretime); + hashTypeAddToExpires(db, kv, minExpiredField); } /* Set usage information (for eviction). */ @@ -3662,6 +3657,9 @@ int rdbLoadRioWithLoadingCtx(rio *rdb, int rdbflags, rdbSaveInfo *rsi, rdbLoadin /* call key space notification on key loaded for modules only */ moduleNotifyKeyspaceEvent(NOTIFY_LOADED, "loaded", &keyobj, db->id); + + /* Release key (sds), dictEntry stores a copy of it in embedded data */ + sdsfree(key); } /* Loading the database more slowly is useful in order to test diff --git a/src/redis-benchmark.c b/src/redis-benchmark.c index 8319b3a56..8c9e06298 100644 --- a/src/redis-benchmark.c +++ b/src/redis-benchmark.c @@ -183,7 +183,7 @@ int showThroughput(struct aeEventLoop *eventLoop, long long id, /* Dict callbacks */ static uint64_t dictSdsHash(const void *key); -static int dictSdsKeyCompare(dict *d, const void *key1, const void *key2); +static int dictSdsKeyCompare(dictCmpCache *cache, const void *key1, const void *key2); /* Implementation */ static long long ustime(void) { @@ -204,10 +204,10 @@ static uint64_t dictSdsHash(const void *key) { return dictGenHashFunction((unsigned char*)key, sdslen((char*)key)); } -static int dictSdsKeyCompare(dict *d, const void *key1, const void *key2) +static int dictSdsKeyCompare(dictCmpCache *cache, const void *key1, const void *key2) { int l1,l2; - UNUSED(d); + UNUSED(cache); l1 = sdslen((sds)key1); l2 = sdslen((sds)key2); diff --git a/src/redis-cli.c b/src/redis-cli.c index bbaabca52..f1d25b9be 100644 --- a/src/redis-cli.c +++ b/src/redis-cli.c @@ -163,7 +163,7 @@ static struct termios orig_termios; /* To restore terminal at exit.*/ /* Dict Helpers */ static uint64_t dictSdsHash(const void *key); -static int dictSdsKeyCompare(dict *d, const void *key1, +static int dictSdsKeyCompare(dictCmpCache *cache, const void *key1, const void *key2); static void dictSdsDestructor(dict *d, void *val); static void dictListDestructor(dict *d, void *val); @@ -369,10 +369,10 @@ static uint64_t dictSdsHash(const void *key) { return dictGenHashFunction((unsigned char*)key, sdslen((char*)key)); } -static int dictSdsKeyCompare(dict *d, const void *key1, const void *key2) +static int dictSdsKeyCompare(dictCmpCache *cache, const void *key1, const void *key2) { int l1,l2; - UNUSED(d); + UNUSED(cache); l1 = sdslen((sds)key1); l2 = sdslen((sds)key2); diff --git a/src/redisassert.h b/src/redisassert.h index 179833187..4de8d5c58 100644 --- a/src/redisassert.h +++ b/src/redisassert.h @@ -26,4 +26,10 @@ void _serverAssert(const char *estr, const char *file, int line); void _serverPanic(const char *file, int line, const char *msg, ...); +#ifdef DEBUG_ASSERTIONS +#define debugAssert(_e) assert(_e) +#else +#define debugAssert(_e) ((void)0) +#endif + #endif diff --git a/src/sds.c b/src/sds.c index 7f1429c46..8684a0442 100644 --- a/src/sds.c +++ b/src/sds.c @@ -2,7 +2,10 @@ * * Copyright (c) 2006-Present, Redis Ltd. * All rights reserved. - * + * + * Copyright (c) 2024-present, Valkey contributors. + * All rights reserved. + * * Licensed under your choice of (a) the Redis Source Available License 2.0 * (RSALv2); or (b) the Server Side Public License v1 (SSPLv1); or (c) the * GNU Affero General Public License v3 (AGPLv3). @@ -20,32 +23,19 @@ const char *SDS_NOINIT = "SDS_NOINIT"; -static inline int sdsHdrSize(char type) { - switch(type&SDS_TYPE_MASK) { - case SDS_TYPE_5: - return sizeof(struct sdshdr5); - case SDS_TYPE_8: - return sizeof(struct sdshdr8); - case SDS_TYPE_16: - return sizeof(struct sdshdr16); - case SDS_TYPE_32: - return sizeof(struct sdshdr32); - case SDS_TYPE_64: - return sizeof(struct sdshdr64); - } - return 0; -} - -static inline char sdsReqType(size_t string_size) { - if (string_size < 1<<5) - return SDS_TYPE_5; - if (string_size < 1<<8) - return SDS_TYPE_8; - if (string_size < 1<<16) - return SDS_TYPE_16; +/* Returns the minimum SDS type required to store a string of the given length. + * + * Previously, the SDS type was determined solely by the logical string + * length, ignoring header size, which could lead to inconsistencies. If the + * allocated buffer was larger than the maximum size supported by a type, `alloc` + * would be truncated, creating a mismatch between `alloc` and the actual buffer size. + */ +char sdsReqType(size_t string_size) { + if (string_size < 1 << 5) return SDS_TYPE_5; + if (string_size <= (1 << 8) - sizeof(struct sdshdr8) - 1) return SDS_TYPE_8; + if (string_size <= (1 << 16) - sizeof(struct sdshdr16) - 1) return SDS_TYPE_16; #if (LONG_MAX == LLONG_MAX) - if (string_size < 1ll<<32) - return SDS_TYPE_32; + if (string_size <= (1ll << 32) - sizeof(struct sdshdr32) - 1) return SDS_TYPE_32; return SDS_TYPE_64; #else return SDS_TYPE_32; @@ -66,6 +56,32 @@ static inline size_t sdsTypeMaxSize(char type) { return -1; /* this is equivalent to the max SDS_TYPE_64 or SDS_TYPE_32 */ } +/* + * Adjusts the SDS type if the allocated buffer size exceeds the maximum size + * addressable by the current type. + * + * The SDS type is initially determined based on the logical length of the string. + * However, allocators like jemalloc may return a buffer larger than requested, + * potentially exceeding the maximum size the selected SDS type can handle. This + * can cause a mismatch between the `alloc` field and the actual buffer size, + * leading to wasted memory and possible inconsistencies. + * + * This function ensures that the SDS type is selected based on the actual buffer + * size rather than just the logical length. If the buffer size supports a larger + * SDS type, it updates `type` and `hdrlen` accordingly. + * + * Returns 1 if the type was adjusted, 0 otherwise. + */ +static inline int adjustTypeIfNeeded(char *type, int *hdrlen, size_t bufsize) { + size_t usable = bufsize - *hdrlen - 1; + if (*type != SDS_TYPE_5 && usable > sdsTypeMaxSize(*type)) { + *type = sdsReqType(usable); + *hdrlen = sdsHdrSize(*type); + return 1; + } + return 0; +} + /* Create a new sds string with the content specified by the 'init' pointer * and 'initlen'. * If NULL is used for 'init' the string is initialized with zero bytes. @@ -81,29 +97,45 @@ static inline size_t sdsTypeMaxSize(char type) { * \0 characters in the middle, as the length is stored in the sds header. */ sds _sdsnewlen(const void *init, size_t initlen, int trymalloc) { void *sh; - sds s; + char type = sdsReqType(initlen); /* Empty strings are usually created in order to append. Use type 8 * since type 5 is not good at this. */ if (type == SDS_TYPE_5 && initlen == 0) type = SDS_TYPE_8; int hdrlen = sdsHdrSize(type); - unsigned char *fp; /* flags pointer. */ - size_t usable; + size_t bufsize; assert(initlen + hdrlen + 1 > initlen); /* Catch size_t overflow */ sh = trymalloc? - s_trymalloc_usable(hdrlen+initlen+1, &usable) : - s_malloc_usable(hdrlen+initlen+1, &usable); + s_trymalloc_usable(hdrlen+initlen+1, &bufsize) : + s_malloc_usable(hdrlen+initlen+1, &bufsize); if (sh == NULL) return NULL; - if (init==SDS_NOINIT) - init = NULL; - else if (!init) - memset(sh, 0, hdrlen+initlen+1); - s = (char*)sh+hdrlen; - fp = ((unsigned char*)s)-1; - usable = usable-hdrlen-1; - if (usable > sdsTypeMaxSize(type)) - usable = sdsTypeMaxSize(type); + + adjustTypeIfNeeded(&type, &hdrlen, bufsize); + return sdsnewplacement(sh, bufsize, type, init, initlen); +} + +/* Initializes an SDS within pre-allocated buffer. Like, placement new in C++. + * + * Parameters: + * - `buf` : A pre-allocated buffer for the SDS. + * - `bufsize`: Total size of the buffer (>= `sdsReqSize(initlen, type)`). Can use + * a larger `bufsize` than required, but usable size won't be greater + * than `sdsTypeMaxSize(type)`. + * - `type` : The SDS type. Can assist `sdsReqType(length)` to compute the type. + * - `init` : Initial string to copy, or `SDS_NOINIT` to skip initialization. + * - `initlen`: Length of the initial string. + * + * Returns: + * - A pointer to the SDS inside `buf`. + */ +sds sdsnewplacement(char *buf, size_t bufsize, char type, const char *init, size_t initlen) { + assert(bufsize >= sdsReqSize(initlen, type)); + int hdrlen = sdsHdrSize(type); + size_t usable = bufsize - hdrlen - 1; + sds s = buf + hdrlen; + unsigned char *fp = ((unsigned char *)s) - 1; /* flags pointer. */ + switch(type) { case SDS_TYPE_5: { *fp = type | (initlen << SDS_TYPE_BITS); @@ -112,6 +144,7 @@ sds _sdsnewlen(const void *init, size_t initlen, int trymalloc) { case SDS_TYPE_8: { SDS_HDR_VAR(8,s); sh->len = initlen; + debugAssert(usable <= sdsTypeMaxSize(type)); sh->alloc = usable; *fp = type; break; @@ -119,6 +152,7 @@ sds _sdsnewlen(const void *init, size_t initlen, int trymalloc) { case SDS_TYPE_16: { SDS_HDR_VAR(16,s); sh->len = initlen; + debugAssert(usable <= sdsTypeMaxSize(type)); sh->alloc = usable; *fp = type; break; @@ -126,6 +160,7 @@ sds _sdsnewlen(const void *init, size_t initlen, int trymalloc) { case SDS_TYPE_32: { SDS_HDR_VAR(32,s); sh->len = initlen; + debugAssert(usable <= sdsTypeMaxSize(type)); sh->alloc = usable; *fp = type; break; @@ -133,13 +168,19 @@ sds _sdsnewlen(const void *init, size_t initlen, int trymalloc) { case SDS_TYPE_64: { SDS_HDR_VAR(64,s); sh->len = initlen; + debugAssert(usable <= sdsTypeMaxSize(type)); sh->alloc = usable; *fp = type; break; } } - if (initlen && init) + if (init == SDS_NOINIT) + init = NULL; + else if (!init) + memset(s, 0, initlen); + else if (initlen) memcpy(s, init, initlen); + s[initlen] = '\0'; return s; } @@ -224,9 +265,10 @@ sds _sdsMakeRoomFor(sds s, size_t addlen, int greedy) { void *sh, *newsh; size_t avail = sdsavail(s); size_t len, newlen, reqlen; - char type, oldtype = s[-1] & SDS_TYPE_MASK; + char type, oldtype = sdsType(s); int hdrlen; - size_t usable; + size_t bufsize, usable; + int use_realloc; /* Return ASAP if there is enough space left. */ if (avail >= addlen) return s; @@ -251,24 +293,31 @@ sds _sdsMakeRoomFor(sds s, size_t addlen, int greedy) { hdrlen = sdsHdrSize(type); assert(hdrlen + newlen + 1 > reqlen); /* Catch size_t overflow */ - if (oldtype==type) { - newsh = s_realloc_usable(sh, hdrlen+newlen+1, &usable); + use_realloc = (oldtype == type); + if (use_realloc) { + newsh = s_realloc_usable(sh, hdrlen + newlen + 1, &bufsize); if (newsh == NULL) return NULL; - s = (char*)newsh+hdrlen; + s = (char*)newsh + hdrlen; + if (adjustTypeIfNeeded(&type, &hdrlen, bufsize)) { + memmove((char *)newsh + hdrlen, s, len + 1); + s = (char *)newsh + hdrlen; + s[-1] = type; + sdssetlen(s, len); + } } else { /* Since the header size changes, need to move the string forward, * and can't use realloc */ - newsh = s_malloc_usable(hdrlen+newlen+1, &usable); + newsh = s_malloc_usable(hdrlen + newlen + 1, &bufsize); if (newsh == NULL) return NULL; + adjustTypeIfNeeded(&type, &hdrlen, bufsize); memcpy((char*)newsh+hdrlen, s, len+1); s_free(sh); s = (char*)newsh+hdrlen; s[-1] = type; sdssetlen(s, len); } - usable = usable-hdrlen-1; - if (usable > sdsTypeMaxSize(type)) - usable = sdsTypeMaxSize(type); + usable = bufsize - hdrlen - 1; + assert(type == SDS_TYPE_5 || usable <= sdsTypeMaxSize(type)); sdssetalloc(s, usable); return s; } @@ -304,8 +353,8 @@ sds sdsRemoveFreeSpace(sds s, int would_regrow) { * allocation size, this is done in order to avoid repeated calls to this * function when the caller detects that it has excess space. */ sds sdsResize(sds s, size_t size, int would_regrow) { - void *sh, *newsh; - char type, oldtype = s[-1] & SDS_TYPE_MASK; + void *sh, *newsh = NULL; + char type, oldtype = sdsType(s); int hdrlen, oldhdrlen = sdsHdrSize(oldtype); size_t len = sdslen(s); sh = (char*)s-oldhdrlen; @@ -332,6 +381,8 @@ sds sdsResize(sds s, size_t size, int would_regrow) { * type. */ int use_realloc = (oldtype==type || (type < oldtype && type > SDS_TYPE_8)); size_t newlen = use_realloc ? oldhdrlen+size+1 : hdrlen+size+1; + size_t bufsize = 0; + size_t newsize; if (use_realloc) { int alloc_already_optimal = 0; @@ -340,24 +391,37 @@ sds sdsResize(sds s, size_t size, int would_regrow) { * We aim to avoid calling realloc() when using Jemalloc if there is no * change in the allocation size, as it incurs a cost even if the * allocation size stays the same. */ - alloc_already_optimal = (je_nallocx(newlen, 0) == zmalloc_size(sh)); + bufsize = sdsAllocSize(s); + alloc_already_optimal = (je_nallocx(newlen, 0) == bufsize); #endif if (!alloc_already_optimal) { - newsh = s_realloc(sh, newlen); + newsh = s_realloc_usable(sh, newlen, &bufsize); if (newsh == NULL) return NULL; - s = (char*)newsh+oldhdrlen; + s = (char *)newsh + oldhdrlen; + + if (adjustTypeIfNeeded(&oldtype, &oldhdrlen, bufsize)) { + memmove((char *)newsh + oldhdrlen, s, len + 1); + s = (char *)newsh + oldhdrlen; + s[-1] = oldtype; + sdssetlen(s, len); + } } + newsize = bufsize - oldhdrlen - 1; + debugAssert(oldtype == SDS_TYPE_5 || newsize <= sdsTypeMaxSize(oldtype)); } else { - newsh = s_malloc(newlen); + newsh = s_malloc_usable(newlen, &bufsize); if (newsh == NULL) return NULL; - memcpy((char*)newsh+hdrlen, s, len); + adjustTypeIfNeeded(&type, &hdrlen, bufsize); + memcpy((char *)newsh + hdrlen, s, len + 1); s_free(sh); - s = (char*)newsh+hdrlen; + s = (char *)newsh + hdrlen; s[-1] = type; + newsize = bufsize - hdrlen - 1; + debugAssert(type == SDS_TYPE_5 || newsize <= sdsTypeMaxSize(type)); } s[len] = 0; sdssetlen(s, len); - sdssetalloc(s, size); + sdssetalloc(s, newsize); return s; } @@ -403,12 +467,11 @@ void *sdsAllocPtr(sds s) { * sdsIncrLen(s, nread); */ void sdsIncrLen(sds s, ssize_t incr) { - unsigned char flags = s[-1]; size_t len; - switch(flags&SDS_TYPE_MASK) { + switch(sdsType(s)) { case SDS_TYPE_5: { unsigned char *fp = ((unsigned char*)s)-1; - unsigned char oldlen = SDS_TYPE_5_LEN(flags); + unsigned char oldlen = SDS_TYPE_5_LEN(s); assert((incr > 0 && oldlen+incr < 32) || (incr < 0 && oldlen >= (unsigned int)(-incr))); *fp = SDS_TYPE_5 | ((oldlen+incr) << SDS_TYPE_BITS); len = oldlen+incr; @@ -1401,7 +1464,7 @@ int sdsTest(int argc, char **argv, int flags) { for (i = 0; i < 10; i++) { size_t oldlen = sdslen(x); x = sdsMakeRoomFor(x,step); - int type = x[-1]&SDS_TYPE_MASK; + int type = sdsType(x); test_cond("sdsMakeRoomFor() len", sdslen(x) == oldlen); if (type != SDS_TYPE_5) { @@ -1473,6 +1536,36 @@ int sdsTest(int argc, char **argv, int flags) { test_cond("sdsresize() crop strlen", strlen(x) == 4); test_cond("sdsresize() crop alloc", sdsalloc(x) == 4); sdsfree(x); + + { /* Test adjustTypeIfNeeded() */ + /* Test case: Type should be adjusted when buffer size exceeds max size for current type */ + char type = SDS_TYPE_8; + int hdrlen = sdsHdrSize(type); + size_t bufsize = (1<<8) + hdrlen + 1; /* Exceeds SDS_TYPE_8 max size */ + + int result = adjustTypeIfNeeded(&type, &hdrlen, bufsize); + test_cond("adjustTypeIfNeeded() returns 1 when type needs adjustment", result == 1); + test_cond("adjustTypeIfNeeded() adjusts type correctly", type == SDS_TYPE_16); + test_cond("adjustTypeIfNeeded() adjusts header length correctly", hdrlen == sdsHdrSize(SDS_TYPE_16)); + + /* Test case: Type should not be adjusted when buffer size is within max size for current type */ + type = SDS_TYPE_8; + hdrlen = sdsHdrSize(type); + bufsize = (1<<8) - 10 + hdrlen + 1; /* Within SDS_TYPE_8 max size */ + result = adjustTypeIfNeeded(&type, &hdrlen, bufsize); + test_cond("adjustTypeIfNeeded() returns 0 when type doesn't need adjustment", result == 0); + test_cond("adjustTypeIfNeeded() doesn't change type when not needed", type == SDS_TYPE_8); + test_cond("adjustTypeIfNeeded() doesn't change header length when not needed", hdrlen == sdsHdrSize(SDS_TYPE_8)); + + /* Test case 3: Type 5 should never be adjusted */ + type = SDS_TYPE_5; + hdrlen = sdsHdrSize(type); + bufsize = 1000; /* Large buffer size */ + result = adjustTypeIfNeeded(&type, &hdrlen, bufsize); + test_cond("adjustTypeIfNeeded() returns 0 for SDS_TYPE_5", result == 0); + test_cond("adjustTypeIfNeeded() doesn't change SDS_TYPE_5", type == SDS_TYPE_5); + test_cond("adjustTypeIfNeeded() doesn't change header length for SDS_TYPE_5", hdrlen == sdsHdrSize(SDS_TYPE_5)); + } } return 0; } diff --git a/src/sds.h b/src/sds.h index c4cbf6bf5..5736fd8cd 100644 --- a/src/sds.h +++ b/src/sds.h @@ -2,6 +2,9 @@ * * Copyright (c) 2006-Present, Redis Ltd. * All rights reserved. + * + * Copyright (c) 2024-present, Valkey contributors. + * All rights reserved. * * Licensed under your choice of (a) the Redis Source Available License 2.0 * (RSALv2); or (b) the Server Side Public License v1 (SSPLv1); or (c) the @@ -60,13 +63,16 @@ struct __attribute__ ((__packed__)) sdshdr64 { #define SDS_TYPE_BITS 3 #define SDS_HDR_VAR(T,s) struct sdshdr##T *sh = (void*)((s)-(sizeof(struct sdshdr##T))); #define SDS_HDR(T,s) ((struct sdshdr##T *)((s)-(sizeof(struct sdshdr##T)))) -#define SDS_TYPE_5_LEN(f) ((f)>>SDS_TYPE_BITS) +#define SDS_TYPE_5_LEN(s) (((unsigned char)(s[-1])) >> SDS_TYPE_BITS) + +static inline unsigned char sdsType(const sds s) { + unsigned char flags = s[-1]; + return flags & SDS_TYPE_MASK; +} static inline size_t sdslen(const sds s) { - unsigned char flags = s[-1]; - switch(flags&SDS_TYPE_MASK) { - case SDS_TYPE_5: - return SDS_TYPE_5_LEN(flags); + switch (sdsType(s)) { + case SDS_TYPE_5: return SDS_TYPE_5_LEN(s); case SDS_TYPE_8: return SDS_HDR(8,s)->len; case SDS_TYPE_16: @@ -80,8 +86,7 @@ static inline size_t sdslen(const sds s) { } static inline size_t sdsavail(const sds s) { - unsigned char flags = s[-1]; - switch(flags&SDS_TYPE_MASK) { + switch(sdsType(s)) { case SDS_TYPE_5: { return 0; } @@ -106,8 +111,7 @@ static inline size_t sdsavail(const sds s) { } static inline void sdssetlen(sds s, size_t newlen) { - unsigned char flags = s[-1]; - switch(flags&SDS_TYPE_MASK) { + switch(sdsType(s)) { case SDS_TYPE_5: { unsigned char *fp = ((unsigned char*)s)-1; @@ -130,12 +134,11 @@ static inline void sdssetlen(sds s, size_t newlen) { } static inline void sdsinclen(sds s, size_t inc) { - unsigned char flags = s[-1]; - switch(flags&SDS_TYPE_MASK) { + switch(sdsType(s)) { case SDS_TYPE_5: { unsigned char *fp = ((unsigned char*)s)-1; - unsigned char newlen = SDS_TYPE_5_LEN(flags)+inc; + unsigned char newlen = SDS_TYPE_5_LEN(s)+inc; *fp = SDS_TYPE_5 | (newlen << SDS_TYPE_BITS); } break; @@ -156,10 +159,9 @@ static inline void sdsinclen(sds s, size_t inc) { /* sdsalloc() = sdsavail() + sdslen() */ static inline size_t sdsalloc(const sds s) { - unsigned char flags = s[-1]; - switch(flags&SDS_TYPE_MASK) { + switch(sdsType(s)) { case SDS_TYPE_5: - return SDS_TYPE_5_LEN(flags); + return SDS_TYPE_5_LEN(s); case SDS_TYPE_8: return SDS_HDR(8,s)->alloc; case SDS_TYPE_16: @@ -173,8 +175,7 @@ static inline size_t sdsalloc(const sds s) { } static inline void sdssetalloc(sds s, size_t newlen) { - unsigned char flags = s[-1]; - switch(flags&SDS_TYPE_MASK) { + switch(sdsType(s)) { case SDS_TYPE_5: /* Nothing to do, this type has no total allocation info. */ break; @@ -193,9 +194,27 @@ static inline void sdssetalloc(sds s, size_t newlen) { } } +static inline int sdsHdrSize(char type) { + switch(type&SDS_TYPE_MASK) { + case SDS_TYPE_5: + return sizeof(struct sdshdr5); + case SDS_TYPE_8: + return sizeof(struct sdshdr8); + case SDS_TYPE_16: + return sizeof(struct sdshdr16); + case SDS_TYPE_32: + return sizeof(struct sdshdr32); + case SDS_TYPE_64: + return sizeof(struct sdshdr64); + } + return 0; +} + sds sdsnewlen(const void *init, size_t initlen); sds sdstrynewlen(const void *init, size_t initlen); sds sdsnew(const char *init); +sds sdsnewplacement(char *buf, size_t bufsize, char type, const char *init, size_t initlen); + sds sdsempty(void); sds sdsdup(const sds s); void sdsfree(sds s); @@ -243,6 +262,7 @@ typedef sds (*sdstemplate_callback_t)(const sds variable, void *arg); sds sdstemplate(const char *template, sdstemplate_callback_t cb_func, void *cb_arg); /* Low level functions exposed to the user API */ +char sdsReqType(size_t string_size); sds sdsMakeRoomFor(sds s, size_t addlen); sds sdsMakeRoomForNonGreedy(sds s, size_t addlen); void sdsIncrLen(sds s, ssize_t incr); @@ -251,6 +271,12 @@ sds sdsResize(sds s, size_t size, int would_regrow); size_t sdsAllocSize(sds s); void *sdsAllocPtr(sds s); +/* Returns the minimum required size to store an sds string of the given length + * and type. */ +static inline size_t sdsReqSize(size_t len, char type) { + return len + sdsHdrSize(type) + 1; +} + /* Export the allocator used by SDS to the program using SDS. * Sometimes the program SDS is linked to, may use a different set of * allocators, but may want to allocate or free things that SDS will diff --git a/src/sentinel.c b/src/sentinel.c index bf749e9b4..51df378bf 100644 --- a/src/sentinel.c +++ b/src/sentinel.c @@ -4446,7 +4446,7 @@ void sentinelSetCommand(client *c) { /* If the target name is the same as the source name there * is no need to add an entry mapping to itself. */ - if (!dictSdsKeyCaseCompare(ri->renamed_commands,oldname,newname)) { + if (strcasecmp(oldname, newname) != 0) { oldname = sdsdup(oldname); newname = sdsdup(newname); dictAdd(ri->renamed_commands,oldname,newname); diff --git a/src/server.c b/src/server.c index 0ee76baf5..cbdd8964d 100644 --- a/src/server.c +++ b/src/server.c @@ -295,19 +295,53 @@ size_t dictSdsKeyLen(dict *d, const void *key) { return sdslen((sds)key); } -int dictSdsKeyCompareWithLen(dict *d, const void *key1, const size_t l1, - const void *key2, const size_t l2) -{ - UNUSED(d); +static uint64_t dictHashKV(const void *kv) { + sds sdsKey = kvobjGetKey((kvobj *) kv); + return dictGenHashFunction(sdsKey, sdslen(sdsKey)); +} + +int dictCompareKV(dictCmpCache *cache, const void *kv1, const void *kv2) { + /* Use caching to avoid compute key&len for each comparison on given lookup */ + if (cache->useCache == 0) { + cache->useCache = 1; + cache->data[0].p = kvobjGetKey((kvobj *) kv1); + cache->data[1].sz = sdslen((sds) cache->data[0].p); + } + + sds key1 = cache->data[0].p; + sds key2 = kvobjGetKey((kvobj *) kv2); + int l1 = (int) cache->data[1].sz; + int l2 = sdslen((sds)key2); if (l1 != l2) return 0; return memcmp(key1, key2, l1) == 0; } -int dictSdsKeyCompare(dict *d, const void *key1, +int dictSdsCompareKV(dictCmpCache *cache, const void *sdsLookup, const void *kv) +{ + /* is first cmp call of a new lookup */ + if (cache->useCache == 0) { + cache->useCache = 1; + cache->data[0].sz = sdslen((sds) sdsLookup); + } + + sds key2 = kvobjGetKey((kvobj *)kv); + size_t l1 = cache->data[0].sz; + size_t l2 = sdslen((sds)key2); + if (l1 != l2) return 0; + return memcmp(sdsLookup, key2, l1) == 0; +} + +static void dictDestructorKV(dict *d, void *kv) { + UNUSED(d); + if (kv == NULL) return; + decrRefCount(kv); +} + +int dictSdsKeyCompare(dictCmpCache *cache, const void *key1, const void *key2) { int l1,l2; - UNUSED(d); + UNUSED(cache); l1 = sdslen((sds)key1); l2 = sdslen((sds)key2); @@ -315,10 +349,10 @@ int dictSdsKeyCompare(dict *d, const void *key1, return memcmp(key1, key2, l1) == 0; } -int dictSdsMstrKeyCompare(dict *d, const void *sdsLookup, const void *mstrStored) +int dictSdsMstrKeyCompare(dictCmpCache *cache, const void *sdsLookup, const void *mstrStored) { int l1,l2; - UNUSED(d); + UNUSED(cache); l1 = sdslen((sds)sdsLookup); l2 = hfieldlen((hfield)mstrStored); @@ -329,10 +363,10 @@ int dictSdsMstrKeyCompare(dict *d, const void *sdsLookup, const void *mstrStored /* A case insensitive version used for the command lookup table and other * places where case insensitive non binary-safe comparison is needed. */ -int dictSdsKeyCaseCompare(dict *d, const void *key1, +int dictSdsKeyCaseCompare(dictCmpCache *cache, const void *key1, const void *key2) { - UNUSED(d); + UNUSED(cache); return strcasecmp(key1, key2) == 0; } @@ -354,11 +388,11 @@ void *dictSdsDup(dict *d, const void *key) { return sdsdup((const sds) key); } -int dictObjKeyCompare(dict *d, const void *key1, +int dictObjKeyCompare(dictCmpCache *cache, const void *key1, const void *key2) { const robj *o1 = key1, *o2 = key2; - return dictSdsKeyCompare(d, o1->ptr,o2->ptr); + return dictSdsKeyCompare(cache, o1->ptr,o2->ptr); } uint64_t dictObjHash(const void *key) { @@ -394,15 +428,15 @@ uint64_t dictClientHash(const void *key) { } /* Dict compare function for client */ -int dictClientKeyCompare(dict *d, const void *key1, const void *key2) { - UNUSED(d); +int dictClientKeyCompare(dictCmpCache *cache, const void *key1, const void *key2) { + UNUSED(cache); return ((client *)key1)->id == ((client *)key2)->id; } /* Dict compare function for null terminated string */ -int dictCStrKeyCompare(dict *d, const void *key1, const void *key2) { +int dictCStrKeyCompare(dictCmpCache *cache, const void *key1, const void *key2) { int l1,l2; - UNUSED(d); + UNUSED(cache); l1 = strlen((char*)key1); l2 = strlen((char*)key2); @@ -411,12 +445,12 @@ int dictCStrKeyCompare(dict *d, const void *key1, const void *key2) { } /* Dict case insensitive compare function for null terminated string */ -int dictCStrKeyCaseCompare(dict *d, const void *key1, const void *key2) { - UNUSED(d); +int dictCStrKeyCaseCompare(dictCmpCache *cache, const void *key1, const void *key2) { + UNUSED(cache); return strcasecmp(key1, key2) == 0; } -int dictEncObjKeyCompare(dict *d, const void *key1, const void *key2) +int dictEncObjKeyCompare(dictCmpCache *cache, const void *key1, const void *key2) { robj *o1 = (robj*) key1, *o2 = (robj*) key2; int cmp; @@ -431,7 +465,7 @@ int dictEncObjKeyCompare(dict *d, const void *key1, const void *key2) * objects as well. */ if (o1->refcount != OBJ_STATIC_REFCOUNT) o1 = getDecodedObject(o1); if (o2->refcount != OBJ_STATIC_REFCOUNT) o2 = getDecodedObject(o2); - cmp = dictSdsKeyCompare(d,o1->ptr,o2->ptr); + cmp = dictSdsKeyCompare(cache,o1->ptr,o2->ptr); if (o1->refcount != OBJ_STATIC_REFCOUNT) decrRefCount(o1); if (o2->refcount != OBJ_STATIC_REFCOUNT) decrRefCount(o2); return cmp; @@ -518,17 +552,19 @@ dictType zsetDictType = { NULL, /* allow to expand */ }; -/* Db->dict, keys are sds strings, vals are Redis objects. */ +/* Db->dict, keys are of type kvobj, unification of key and value */ dictType dbDictType = { - dictSdsHash, /* hash function */ - NULL, /* key dup */ - NULL, /* val dup */ - dictSdsKeyCompare, /* key compare */ - dictSdsDestructor, /* key destructor */ - dictObjectDestructor, /* val destructor */ - dictResizeAllowed, /* allow to resize */ - .keyLen = dictSdsKeyLen, /* key length */ - .keyCompareWithLen = dictSdsKeyCompareWithLen /* key compare with length */ + dictSdsHash, /* hash function */ + NULL, /* key dup */ + NULL, /* val dup */ + dictSdsCompareKV, /* lookup key compare */ + dictDestructorKV, /* key destructor */ + NULL, /* val destructor */ + dictResizeAllowed, /* allow to resize */ + .no_value = 1, /* keys and values are unified (kvobj) */ + .keys_are_odd = 0, /* simple kvobj (robj) struct */ + .storedHashFunction = dictHashKV, /* stored hash function */ + .storedKeyCompare = dictCompareKV, /* stored key compare */ }; /* Db->expires */ @@ -536,10 +572,14 @@ dictType dbExpiresDictType = { dictSdsHash, /* hash function */ NULL, /* key dup */ NULL, /* val dup */ - dictSdsKeyCompare, /* key compare */ + dictSdsCompareKV, /* key compare */ NULL, /* key destructor */ NULL, /* val destructor */ dictResizeAllowed, /* allow to resize */ + .no_value = 1, /* keys and values are unified (kvobj) */ + .keys_are_odd = 0, /* simple kvobj (robj) struct */ + .storedHashFunction = dictHashKV, /* stored hash function */ + .storedKeyCompare = dictCompareKV, /* stored key compare */ }; /* Command table. sds string -> command struct pointer. */ diff --git a/src/server.h b/src/server.h index fbaaaa78a..70064bdd8 100644 --- a/src/server.h +++ b/src/server.h @@ -66,6 +66,29 @@ typedef long long ustime_t; /* microsecond time type. */ #define REDISMODULE_CORE 1 typedef struct redisObject robj; + +/* kvobj - A specific type of robj that holds also embedded key + * + * Since robj is being overused as general purpose object, `kvobj` distincts only + * at the declarative level. This distinction assist to the clarity of the code + * and can optionally enforce explicit casting later on. An `robj` is identified + * to be `kvobj` if `iskvobj` flag is set. + * + * Example to kvobj layout with key "mykey" and expiration time: + * +--------------+--------------+--------------+--------------------+ + * | serverObject | Expiry Time | key-hdr-size | sdshdr5 "mykey" \0 | + * | 16 bytes | 8 byte | 1 byte | 1 + 5 + 1 | + * +--------------+--------------+--------------+--------------------+ + * + * Example to kvobj layout with key and embedded value "myvalue": + * +--------------+--------------+--------------------+----------------------+ + * | serverObject | key-hdr-size | sdshdr5 "mykey" \0 | sdshdr8 "myvalue" \0 | + * | 16 bytes | 1 byte | 1 + 5 + 1 | 3 + 7 + 1 | + * +--------------+--------------+--------------------+----------------------+ + * + */ +typedef struct redisObject kvobj; + #include "redismodule.h" /* Redis modules API defines. */ /* Following includes allow test functions to be called from Redis main() */ @@ -998,16 +1021,21 @@ struct RedisModuleDigest { #define LRU_CLOCK_MAX ((1<lru */ #define LRU_CLOCK_RESOLUTION 1000 /* LRU clock resolution in ms */ -#define OBJ_SHARED_REFCOUNT INT_MAX /* Global object never destroyed. */ -#define OBJ_STATIC_REFCOUNT (INT_MAX-1) /* Object allocated in the stack. */ +#define OBJ_REFCOUNT_BITS 30 +#define OBJ_SHARED_REFCOUNT ((1 << OBJ_REFCOUNT_BITS) - 1) /* Global object never destroyed. */ +#define OBJ_STATIC_REFCOUNT ((1 << OBJ_REFCOUNT_BITS) - 2) /* Object allocated in the stack. */ #define OBJ_FIRST_SPECIAL_REFCOUNT OBJ_STATIC_REFCOUNT + struct redisObject { unsigned type:4; unsigned encoding:4; unsigned lru:LRU_BITS; /* LRU time (relative to global lru_clock) or * LFU data (least significant 8 bits frequency * and most significant 16 bits access time). */ - int refcount; + unsigned iskvobj : 1; /* 1 if this struct serves as a kvobj base */ + unsigned expirable : 1; /* 1 if this key has expiration time attached. + * If set, then this object is of type kvobj */ + unsigned refcount : OBJ_REFCOUNT_BITS; void *ptr; }; @@ -1024,6 +1052,8 @@ char *getObjectTypeName(robj*); _var.refcount = OBJ_STATIC_REFCOUNT; \ _var.type = OBJ_STRING; \ _var.encoding = OBJ_ENCODING_RAW; \ + _var.expirable = 0; \ + _var.iskvobj = 0; \ _var.ptr = _ptr; \ } while(0) @@ -2719,7 +2749,7 @@ void moduleUnblockClient(client *c); int moduleBlockedClientMayTimeout(client *c); int moduleClientIsBlockedOnKeys(client *c); void moduleNotifyUserChanged(client *c); -void moduleNotifyKeyUnlink(robj *key, robj *val, int dbid, int flags); +void moduleNotifyKeyUnlink(robj *key, kvobj *kv, int dbid, int flags); size_t moduleGetFreeEffort(robj *key, robj *val, int dbid); size_t moduleGetMemUsage(robj *key, robj *val, size_t sample_size, int dbid); robj *moduleTypeDupOrReply(client *c, robj *fromkey, robj *tokey, int todb, robj *value); @@ -2961,7 +2991,6 @@ void execCommandAbort(client *c, sds error); /* Redis object implementation */ void decrRefCount(robj *o); -void decrRefCountVoid(void *o); void incrRefCount(robj *o); robj *makeObjectShared(robj *o); void freeStringObject(robj *o); @@ -2974,7 +3003,6 @@ robj *createObject(int type, void *ptr); void initObjectLRUOrLFU(robj *o); robj *createStringObject(const char *ptr, size_t len); robj *createRawStringObject(const char *ptr, size_t len); -robj *createEmbeddedStringObject(const char *ptr, size_t len); robj *tryCreateRawStringObject(const char *ptr, size_t len); robj *tryCreateStringObject(const char *ptr, size_t len); robj *dupStringObject(const robj *o); @@ -3018,6 +3046,12 @@ unsigned long long estimateObjectIdleTime(robj *o); void trimStringObjectIfNeeded(robj *o, int trim_small_values); #define sdsEncodedObject(objptr) (objptr->encoding == OBJ_ENCODING_RAW || objptr->encoding == OBJ_ENCODING_EMBSTR) +kvobj *kvobjCreate(int type, const sds key, void *ptr, long long expire); +kvobj *kvobjSet(sds key, robj *val, long long expire); +kvobj *kvobjSetExpire(kvobj *kv, long long expire); +sds kvobjGetKey(const kvobj *kv); +long long kvobjGetExpire(const kvobj *val); + /* Synchronous I/O with timeout */ ssize_t syncWrite(int fd, char *ptr, ssize_t size, long long timeout); ssize_t syncRead(int fd, char *ptr, ssize_t size, long long timeout); @@ -3345,8 +3379,9 @@ int calculateKeySlot(sds key); /* kvstore wrappers */ int dbExpand(redisDb *db, uint64_t db_size, int try_expand); int dbExpandExpires(redisDb *db, uint64_t db_size, int try_expand); -dictEntry *dbFind(redisDb *db, void *key); -dictEntry *dbFindExpires(redisDb *db, void *key); +kvobj *dbFind(redisDb *db, sds key); +kvobj *dbFindByLink(redisDb *db, sds key, dictEntryLink *link); +kvobj *dbFindExpires(redisDb *db, sds key); unsigned long long dbSize(redisDb *db); unsigned long long dbScan(redisDb *db, unsigned long long cursor, dictScanFunction *scan_cb, void *privdata); @@ -3376,10 +3411,6 @@ typedef struct listpackEx { minimum, hash-field to expire. TTL value might be inaccurate up-to few seconds due to optimization consideration. */ - sds key; /* reference to the key, same one that stored in - db->dict. Will be used from active-expiration flow - for notification and deletion of the object, if - needed. */ void *lp; /* listpack that contains 'key-value-ttl' tuples which are ordered by ttl. */ } listpackEx; @@ -3394,10 +3425,6 @@ typedef struct dictExpireMetadata { inaccurate up-to few seconds due to optimization consideration. */ ebuckets hfe; /* DS of Hash Fields Expiration, associated to each hash */ - sds key; /* reference to the key, same one that stored in - db->dict. Will be used from active-expiration flow - for notification and deletion of the object, if - needed. */ } dictExpireMetadata; /* Hash data type */ @@ -3418,7 +3445,7 @@ typedef struct dictExpireMetadata { void hashTypeConvert(robj *o, int enc, ebuckets *hexpires); void hashTypeTryConversion(redisDb *db, robj *subject, robj **argv, int start, int end); -int hashTypeExists(redisDb *db, robj *o, sds key, int hfeFlags, int *isHashDeleted); +int hashTypeExists(redisDb *db, kvobj *kv, sds field, int hfeFlags, int *isHashDeleted); int hashTypeDelete(robj *o, void *key, int isSdsField); unsigned long hashTypeLength(const robj *o, int subtractExpiredFields); hashTypeIterator *hashTypeInitIterator(robj *subject); @@ -3435,19 +3462,18 @@ void hashTypeCurrentObject(hashTypeIterator *hi, int what, unsigned char **vstr, unsigned int *vlen, long long *vll, uint64_t *expireTime); sds hashTypeCurrentObjectNewSds(hashTypeIterator *hi, int what); hfield hashTypeCurrentObjectNewHfield(hashTypeIterator *hi); -int hashTypeGetValueObject(redisDb *db, robj *o, sds field, int hfeFlags, +int hashTypeGetValueObject(redisDb *db, kvobj *kv, sds field, int hfeFlags, robj **val, uint64_t *expireTime, int *isHashDeleted); int hashTypeSet(redisDb *db, robj *o, sds field, sds value, int flags); -robj *hashTypeDup(robj *o, sds newkey, uint64_t *minHashExpire); +robj *hashTypeDup(kvobj *kv, uint64_t *minHashExpire); uint64_t hashTypeRemoveFromExpires(ebuckets *hexpires, robj *o); -void hashTypeAddToExpires(redisDb *db, sds key, robj *hashObj, uint64_t expireTime); +void hashTypeAddToExpires(redisDb *db, kvobj *hashObj, uint64_t expireTime); void hashTypeFree(robj *o); int hashTypeIsExpired(const robj *o, uint64_t expireAt); unsigned char *hashTypeListpackGetLp(robj *o); uint64_t hashTypeGetMinExpire(robj *o, int accurate); -void hashTypeUpdateKeyRef(robj *o, sds newkey); ebuckets *hashTypeGetDictMetaHFE(dict *d); -void initDictExpireMetadata(sds key, robj *o); +void initDictExpireMetadata(robj *o); struct listpackEx *listpackExCreate(void); void listpackExAddNew(robj *o, char *field, size_t flen, char *value, size_t vlen, uint64_t expireAt); @@ -3561,21 +3587,21 @@ int removeExpire(redisDb *db, robj *key); void deleteExpiredKeyAndPropagate(redisDb *db, robj *keyobj); void deleteEvictedKeyAndPropagate(redisDb *db, robj *keyobj, long long *key_mem_freed); void propagateDeletion(redisDb *db, robj *key, int lazy); -int keyIsExpired(redisDb *db, robj *key); -long long getExpire(redisDb *db, robj *key); -void setExpire(client *c, redisDb *db, robj *key, long long when); -void setExpireWithDictEntry(client *c, redisDb *db, robj *key, long long when, dictEntry *kde); +int keyIsExpired(redisDb *db, sds key, kvobj *kv); +long long getExpire(redisDb *db, sds key, kvobj *kv); +kvobj *setExpire(client *c, redisDb *db, robj *key, long long when); +kvobj *setExpireByLink(client *c, redisDb *db, sds key, long long when, dictEntryLink link); int checkAlreadyExpired(long long when); int parseExtendedExpireArgumentsOrReply(client *c, int *flags); -robj *lookupKeyRead(redisDb *db, robj *key); -robj *lookupKeyWrite(redisDb *db, robj *key); -robj *lookupKeyWriteWithDictEntry(redisDb *db, robj *key, dictEntry **deref); -robj *lookupKeyReadOrReply(client *c, robj *key, robj *reply); -robj *lookupKeyWriteOrReply(client *c, robj *key, robj *reply); -robj *lookupKeyReadWithFlags(redisDb *db, robj *key, int flags); -robj *lookupKeyWriteWithFlags(redisDb *db, robj *key, int flags); -robj *objectCommandLookup(client *c, robj *key); -robj *objectCommandLookupOrReply(client *c, robj *key, robj *reply); +kvobj *lookupKeyRead(redisDb *db, robj *key); +kvobj *lookupKeyWrite(redisDb *db, robj *key); +kvobj *lookupKeyWriteWithLink(redisDb *db, robj *key, dictEntryLink *link); +kvobj *lookupKeyReadOrReply(client *c, robj *key, robj *reply); +kvobj *lookupKeyWriteOrReply(client *c, robj *key, robj *reply); +kvobj *lookupKeyReadWithFlags(redisDb *db, robj *key, int flags); +kvobj *lookupKeyWriteWithFlags(redisDb *db, robj *key, int flags); +kvobj *kvobjCommandLookup(client *c, robj *key); +kvobj *kvobjCommandLookupOrReply(client *c, robj *key, robj *reply); int objectSetLRUOrLFU(robj *val, long long lfu_freq, long long lru_idle, long long lru_clock, int lru_multiplier); #define LOOKUP_NONE 0 @@ -3587,25 +3613,27 @@ int objectSetLRUOrLFU(robj *val, long long lfu_freq, long long lru_idle, #define LOOKUP_ACCESS_EXPIRED (1<<5) /* Allow lookup to expired key. */ #define LOOKUP_NOEFFECTS (LOOKUP_NONOTIFY | LOOKUP_NOSTATS | LOOKUP_NOTOUCH | LOOKUP_NOEXPIRE) /* Avoid any effects from fetching the key */ -dictEntry *dbAdd(redisDb *db, robj *key, robj *val); -int dbAddRDBLoad(redisDb *db, sds key, robj *val); -void dbReplaceValue(redisDb *db, robj *key, robj *val); -void dbReplaceValueWithDictEntry(redisDb *db, robj *key, robj *val, dictEntry *de); +static inline kvobj *dictGetKV(const dictEntry *de) {return (kvobj *) dictGetKey(de);} +kvobj *dbAdd(redisDb *db, robj *key, robj **valref); +kvobj *dbAddByLink(redisDb *db, robj *key, robj **valref, dictEntryLink *link); +kvobj *dbAddRDBLoad(redisDb *db, sds key, robj **valref, long long expire); +void dbReplaceValue(redisDb *db, robj *key, kvobj **ioKeyVal); +void dbReplaceValueWithLink(redisDb *db, robj *key, robj **val, dictEntryLink link); #define SETKEY_KEEPTTL 1 #define SETKEY_NO_SIGNAL 2 #define SETKEY_ALREADY_EXIST 4 #define SETKEY_DOESNT_EXIST 8 -#define SETKEY_ADD_OR_UPDATE 16 /* Key most likely doesn't exists */ -void setKey(client *c, redisDb *db, robj *key, robj *val, int flags); -void setKeyWithDictEntry(client *c, redisDb *db, robj *key, robj *val, int flags, dictEntry *de); + +void setKey(client *c, redisDb *db, robj *key, robj **ioval, int flags); +void setKeyByLink(client *c, redisDb *db, robj *key, robj **valref, int flags, dictEntryLink *link); robj *dbRandomKey(redisDb *db); int dbGenericDelete(redisDb *db, robj *key, int async, int flags); int dbSyncDelete(redisDb *db, robj *key); int dbDelete(redisDb *db, robj *key); int dbDeleteSkipKeysizesUpdate(redisDb *db, robj *key); -robj *dbUnshareStringValue(redisDb *db, robj *key, robj *o); -robj *dbUnshareStringValueWithDictEntry(redisDb *db, robj *key, robj *o, dictEntry *de); +kvobj *dbUnshareStringValue(redisDb *db, robj *key, kvobj *o); +kvobj *dbUnshareStringValueByLink(redisDb *db, robj *key, kvobj *kv, dictEntryLink link); #define FLUSH_TYPE_ALL 0 #define FLUSH_TYPE_DB 1 @@ -3755,7 +3783,7 @@ int clientsCronHandleTimeout(client *c, mstime_t now_ms); /* expire.c -- Handling of expired keys */ void activeExpireCycle(int type); void expireSlaveKeys(void); -void rememberSlaveKeyWithExpire(redisDb *db, robj *key); +void rememberSlaveKeyWithExpire(redisDb *db, sds key); void flushSlaveKeysWithExpireList(void); size_t getSlaveKeyWithExpireCount(void); uint64_t hashTypeDbActiveExpire(redisDb *db, uint32_t maxFieldsToExpire); @@ -3777,10 +3805,9 @@ uint64_t dictSdsHash(const void *key); uint64_t dictPtrHash(const void *key); uint64_t dictSdsCaseHash(const void *key); size_t dictSdsKeyLen(dict *d, const void *key); -int dictSdsKeyCompare(dict *d, const void *key1, const void *key2); -int dictSdsKeyCompareWithLen(dict *d, const void *key1, const size_t l1,const void *key2, const size_t l2); -int dictSdsMstrKeyCompare(dict *d, const void *sdsLookup, const void *mstrStored); -int dictSdsKeyCaseCompare(dict *d, const void *key1, const void *key2); +int dictSdsKeyCompare(dictCmpCache *cache, const void *key1, const void *key2); +int dictSdsMstrKeyCompare(dictCmpCache *cache, const void *sdsLookup, const void *mstrStored); +int dictSdsKeyCaseCompare(dictCmpCache *cache, const void *key1, const void *key2); void dictSdsDestructor(dict *d, void *val); void dictListDestructor(dict *d, void *val); void *dictSdsDup(dict *d, const void *key); diff --git a/src/sort.c b/src/sort.c index 1aed59777..3ae6e9f45 100644 --- a/src/sort.c +++ b/src/sort.c @@ -40,9 +40,11 @@ redisSortOperation *createSortOperation(int type, robj *pattern) { * The returned object will always have its refcount increased by 1 * when it is non-NULL. */ robj *lookupKeyByPattern(redisDb *db, robj *pattern, robj *subst) { + kvobj *kv; char *p, *f, *k; sds spat, ssub; - robj *keyobj, *fieldobj = NULL, *o, *val; + robj *keyobj, *fieldobj = NULL, *val; + int prefixlen, sublen, postfixlen, fieldlen; /* If the pattern is "#" return the substitution object itself in order @@ -87,31 +89,31 @@ robj *lookupKeyByPattern(redisDb *db, robj *pattern, robj *subst) { decrRefCount(subst); /* Incremented by decodeObject() */ /* Lookup substituted key */ - o = lookupKeyRead(db, keyobj); - if (o == NULL) goto noobj; + kv = lookupKeyRead(db, keyobj); + if (kv == NULL) goto noobj; if (fieldobj) { - if (o->type != OBJ_HASH) goto noobj; + if (kv->type != OBJ_HASH) goto noobj; /* Retrieve value from hash by the field name. The returned object * is a new object with refcount already incremented. */ int isHashDeleted; - hashTypeGetValueObject(db, o, fieldobj->ptr, HFE_LAZY_EXPIRE, &val, NULL, &isHashDeleted); - o = val; + hashTypeGetValueObject(db, kv, fieldobj->ptr, HFE_LAZY_EXPIRE, &val, NULL, &isHashDeleted); + kv = val; if (isHashDeleted) goto noobj; } else { - if (o->type != OBJ_STRING) goto noobj; + if (kv->type != OBJ_STRING) goto noobj; /* Every object that this function returns needs to have its refcount * increased. sortCommand decreases it again. */ - incrRefCount(o); + incrRefCount(kv); } decrRefCount(keyobj); if (fieldobj) decrRefCount(fieldobj); - return o; + return kv; noobj: decrRefCount(keyobj); @@ -600,18 +602,26 @@ void sortCommandGeneric(client *c, int readonly) { } } } + if (outputlen) { listTypeTryConversion(sobj,LIST_CONV_AUTO,NULL,NULL); - setKey(c,c->db,storekey,sobj,0); + setKey(c, c->db, storekey, &sobj, 0); + /* Ownership of sobj transferred to the db. Set to NULL to prevent + * freeing it below. */ + sobj = NULL; notifyKeyspaceEvent(NOTIFY_LIST,"sortstore",storekey, c->db->id); server.dirty += outputlen; - } else if (dbDelete(c->db,storekey)) { - signalModifiedKey(c,c->db,storekey); - notifyKeyspaceEvent(NOTIFY_GENERIC,"del",storekey,c->db->id); - server.dirty++; + /* Ownership of sobj transferred to the db. No need to free it. */ + } else { + if (dbDelete(c->db, storekey)) { + signalModifiedKey(c, c->db, storekey); + notifyKeyspaceEvent(NOTIFY_GENERIC, "del", storekey, c->db->id); + server.dirty++; + } + decrRefCount(sobj); } - decrRefCount(sobj); + addReplyLongLong(c,outputlen); } diff --git a/src/t_hash.c b/src/t_hash.c index 33417dd0b..2bbdf63f4 100644 --- a/src/t_hash.c +++ b/src/t_hash.c @@ -51,17 +51,17 @@ static ExpireMeta* hfieldGetExpireMeta(const eItem field); static ExpireMeta *hashGetExpireMeta(const eItem hash); static void hexpireGenericCommand(client *c, long long basetime, int unit); static ExpireAction hashTypeActiveExpire(eItem hashObj, void *ctx); -static uint64_t hashTypeExpire(robj *o, ExpireCtx *expireCtx, int updateGlobalHFE); +static uint64_t hashTypeExpire(kvobj *kv, ExpireCtx *expireCtx, int updateGlobalHFE); static void hfieldPersist(robj *hashObj, hfield field); static void propagateHashFieldDeletion(redisDb *db, sds key, char *field, size_t fieldLen); /* hash dictType funcs */ -static int dictHfieldKeyCompare(dict *d, const void *key1, const void *key2); +static int dictHfieldKeyCompare(dictCmpCache *cache, const void *key1, const void *key2); static uint64_t dictMstrHash(const void *key); static void dictHfieldDestructor(dict *d, void *field); static size_t hashDictWithExpireMetadataBytes(dict *d); static void hashDictWithExpireOnRelease(dict *d); -static robj* hashTypeLookupWriteOrCreate(client *c, robj *key); +static kvobj* hashTypeLookupWriteOrCreate(client *c, robj *key); /*----------------------------------------------------------------------------- * Define dictType of hash @@ -221,7 +221,7 @@ typedef struct HashTypeSetEx { const char *cmd; } HashTypeSetEx; -int hashTypeSetExInit(robj *key, robj *o, client *c, redisDb *db, +int hashTypeSetExInit(robj *key, kvobj *kvo, client *c, redisDb *db, ExpireSetCond expireSetCond, HashTypeSetEx *ex); SetExRes hashTypeSetEx(robj *o, sds field, uint64_t expireAt, HashTypeSetEx *exInfo); @@ -232,10 +232,10 @@ void hashTypeSetExDone(HashTypeSetEx *e); * Accessor functions for dictType of hash *----------------------------------------------------------------------------*/ -static int dictHfieldKeyCompare(dict *d, const void *key1, const void *key2) +static int dictHfieldKeyCompare(dictCmpCache *cache, const void *key1, const void *key2) { int l1,l2; - UNUSED(d); + UNUSED(cache); l1 = hfieldlen((hfield)key1); l2 = hfieldlen((hfield)key2); @@ -282,12 +282,12 @@ static void hashDictWithExpireOnRelease(dict *d) { /* * If any of hash field expiration command is called on a listpack hash object * for the first time, we convert it to OBJ_ENCODING_LISTPACK_EX encoding. - * We allocate "struct listpackEx" which holds listpack pointer and metadata to - * register key to the global DS. In the listpack, we append another TTL entry - * for each field-value pair. From now on, listpack will have triplets in it: - * field-value-ttl. If TTL is not set for a field, we store 'zero' as the TTL - * value. 'zero' is encoded as two bytes in the listpack. Memory overhead of a - * non-existing TTL will be two bytes per field. + * We allocate "struct listpackEx" which holds listpack pointer and expiry + * metadata. In the listpack string, we append another TTL entry for each field + * value pair. From now on, listpack will have triplets in it: field-value-ttl. + * If TTL is not set for a field, we store 'zero' as the TTL value. 'zero' is + * encoded as two bytes in the listpack. Memory overhead of a non-existing TTL + * will be two bytes per field. * * Fields in the listpack will be ordered by TTL. Field with the smallest expiry * time will be the first item. Fields without TTL will be at the end of the @@ -300,7 +300,6 @@ struct listpackEx *listpackExCreate(void) { listpackEx *lpt = zcalloc(sizeof(*lpt)); lpt->meta.trash = 1; lpt->lp = NULL; - lpt->key = NULL; return lpt; } @@ -387,14 +386,16 @@ static uint64_t listpackExGetMinExpire(robj *o) { } /* Walk over fields and delete the expired ones. */ -void listpackExExpire(redisDb *db, robj *o, ExpireInfo *info) { - serverAssert(o->encoding == OBJ_ENCODING_LISTPACK_EX); +void listpackExExpire(redisDb *db, kvobj *kv, ExpireInfo *info) { + serverAssert(kv->encoding == OBJ_ENCODING_LISTPACK_EX); uint64_t expired = 0, min = EB_EXPIRE_TIME_INVALID; unsigned char *ptr; - listpackEx *lpt = o->ptr; + listpackEx *lpt = kv->ptr; ptr = lpFirst(lpt->lp); + sds key = kvobjGetKey(kv); + while (ptr != NULL && (info->itemsExpired < info->maxToExpire)) { long long val; int64_t flen; @@ -412,7 +413,7 @@ void listpackExExpire(redisDb *db, robj *o, ExpireInfo *info) { if (val == HASH_LP_NO_TTL || (uint64_t) val > info->now) break; - propagateHashFieldDeletion(db, ((listpackEx *) o->ptr)->key, (char *)((fref) ? fref : intbuf), flen); + propagateHashFieldDeletion(db, key, (char *)((fref) ? fref : intbuf), flen); server.stat_expired_subkeys++; ptr = lpNext(lpt->lp, ptr); @@ -426,10 +427,10 @@ void listpackExExpire(redisDb *db, robj *o, ExpireInfo *info) { /* update keysizes */ unsigned long l = lpLength(lpt->lp) / 3; - updateKeysizesHist(db, getKeySlot(lpt->key), OBJ_HASH, l + expired, l); + updateKeysizesHist(db, getKeySlot(key), OBJ_HASH, l + expired, l); } - min = hashTypeGetMinExpire(o, 1 /*accurate*/); + min = hashTypeGetMinExpire(kv, 1 /*accurate*/); info->nextExpireTime = min; } @@ -720,7 +721,7 @@ GetFieldRes hashTypeGetFromHashTable(robj *o, sds field, sds *value, uint64_t *e * expiredAt - if the field has an expiration time, it will be set to the expiration * time of the field. Otherwise, will be set to EB_EXPIRE_TIME_INVALID. */ -GetFieldRes hashTypeGetValue(redisDb *db, robj *o, sds field, unsigned char **vstr, +GetFieldRes hashTypeGetValue(redisDb *db, kvobj *o, sds field, unsigned char **vstr, unsigned int *vlen, long long *vll, int hfeFlags, uint64_t *expiredAt) { @@ -768,10 +769,7 @@ GetFieldRes hashTypeGetValue(redisDb *db, robj *o, sds field, unsigned char **vs (isPausedActionsWithUpdate(PAUSE_ACTION_EXPIRE))) return GETF_EXPIRED; - if (o->encoding == OBJ_ENCODING_LISTPACK_EX) - key = ((listpackEx *) o->ptr)->key; - else - key = ((dictExpireMetadata *) dictMetadata((dict*)o->ptr))->key; + key = kvobjGetKey(o); /* delete the field and propagate the deletion */ serverAssert(hashTypeDelete(o, field, 1) == 1); @@ -815,7 +813,7 @@ GetFieldRes hashTypeGetValue(redisDb *db, robj *o, sds field, unsigned char **vs * * Returns 1 if the field exists, and 0 when it doesn't. */ -int hashTypeGetValueObject(redisDb *db, robj *o, sds field, int hfeFlags, +int hashTypeGetValueObject(redisDb *db, kvobj *o, sds field, int hfeFlags, robj **val, uint64_t *expireTime, int *isHashDeleted) { unsigned char *vstr; unsigned int vlen; @@ -857,7 +855,7 @@ int hashTypeGetValueObject(redisDb *db, robj *o, sds field, int hfeFlags, * * Returns 1 if the field exists, and 0 when it doesn't. */ -int hashTypeExists(redisDb *db, robj *o, sds field, int hfeFlags, int *isHashDeleted) { +int hashTypeExists(redisDb *db, kvobj *o, sds field, int hfeFlags, int *isHashDeleted) { unsigned char *vstr = NULL; unsigned int vlen = UINT_MAX; long long vll = LLONG_MAX; @@ -973,27 +971,25 @@ int hashTypeSet(redisDb *db, robj *o, sds field, sds value, int flags) { } else if (o->encoding == OBJ_ENCODING_HT) { dict *ht = o->ptr; - dictEntry *de, *existing; - const uint64_t hash = dictGetHash(ht,field); + dictEntry *de; /* check if field already exists */ - existing = dictFindByHash(ht, field, hash); + dictEntryLink bucket, link = dictFindLink(ht, field, &bucket); /* check if field already exists */ - if (existing == NULL) { + if (link == NULL) { hfield newField = hfieldNew(field, sdslen(field), 0); - dictUseStoredKeyApi(ht, 1); - de = dictAddNonExistsByHash(ht, newField, hash); - dictUseStoredKeyApi(ht, 0); + dictSetKeyAtLink(ht, newField, &bucket, 1); + de = *bucket; } else { /* If attached TTL to the old field, then remove it from hash's * private ebuckets when HASH_SET_KEEP_TTL is not set. */ if (!(flags & HASH_SET_KEEP_TTL)) { - hfield oldField = dictGetKey(existing); + hfield oldField = dictGetKey(*link); hfieldPersist(o, oldField); } /* Free the old value */ - sdsfree(dictGetVal(existing)); + sdsfree(dictGetVal(*link)); update = 1; - de = existing; + de = *link; } if (flags & HASH_SET_TAKE_VALUE) { @@ -1133,17 +1129,16 @@ SetExRes hashTypeSetEx(robj *o, sds field, uint64_t expireAt, HashTypeSetEx *exI return HSETEX_OK; /* never reach here */ } -void initDictExpireMetadata(sds key, robj *o) { +void initDictExpireMetadata(robj *o) { dict *ht = o->ptr; dictExpireMetadata *m = (dictExpireMetadata *) dictMetadata(ht); - m->key = key; m->hfe = ebCreate(); /* Allocate HFE DS */ m->expireMeta.trash = 1; /* mark as trash (as long it wasn't ebAdd()) */ } /* Init HashTypeSetEx struct before calling hashTypeSetEx() */ -int hashTypeSetExInit(robj *key, robj *o, client *c, redisDb *db, +int hashTypeSetExInit(robj *key, kvobj *o, client *c, redisDb *db, ExpireSetCond expireSetCond, HashTypeSetEx *ex) { dict *ht = o->ptr; @@ -1158,25 +1153,6 @@ int hashTypeSetExInit(robj *key, robj *o, client *c, redisDb *db, /* Take care that HASH support expiration */ if (o->encoding == OBJ_ENCODING_LISTPACK) { hashTypeConvert(o, OBJ_ENCODING_LISTPACK_EX, &c->db->hexpires); - - listpackEx *lpt = o->ptr; - dictEntry *de = dbFind(c->db, key->ptr); - serverAssert(de != NULL); - lpt->key = dictGetKey(de); - } else if (o->encoding == OBJ_ENCODING_LISTPACK_EX) { - listpackEx *lpt = o->ptr; - - /* If the hash previously had HFEs but later no longer does, the key ref - * (lpt->key) in the hash might become outdated after a MOVE/COPY/RENAME/RESTORE - * operation. These commands maintain the key ref only if HFEs are present. - * That is, we can only be sure that key ref is valid as long as it is not - * "trash". (TODO: dbFind() can be avoided. Instead need to extend the - * lookupKey*() to return dictEntry). */ - if (lpt->meta.trash) { - dictEntry *de = dbFind(c->db, key->ptr); - serverAssert(de != NULL); - lpt->key = dictGetKey(de); - } } else if (o->encoding == OBJ_ENCODING_HT) { /* Take care dict has HFE metadata */ if (!isDictWithMetaHFE(ht)) { @@ -1187,25 +1163,10 @@ int hashTypeSetExInit(robj *key, robj *o, client *c, redisDb *db, /* Find the key in the keyspace. Need to keep reference to the key for * notifications or even removal of the hash */ - dictEntry *de = dbFind(db, key->ptr); - serverAssert(de != NULL); /* Fillup dict HFE metadata */ - m->key = dictGetKey(de); /* reference key in keyspace */ m->hfe = ebCreate(); /* Allocate HFE DS */ m->expireMeta.trash = 1; /* mark as trash (as long it wasn't ebAdd()) */ - } else { - dictExpireMetadata *m = (dictExpireMetadata *) dictMetadata(ht); - /* If the hash previously had HFEs but later no longer does, the key ref - * (m->key) in the hash might become outdated after a MOVE/COPY/RENAME/RESTORE - * operation. These commands maintain the key ref only if HFEs are present. - * That is, we can only be sure that key ref is valid as long as it is not - * "trash". */ - if (m->expireMeta.trash) { - dictEntry *de = dbFind(db, key->ptr); - serverAssert(de != NULL); - m->key = dictGetKey(de); /* reference key in keyspace */ - } } } @@ -1555,15 +1516,16 @@ hfield hashTypeCurrentObjectNewHfield(hashTypeIterator *hi) { return hf; } -static robj *hashTypeLookupWriteOrCreate(client *c, robj *key) { - robj *o = lookupKeyWrite(c->db,key); - if (checkType(c,o,OBJ_HASH)) return NULL; +static kvobj *hashTypeLookupWriteOrCreate(client *c, robj *key) { + dictEntryLink link; + kvobj *kv = lookupKeyWriteWithLink(c->db, key, &link); + if (checkType(c, kv, OBJ_HASH)) return NULL; - if (o == NULL) { - o = createHashObject(); - dbAdd(c->db,key,o); + if (kv == NULL) { + robj *o = createHashObject(); + kv = dbAddByLink(c->db, key, &o, &link); } - return o; + return kv; } @@ -1646,7 +1608,6 @@ void hashTypeConvertListpackEx(robj *o, int enc, ebuckets *hexpires) { dictExpireMeta = (dictExpireMetadata *) dictMetadata(dict); /* Fillup dict HFE metadata */ - dictExpireMeta->key = lpt->key; /* reference key in keyspace */ dictExpireMeta->hfe = ebCreate(); /* Allocate HFE DS */ dictExpireMeta->expireMeta.trash = 1; /* mark as trash (as long it wasn't ebAdd()) */ @@ -1700,7 +1661,7 @@ void hashTypeConvert(robj *o, int enc, ebuckets *hexpires) { * has the same encoding as the original one. * * The resulting object always has refcount set to 1 */ -robj *hashTypeDup(robj *o, sds newkey, uint64_t *minHashExpire) { +robj *hashTypeDup(kvobj *o, uint64_t *minHashExpire) { robj *hobj; hashTypeIterator *hi; @@ -1720,7 +1681,6 @@ robj *hashTypeDup(robj *o, sds newkey, uint64_t *minHashExpire) { *minHashExpire = ebGetMetaExpTime(&lpt->meta); listpackEx *dup = listpackExCreate(); - dup->key = newkey; size_t sz = lpBytes(lpt->lp); dup->lp = lpNew(sz); @@ -1740,7 +1700,6 @@ robj *hashTypeDup(robj *o, sds newkey, uint64_t *minHashExpire) { d = dictCreate(&mstrHashDictTypeWithHFE); dictExpireMetaSrc = (dictExpireMetadata *) dictMetadata((dict *) o->ptr); dictExpireMetaDst = (dictExpireMetadata *) dictMetadata(d); - dictExpireMetaDst->key = newkey; /* reference key in keyspace */ dictExpireMetaDst->hfe = ebCreate(); /* Allocate HFE DS */ dictExpireMetaDst->expireMeta.trash = 1; /* mark as trash (as long it wasn't ebAdd()) */ @@ -1844,7 +1803,7 @@ static ExpireAction hashTypeActiveExpire(eItem item, void *ctx) { if (expireCtx->fieldsToExpireQuota == 0) return ACT_STOP_ACTIVE_EXP; - uint64_t nextExpTime = hashTypeExpire((robj *) item, expireCtx, 0); + uint64_t nextExpTime = hashTypeExpire((kvobj *) item, expireCtx, 0); /* If hash has no more fields to expire or got deleted, indicate * to remove it from HFE DB to the caller ebExpire() */ @@ -1867,10 +1826,9 @@ static ExpireAction hashTypeActiveExpire(eItem item, void *ctx) { * - 0 if hash got deleted * - EB_EXPIRE_TIME_INVALID if no more fields to expire */ -static uint64_t hashTypeExpire(robj *o, ExpireCtx *expireCtx, int updateGlobalHFE) { +static uint64_t hashTypeExpire(kvobj *o, ExpireCtx *expireCtx, int updateGlobalHFE) { uint64_t noExpireLeftRes = EB_EXPIRE_TIME_INVALID; redisDb *db = expireCtx->db; - sds keystr = NULL; ExpireInfo info = {0}; if (o->encoding == OBJ_ENCODING_LISTPACK_EX) { @@ -1880,7 +1838,6 @@ static uint64_t hashTypeExpire(robj *o, ExpireCtx *expireCtx, int updateGlobalHF .itemsExpired = 0}; listpackExExpire(db, o, &info); - keystr = ((listpackEx*)o->ptr)->key; } else { serverAssert(o->encoding == OBJ_ENCODING_HT); @@ -1897,7 +1854,6 @@ static uint64_t hashTypeExpire(robj *o, ExpireCtx *expireCtx, int updateGlobalHF }; ebExpire(&dictExpireMeta->hfe, &hashFieldExpireBucketsType, &info); - keystr = dictExpireMeta->key; } /* Update quota left */ @@ -1907,6 +1863,7 @@ static uint64_t hashTypeExpire(robj *o, ExpireCtx *expireCtx, int updateGlobalHF * As a result, active-expire might not expire any fields, in such cases, * we don't need to send notifications or perform other operations for this key. */ if (info.itemsExpired) { + sds keystr = kvobjGetKey(o); robj *key = createStringObject(keystr, sdslen(keystr)); notifyKeyspaceEvent(NOTIFY_HASH, "hexpired", key, db->id); @@ -1914,8 +1871,8 @@ static uint64_t hashTypeExpire(robj *o, ExpireCtx *expireCtx, int updateGlobalHF ebRemove(&db->hexpires, &hashExpireBucketsType, o); if (hashTypeLength(o, 0) == 0) { - dbDelete(db, key); notifyKeyspaceEvent(NOTIFY_GENERIC, "del", key, db->id); + dbDelete(db, key); noExpireLeftRes = 0; } else { if ((updateGlobalHFE) && (info.nextExpireTime != EB_EXPIRE_TIME_INVALID)) @@ -1936,7 +1893,7 @@ static uint64_t hashTypeExpire(robj *o, ExpireCtx *expireCtx, int updateGlobalHF * Return 1 if the entire hash was deleted, 0 otherwise. * This function might be pricy in case there are many expired fields. */ -static int hashTypeExpireIfNeeded(redisDb *db, robj *o) { +static int hashTypeExpireIfNeeded(redisDb *db, kvobj *o) { uint64_t nextExpireTime; uint64_t minExpire = hashTypeGetMinExpire(o, 1 /*accurate*/); @@ -2045,7 +2002,6 @@ int hashTypeIsFieldsWithExpire(robj *o) { /* Add hash to global HFE DS and update key for notifications. * - * key - must be the same key instance that is persisted in db->dict * expireTime - expiration in msec. * If eq. 0 then the hash will be added to the global HFE DS with * the minimum expiration time that is already written in advance @@ -2054,13 +2010,12 @@ int hashTypeIsFieldsWithExpire(robj *o) { * * Precondition: It is a hash of type listpackex or HT with HFE metadata. */ -void hashTypeAddToExpires(redisDb *db, sds key, robj *hashObj, uint64_t expireTime) { +void hashTypeAddToExpires(redisDb *db, kvobj *hashObj, uint64_t expireTime) { if (expireTime > EB_EXPIRE_TIME_MAX) return; if (hashObj->encoding == OBJ_ENCODING_LISTPACK_EX) { listpackEx *lpt = hashObj->ptr; - lpt->key = key; expireTime = (expireTime) ? expireTime : ebGetMetaExpTime(&lpt->meta); ebAdd(&db->hexpires, &hashExpireBucketsType, hashObj, expireTime); } else if (hashObj->encoding == OBJ_ENCODING_HT) { @@ -2068,7 +2023,6 @@ void hashTypeAddToExpires(redisDb *db, sds key, robj *hashObj, uint64_t expireTi if (isDictWithMetaHFE(d)) { dictExpireMetadata *meta = (dictExpireMetadata *) dictMetadata(d); expireTime = (expireTime) ? expireTime : ebGetMetaExpTime(&meta->expireMeta); - meta->key = key; ebAdd(&db->hexpires, &hashExpireBucketsType, hashObj, expireTime); } } @@ -2077,7 +2031,7 @@ void hashTypeAddToExpires(redisDb *db, sds key, robj *hashObj, uint64_t expireTi /* DB active expire and update hashes with time-expiration on fields. * * The callback function hashTypeActiveExpire() is invoked for each hash registered - * in the HFE DB (db->expires) with an expiration-time less than or equal to the + * in the HFE DB (db->hexpires) with an expiration-time less than or equal to the * current time. This callback performs the following actions for each hash: * - If the hash has one or more fields to expire, it will delete those fields. * - If there are more fields to expire, it will update the hash with the next @@ -2126,19 +2080,6 @@ void hashTypeFree(robj *o) { } } -/* Attempts to update the reference to the new key. Now it's only used in defrag. */ -void hashTypeUpdateKeyRef(robj *o, sds newkey) { - if (o->encoding == OBJ_ENCODING_LISTPACK_EX) { - listpackEx *lpt = o->ptr; - lpt->key = newkey; - } else if (o->encoding == OBJ_ENCODING_HT && isDictWithMetaHFE(o->ptr)) { - dictExpireMetadata *dictExpireMeta = (dictExpireMetadata *)dictMetadata((dict*)o->ptr); - dictExpireMeta->key = newkey; - } else { - /* Nothing to do. */ - } -} - ebuckets *hashTypeGetDictMetaHFE(dict *d) { dictExpireMetadata *dictExpireMeta = (dictExpireMetadata *) dictMetadata(d); return &dictExpireMeta->hfe; @@ -2151,44 +2092,44 @@ ebuckets *hashTypeGetDictMetaHFE(dict *d) { void hsetnxCommand(client *c) { unsigned long hlen; int isHashDeleted; - robj *o; - if ((o = hashTypeLookupWriteOrCreate(c,c->argv[1])) == NULL) return; + robj *kv = hashTypeLookupWriteOrCreate(c,c->argv[1]); + if (kv == NULL) return; - if (hashTypeExists(c->db, o, c->argv[2]->ptr, HFE_LAZY_EXPIRE, &isHashDeleted)) { + if (hashTypeExists(c->db, kv, c->argv[2]->ptr, HFE_LAZY_EXPIRE, &isHashDeleted)) { addReply(c, shared.czero); return; } /* Field expired and in turn hash deleted. Create new one! */ if (isHashDeleted) { - o = createHashObject(); - dbAdd(c->db,c->argv[1],o); + robj *o = createHashObject(); + kv = dbAdd(c->db,c->argv[1],&o); } - hashTypeTryConversion(c->db, o,c->argv,2,3); - hashTypeSet(c->db, o,c->argv[2]->ptr,c->argv[3]->ptr,HASH_SET_COPY); + hashTypeTryConversion(c->db, kv, c->argv, 2, 3); + hashTypeSet(c->db, kv, c->argv[2]->ptr, c->argv[3]->ptr, HASH_SET_COPY); addReply(c, shared.cone); signalModifiedKey(c,c->db,c->argv[1]); notifyKeyspaceEvent(NOTIFY_HASH,"hset",c->argv[1],c->db->id); - hlen = hashTypeLength(o, 0); + hlen = hashTypeLength(kv, 0); updateKeysizesHist(c->db, getKeySlot(c->argv[1]->ptr), OBJ_HASH, hlen - 1, hlen); server.dirty++; } void hsetCommand(client *c) { int i, created = 0; - robj *o; + kvobj *kv; if ((c->argc % 2) == 1) { addReplyErrorArity(c); return; } - if ((o = hashTypeLookupWriteOrCreate(c,c->argv[1])) == NULL) return; - hashTypeTryConversion(c->db,o,c->argv,2,c->argc-1); + if ((kv = hashTypeLookupWriteOrCreate(c,c->argv[1])) == NULL) return; + hashTypeTryConversion(c->db, kv, c->argv, 2, c->argc-1); for (i = 2; i < c->argc; i += 2) - created += !hashTypeSet(c->db, o,c->argv[i]->ptr,c->argv[i+1]->ptr,HASH_SET_COPY); + created += !hashTypeSet(c->db, kv, c->argv[i]->ptr, c->argv[i+1]->ptr, HASH_SET_COPY); /* HMSET (deprecated) and HSET return value is different. */ char *cmdname = c->argv[0]->ptr; @@ -2200,7 +2141,7 @@ void hsetCommand(client *c) { addReply(c, shared.ok); } signalModifiedKey(c,c->db,c->argv[1]); - unsigned long l = hashTypeLength(o, 0); + unsigned long l = hashTypeLength(kv, 0); updateKeysizesHist(c->db, getKeySlot(c->argv[1]->ptr), OBJ_HASH, l - created, l); notifyKeyspaceEvent(NOTIFY_HASH,"hset",c->argv[1],c->db->id); server.dirty += (c->argc - 2)/2; @@ -2388,14 +2329,14 @@ void hsetexCommand(client *c) { int updated = 0, deleted = 0, set_expiry; long long expire_time = EB_EXPIRE_TIME_INVALID; int64_t oldlen, newlen; - robj *o; HashTypeSetEx setex; + dictEntryLink link; if (hsetexParseArgs(c, &flags, &expire_time, &expire_time_pos, &first_field_pos, &field_count) != C_OK) return; - o = lookupKeyWrite(c->db, c->argv[1]); + kvobj *o = lookupKeyWriteWithLink(c->db, c->argv[1], &link); if (checkType(c, o, OBJ_HASH)) return; @@ -2405,7 +2346,7 @@ void hsetexCommand(client *c) { return; } o = createHashObject(); - dbAdd(c->db, c->argv[1], o); + dbAddByLink(c->db, c->argv[1], &o, &link); } oldlen = (int64_t) hashTypeLength(o, 0); @@ -2508,7 +2449,7 @@ out: void hincrbyCommand(client *c) { long long value, incr, oldvalue; - robj *o; + kvobj *o; sds new; unsigned char *vstr; unsigned int vlen; @@ -2532,7 +2473,7 @@ void hincrbyCommand(client *c) { } else { /* Field expired and in turn hash deleted. Create new one! */ o = createHashObject(); - dbAdd(c->db,c->argv[1],o); + dbAdd(c->db,c->argv[1],&o); value = 0; updateKeysizesHist(c->db, getKeySlot(c->argv[1]->ptr), OBJ_HASH, 0, 1); } @@ -2555,7 +2496,7 @@ void hincrbyCommand(client *c) { void hincrbyfloatCommand(client *c) { long double value, incr; long long ll; - robj *o; + kvobj *o; sds new; unsigned char *vstr; unsigned int vlen; @@ -2584,7 +2525,7 @@ void hincrbyfloatCommand(client *c) { } else { /* Field expired and in turn hash deleted. Create new one! */ o = createHashObject(); - dbAdd(c->db,c->argv[1],o); + dbAdd(c->db, c->argv[1], &o); value = 0; updateKeysizesHist(c->db, getKeySlot(c->argv[1]->ptr), OBJ_HASH, 0, 1); } @@ -2614,7 +2555,7 @@ void hincrbyfloatCommand(client *c) { decrRefCount(newobj); } -static GetFieldRes addHashFieldToReply(client *c, robj *o, sds field, int hfeFlags) { +static GetFieldRes addHashFieldToReply(client *c, kvobj *o, sds field, int hfeFlags) { if (o == NULL) { addReplyNull(c); return GETF_NOT_FOUND; @@ -2638,7 +2579,7 @@ static GetFieldRes addHashFieldToReply(client *c, robj *o, sds field, int hfeFla } void hgetCommand(client *c) { - robj *o; + kvobj *o; if ((o = lookupKeyReadOrReply(c,c->argv[1],shared.null[c->resp])) == NULL || checkType(c,o,OBJ_HASH)) return; @@ -2648,13 +2589,12 @@ void hgetCommand(client *c) { void hmgetCommand(client *c) { GetFieldRes res = GETF_OK; - robj *o; int i; int expired = 0, deleted = 0; /* Don't abort when the key cannot be found. Non-existing keys are empty * hashes, where HMGET should respond with a series of null bulks. */ - o = lookupKeyRead(c->db, c->argv[1]); + kvobj *o = lookupKeyRead(c->db, c->argv[1]); if (checkType(c,o,OBJ_HASH)) return; addReplyArrayLen(c, c->argc-2); @@ -2686,9 +2626,8 @@ void hgetdelCommand(client *c) { int res = 0, hfe = 0, deleted = 0, expired = 0; int64_t oldlen = -1; /* not exists as long as it is not set */ long num_fields = 0; - robj *o; - o = lookupKeyWrite(c->db, c->argv[1]); + kvobj *o = lookupKeyWrite(c->db, c->argv[1]); if (checkType(c, o, OBJ_HASH)) return; @@ -2785,10 +2724,9 @@ void hgetexCommand(client *c) { long num_fields; int64_t oldlen = 0, newlen = -1; long long expire_time = 0; - robj *o; HashTypeSetEx setex; - o = lookupKeyWrite(c->db, c->argv[1]); + kvobj *o = lookupKeyWrite(c->db, c->argv[1]); if (checkType(c, o, OBJ_HASH)) return; @@ -2920,7 +2858,7 @@ void hgetexCommand(client *c) { /* Key may become empty due to lazy expiry in addHashFieldToReply() * or the new expiration time is in the past.*/ newlen = hashTypeLength(o, 0); - + updateKeysizesHist(c->db, getKeySlot(c->argv[1]->ptr), OBJ_HASH, oldlen, newlen); if (newlen == 0) { dbDelete(c->db, c->argv[1]); @@ -2929,7 +2867,7 @@ void hgetexCommand(client *c) { } void hdelCommand(client *c) { - robj *o; + kvobj *o; int j, deleted = 0, keyremoved = 0; if ((o = lookupKeyWriteOrReply(c,c->argv[1],shared.czero)) == NULL || @@ -2974,7 +2912,7 @@ void hdelCommand(client *c) { } void hlenCommand(client *c) { - robj *o; + kvobj *o; if ((o = lookupKeyReadOrReply(c,c->argv[1],shared.czero)) == NULL || checkType(c,o,OBJ_HASH)) return; @@ -2983,7 +2921,7 @@ void hlenCommand(client *c) { } void hstrlenCommand(client *c) { - robj *o; + kvobj *o; unsigned char *vstr = NULL; unsigned int vlen = UINT_MAX; long long vll = LLONG_MAX; @@ -3027,7 +2965,7 @@ static void addHashIteratorCursorToReply(client *c, hashTypeIterator *hi, int wh } void genericHgetallCommand(client *c, int flags) { - robj *o; + kvobj *o; hashTypeIterator *hi; int length, count = 0; @@ -3082,7 +3020,7 @@ void hgetallCommand(client *c) { } void hexistsCommand(client *c) { - robj *o; + kvobj *o; if ((o = lookupKeyReadOrReply(c,c->argv[1],shared.czero)) == NULL || checkType(c,o,OBJ_HASH)) return; @@ -3091,7 +3029,7 @@ void hexistsCommand(client *c) { } void hscanCommand(client *c) { - robj *o; + kvobj *o; unsigned long long cursor; if (parseScanCursorOrReply(c,c->argv[2],&cursor) == C_ERR) return; @@ -3131,7 +3069,7 @@ static void hrandfieldReplyWithListpack(client *c, unsigned int count, listpackE void hrandfieldWithCountCommand(client *c, long l, int withvalues) { unsigned long count, size; int uniq = 1; - robj *hash; + kvobj *hash; if ((hash = lookupKeyReadOrReply(c,c->argv[1],shared.emptyarray)) == NULL || checkType(c,hash,OBJ_HASH)) return; @@ -3376,7 +3314,7 @@ void hrandfieldWithCountCommand(client *c, long l, int withvalues) { void hrandfieldCommand(client *c) { long l; int withvalues = 0; - robj *hash; + kvobj *hash; CommonEntry ele; if (c->argc >= 3) { @@ -3524,13 +3462,13 @@ static void propagateHashFieldDeletion(redisDb *db, sds key, char *field, size_t static ExpireAction onFieldExpire(eItem item, void *ctx) { OnFieldExpireCtx *expCtx = ctx; hfield hf = item; - dict *d = expCtx->hashObj->ptr; - dictExpireMetadata *dictExpireMeta = (dictExpireMetadata *) dictMetadata(d); - propagateHashFieldDeletion(expCtx->db, dictExpireMeta->key, hf, hfieldlen(hf)); + kvobj *kv = expCtx->hashObj; + sds key = kvobjGetKey(kv); + propagateHashFieldDeletion(expCtx->db, key, hf, hfieldlen(hf)); /* update keysizes */ unsigned long l = hashTypeLength(expCtx->hashObj, 0); - updateKeysizesHist(expCtx->db, getKeySlot(dictExpireMeta->key), OBJ_HASH, l, l - 1); + updateKeysizesHist(expCtx->db, getKeySlot(key), OBJ_HASH, l, l - 1); serverAssert(hashTypeDelete(expCtx->hashObj, hf, 0) == 1); server.stat_expired_subkeys++; @@ -3556,7 +3494,7 @@ static ExpireMeta *hashGetExpireMeta(const eItem hash) { /* HTTL key */ static void httlGenericCommand(client *c, const char *cmd, long long basetime, int unit) { UNUSED(cmd); - robj *hashObj; + kvobj *hashObj; long numFields = 0, numFieldsAt = 3; /* Read the hash object */ @@ -3715,10 +3653,10 @@ static void hexpireGenericCommand(client *c, long long basetime, int unit) { long long expire; /* unix time in msec */ int fieldAt, fieldsNotSet = 0, expireSetCond = 0, updated = 0, deleted = 0; int64_t oldlen, newlen; - robj *hashObj, *keyArg = c->argv[1], *expireArg = c->argv[2]; + robj *keyArg = c->argv[1], *expireArg = c->argv[2]; /* Read the hash object */ - hashObj = lookupKeyWrite(c->db, keyArg); + kvobj *hashObj = lookupKeyWrite(c->db, keyArg); if (checkType(c, hashObj, OBJ_HASH)) return; @@ -3880,12 +3818,11 @@ void hpexpiretimeCommand(client *c) { /* HPERSIST key FIELDS numfields */ void hpersistCommand(client *c) { - robj *hashObj; long numFields = 0, numFieldsAt = 3; int changed = 0; /* Used to determine whether to send a notification. */ /* Read the hash object */ - hashObj = lookupKeyWrite(c->db, c->argv[1]); + kvobj *hashObj = lookupKeyWrite(c->db, c->argv[1]); if (checkType(c, hashObj, OBJ_HASH)) return; diff --git a/src/t_list.c b/src/t_list.c index b2ba25b31..e08611881 100644 --- a/src/t_list.c +++ b/src/t_list.c @@ -465,9 +465,10 @@ void listTypeDelRange(robj *subject, long start, long count) { * 'xx': push if key exists. */ void pushGenericCommand(client *c, int where, int xx) { unsigned long llen; + dictEntryLink link; int j; - robj *lobj = lookupKeyWrite(c->db, c->argv[1]); + kvobj *lobj = lookupKeyWriteWithLink(c->db, c->argv[1], &link); if (checkType(c,lobj,OBJ_LIST)) return; if (!lobj) { if (xx) { @@ -476,7 +477,7 @@ void pushGenericCommand(client *c, int where, int xx) { } lobj = createListListpackObject(); - dbAdd(c->db,c->argv[1],lobj); + dbAddByLink(c->db, c->argv[1], &lobj, &link); } listTypeTryConversionAppend(lobj,c->argv,2,c->argc-1,NULL,NULL); @@ -517,7 +518,7 @@ void rpushxCommand(client *c) { /* LINSERT (BEFORE|AFTER) */ void linsertCommand(client *c) { int where; - robj *subject; + kvobj *subject; listTypeIterator *iter; listTypeEntry entry; int inserted = 0; @@ -571,14 +572,14 @@ void linsertCommand(client *c) { /* LLEN */ void llenCommand(client *c) { - robj *o = lookupKeyReadOrReply(c,c->argv[1],shared.czero); - if (o == NULL || checkType(c,o,OBJ_LIST)) return; - addReplyLongLong(c,listTypeLength(o)); + kvobj *kv = lookupKeyReadOrReply(c,c->argv[1],shared.czero); + if (kv == NULL || checkType(c,kv,OBJ_LIST)) return; + addReplyLongLong(c,listTypeLength(kv)); } /* LINDEX */ void lindexCommand(client *c) { - robj *o = lookupKeyReadOrReply(c,c->argv[1],shared.null[c->resp]); + kvobj *o = lookupKeyReadOrReply(c,c->argv[1],shared.null[c->resp]); if (o == NULL || checkType(c,o,OBJ_LIST)) return; long index; @@ -607,7 +608,7 @@ void lindexCommand(client *c) { /* LSET */ void lsetCommand(client *c) { - robj *o = lookupKeyWriteOrReply(c,c->argv[1],shared.nokeyerr); + kvobj *o = lookupKeyWriteOrReply(c, c->argv[1], shared.nokeyerr); if (o == NULL || checkType(c,o,OBJ_LIST)) return; long index; robj *value = c->argv[3]; @@ -775,7 +776,7 @@ void popGenericCommand(client *c, int where) { return; } - robj *o = lookupKeyWriteOrReply(c, c->argv[1], hascount ? shared.nullarray[c->resp]: shared.null[c->resp]); + kvobj *o = lookupKeyWriteOrReply(c, c->argv[1], hascount ? shared.nullarray[c->resp] : shared.null[c->resp]); if (o == NULL || checkType(c, o, OBJ_LIST)) return; @@ -861,7 +862,7 @@ void rpopCommand(client *c) { /* LRANGE */ void lrangeCommand(client *c) { - robj *o; + kvobj *o; long start, end; if ((getLongFromObjectOrReply(c, c->argv[2], &start, NULL) != C_OK) || @@ -875,7 +876,7 @@ void lrangeCommand(client *c) { /* LTRIM */ void ltrimCommand(client *c) { - robj *o; + kvobj *o; long start, end, llen, ltrim, rtrim, llenNew; if ((getLongFromObjectOrReply(c, c->argv[2], &start, NULL) != C_OK) || @@ -945,7 +946,7 @@ void ltrimCommand(client *c) { * The returned elements indexes are always referring to what LINDEX * would return. So first element from head is 0, and so forth. */ void lposCommand(client *c) { - robj *o, *ele; + robj *ele; ele = c->argv[2]; int direction = LIST_TAIL; long rank = 1, count = -1, maxlen = 0; /* Count -1: option not given. */ @@ -989,7 +990,8 @@ void lposCommand(client *c) { /* We return NULL or an empty array if there is no such key (or * if we find no matches, depending on the presence of the COUNT option. */ - if ((o = lookupKeyRead(c->db,c->argv[1])) == NULL) { + kvobj *o = lookupKeyRead(c->db,c->argv[1]); + if (o == NULL) { if (count != -1) addReply(c,shared.emptyarray); else @@ -1042,7 +1044,7 @@ void lposCommand(client *c) { /* LREM */ void lremCommand(client *c) { - robj *subject, *obj; + robj *obj; obj = c->argv[3]; long toremove; long removed = 0; @@ -1050,7 +1052,7 @@ void lremCommand(client *c) { if (getRangeLongFromObjectOrReply(c, c->argv[2], -LONG_MAX, LONG_MAX, &toremove, NULL) != C_OK) return; - subject = lookupKeyWriteOrReply(c,c->argv[1],shared.czero); + kvobj *subject = lookupKeyWriteOrReply(c, c->argv[1], shared.czero); if (subject == NULL || checkType(c,subject,OBJ_LIST)) return; listTypeIterator *li; @@ -1095,7 +1097,7 @@ void lmoveHandlePush(client *c, robj *dstkey, robj *dstobj, robj *value, /* Create the list if the key does not exist */ if (!dstobj) { dstobj = createListListpackObject(); - dbAdd(c->db,dstkey,dstobj); + dbAdd(c->db, dstkey, &dstobj); } listTypeTryConversionAppend(dstobj,&value,0,0,NULL,NULL); listTypePush(dstobj,value,where); @@ -1131,32 +1133,31 @@ robj *getStringObjectFromListPosition(int position) { } void lmoveGenericCommand(client *c, int wherefrom, int whereto) { - robj *sobj, *value; - if ((sobj = lookupKeyWriteOrReply(c,c->argv[1],shared.null[c->resp])) - == NULL || checkType(c,sobj,OBJ_LIST)) return; + kvobj *kvsrc = lookupKeyWriteOrReply(c,c->argv[1],shared.null[c->resp]); + if (kvsrc == NULL || checkType(c,kvsrc,OBJ_LIST)) return; - if (listTypeLength(sobj) == 0) { + if (listTypeLength(kvsrc) == 0) { /* This may only happen after loading very old RDB files. Recent * versions of Redis delete keys of empty lists. */ addReplyNull(c); } else { - robj *dobj, *skey = c->argv[1]; + robj *kvdst, *skey = c->argv[1]; int64_t oldlen = 0, newlen = 1; /* init lengths assuming new dst object */ - - if ((dobj = lookupKeyWrite(c->db,c->argv[2])) != NULL) { - if (checkType(c,dobj,OBJ_LIST)) return; + + if ((kvdst = lookupKeyWrite(c->db,c->argv[2])) != NULL) { + if (checkType(c,kvdst,OBJ_LIST)) return; /* dst object exists */ - oldlen = (int64_t) listTypeLength(dobj); + oldlen = (int64_t) listTypeLength(kvdst); newlen = oldlen + 1; } - - value = listTypePop(sobj,wherefrom); + + robj *value = listTypePop(kvsrc, wherefrom); serverAssert(value); /* assertion for valgrind (avoid NPD) */ - lmoveHandlePush(c,c->argv[2],dobj,value,whereto); + lmoveHandlePush(c, c->argv[2], kvdst, value, whereto); /* Update dst obj cardinality in KEYSIZES */ updateKeysizesHist(c->db, getKeySlot(c->argv[2]->ptr), OBJ_LIST, oldlen, newlen); /* Update src obj cardinality in KEYSIZES by listElementsRemoved() */ - listElementsRemoved(c,skey,wherefrom,sobj,1,1,NULL); + listElementsRemoved(c, skey, wherefrom, kvsrc, 1, 1, NULL); /* listTypePop returns an object with its refcount incremented */ decrRefCount(value); diff --git a/src/t_set.c b/src/t_set.c index b859e15f2..0272a5a23 100644 --- a/src/t_set.c +++ b/src/t_set.c @@ -127,16 +127,17 @@ int setTypeAddAux(robj *set, char *str, size_t len, int64_t llval, int str_is_sd /* Avoid duping the string if it is an sds string. */ sds sdsval = str_is_sds ? (sds)str : sdsnewlen(str, len); dict *ht = set->ptr; - void *position = dictFindPositionForInsert(ht, sdsval, NULL); - if (position) { + dictEntryLink bucket, link = dictFindLink(ht, sdsval, &bucket); + if (link == NULL) { /* Key doesn't already exist in the set. Add it but dup the key. */ if (sdsval == str) sdsval = sdsdup(sdsval); - dictInsertAtPosition(ht, sdsval, position); + dictSetKeyAtLink(ht, sdsval, &bucket, 1); + return 1; } else if (sdsval != str) { /* String is already a member. Free our temporary sds copy. */ sdsfree(sdsval); + return 0; } - return (position != NULL); } else if (set->encoding == OBJ_ENCODING_LISTPACK) { unsigned char *lp = set->ptr; unsigned char *p = lpFirst(lp); @@ -587,15 +588,16 @@ robj *setTypeDup(robj *o) { } void saddCommand(client *c) { - robj *set; + kvobj *set; int j, added = 0; + dictEntryLink link; - set = lookupKeyWrite(c->db,c->argv[1]); + set = lookupKeyWriteWithLink(c->db,c->argv[1], &link); if (checkType(c,set,OBJ_SET)) return; if (set == NULL) { - set = setTypeCreate(c->argv[2]->ptr, c->argc - 2); - dbAdd(c->db,c->argv[1],set); + robj *o = setTypeCreate(c->argv[2]->ptr, c->argc - 2); + set = dbAddByLink(c->db, c->argv[1], &o, &link); } else { setTypeMaybeConvert(set, c->argc - 2); } @@ -614,11 +616,11 @@ void saddCommand(client *c) { } void sremCommand(client *c) { - robj *set; int j, deleted = 0, keyremoved = 0; - if ((set = lookupKeyWriteOrReply(c,c->argv[1],shared.czero)) == NULL || - checkType(c,set,OBJ_SET)) return; + kvobj *set = lookupKeyWriteOrReply(c, c->argv[1], shared.czero); + if (set == NULL || checkType(c, set, OBJ_SET)) + return; unsigned long oldSize = setTypeSize(set); @@ -633,8 +635,8 @@ void sremCommand(client *c) { } } if (deleted) { - int64_t newSize = oldSize - deleted; - + int64_t newSize = oldSize - deleted; + signalModifiedKey(c,c->db,c->argv[1]); notifyKeyspaceEvent(NOTIFY_SET,"srem",c->argv[1],c->db->id); if (keyremoved) { @@ -693,7 +695,7 @@ void smoveCommand(client *c) { /* Create the destination set when it doesn't exist */ if (!dstset) { dstset = setTypeCreate(ele->ptr, 1); - dbAdd(c->db,c->argv[2],dstset); + dbAdd(c->db, c->argv[2], &dstset); } signalModifiedKey(c,c->db,c->argv[1]); @@ -711,7 +713,7 @@ void smoveCommand(client *c) { } void sismemberCommand(client *c) { - robj *set; + kvobj *set; if ((set = lookupKeyReadOrReply(c,c->argv[1],shared.czero)) == NULL || checkType(c,set,OBJ_SET)) return; @@ -723,17 +725,14 @@ void sismemberCommand(client *c) { } void smismemberCommand(client *c) { - robj *set; - int j; - /* Don't abort when the key cannot be found. Non-existing keys are empty * sets, where SMISMEMBER should respond with a series of zeros. */ - set = lookupKeyRead(c->db,c->argv[1]); + kvobj *set = lookupKeyRead(c->db, c->argv[1]); if (set && checkType(c,set,OBJ_SET)) return; addReplyArrayLen(c,c->argc - 2); - for (j = 2; j < c->argc; j++) { + for (int j = 2; j < c->argc; j++) { if (set && setTypeIsMember(set,c->argv[j]->ptr)) addReply(c,shared.cone); else @@ -742,12 +741,12 @@ void smismemberCommand(client *c) { } void scardCommand(client *c) { - robj *o; + kvobj *kv; - if ((o = lookupKeyReadOrReply(c,c->argv[1],shared.czero)) == NULL || - checkType(c,o,OBJ_SET)) return; + if ((kv = lookupKeyReadOrReply(c,c->argv[1],shared.czero)) == NULL || + checkType(c,kv,OBJ_SET)) return; - addReplyLongLong(c,setTypeSize(o)); + addReplyLongLong(c,setTypeSize(kv)); } /* Handle the "SPOP key " variant. The normal version of the @@ -761,16 +760,15 @@ void scardCommand(client *c) { void spopWithCountCommand(client *c) { long l; unsigned long count, size, toRemove; - robj *set; /* Get the count argument */ if (getPositiveLongFromObjectOrReply(c,c->argv[2],&l,NULL) != C_OK) return; count = (unsigned long) l; /* Make sure a key with the name inputted exists, and that it's type is - * indeed a set. Otherwise, return nil */ - if ((set = lookupKeyWriteOrReply(c,c->argv[1],shared.emptyset[c->resp])) - == NULL || checkType(c,set,OBJ_SET)) return; + * indeed a kv. Otherwise, return nil */ + robj *set = lookupKeyWriteOrReply(c, c->argv[1], shared.emptyset[c->resp]); + if (set == NULL || checkType(c, set, OBJ_SET)) return; /* If count is zero, serve an empty set ASAP to avoid special * cases later. */ @@ -785,7 +783,7 @@ void spopWithCountCommand(client *c) { /* Generate an SPOP keyspace notification */ notifyKeyspaceEvent(NOTIFY_SET,"spop",c->argv[1],c->db->id); server.dirty += toRemove; - + /* CASE 1: * The number of requested elements is greater than or equal to * the number of elements inside the set: simply return the whole set. */ @@ -947,7 +945,7 @@ void spopWithCountCommand(client *c) { setTypeReleaseIterator(si); /* Assign the new set as the key value (Also update KEYSIZES histogram) */ - dbReplaceValue(c->db,c->argv[1],newset); + dbReplaceValue(c->db, c->argv[1], &newset); } /* Replicate/AOF the remaining elements as an SREM operation */ @@ -970,7 +968,7 @@ void spopWithCountCommand(client *c) { void spopCommand(client *c) { unsigned long size; - robj *set, *ele; + robj *ele; if (c->argc == 3) { spopWithCountCommand(c); @@ -981,15 +979,15 @@ void spopCommand(client *c) { } /* Make sure a key with the name inputted exists, and that it's type is - * indeed a set */ - if ((set = lookupKeyWriteOrReply(c,c->argv[1],shared.null[c->resp])) - == NULL || checkType(c,set,OBJ_SET)) return; + * indeed a kv */ + kvobj *kv = lookupKeyWriteOrReply(c, c->argv[1], shared.null[c->resp]); + if (kv == NULL || checkType(c, kv, OBJ_SET)) return; - size = setTypeSize(set); + size = setTypeSize(kv); updateKeysizesHist(c->db, getKeySlot(c->argv[1]->ptr), OBJ_SET, size, size-1); - /* Pop a random element from the set */ - ele = setTypePopRandom(set); + /* Pop a random element from the kv */ + ele = setTypePopRandom(kv); notifyKeyspaceEvent(NOTIFY_SET,"spop",c->argv[1],c->db->id); @@ -1000,8 +998,8 @@ void spopCommand(client *c) { addReplyBulk(c, ele); decrRefCount(ele); - /* Delete the set if it's empty */ - if (setTypeSize(set) == 0) { + /* Delete the kv if it's empty */ + if (setTypeSize(kv) == 0) { dbDelete(c->db,c->argv[1]); notifyKeyspaceEvent(NOTIFY_GENERIC,"del",c->argv[1],c->db->id); } @@ -1028,7 +1026,7 @@ void srandmemberWithCountCommand(client *c) { long l; unsigned long count, size; int uniq = 1; - robj *set; + kvobj *set; char *str; size_t len = 0; int64_t llele = 0; @@ -1230,7 +1228,7 @@ void srandmemberWithCountCommand(client *c) { /* SRANDMEMBER [] */ void srandmemberCommand(client *c) { - robj *set; + kvobj *set; char *str; size_t len = 0; int64_t llele = 0; @@ -1284,7 +1282,7 @@ int qsortCompareSetsByRevCardinality(const void *s1, const void *s2) { void sinterGenericCommand(client *c, robj **setkeys, unsigned long setnum, robj *dstkey, int cardinality_only, unsigned long limit) { - robj **sets = zmalloc(sizeof(robj*)*setnum); + kvobj **sets = zmalloc(sizeof(robj*)*setnum); setTypeIterator *si; robj *dstset = NULL; char *str; @@ -1295,18 +1293,18 @@ void sinterGenericCommand(client *c, robj **setkeys, int encoding, empty = 0; for (j = 0; j < setnum; j++) { - robj *setobj = lookupKeyRead(c->db, setkeys[j]); - if (!setobj) { + kvobj *kv = lookupKeyRead(c->db, setkeys[j]); + if (!kv) { /* A NULL is considered an empty set */ empty += 1; sets[j] = NULL; continue; } - if (checkType(c,setobj,OBJ_SET)) { + if (checkType(c, kv, OBJ_SET)) { zfree(sets); return; } - sets[j] = setobj; + sets[j] = kv; } /* Set intersection with an empty set always results in an empty set. @@ -1424,7 +1422,7 @@ void sinterGenericCommand(client *c, robj **setkeys, * frequent reallocs. Therefore, we shrink it now. */ dstset->ptr = lpShrinkToFit(dstset->ptr); } - setKey(c,c->db,dstkey,dstset,0); + setKey(c, c->db, dstkey, &dstset, 0); addReplyLongLong(c,setTypeSize(dstset)); notifyKeyspaceEvent(NOTIFY_SET,"sinterstore", dstkey,c->db->id); @@ -1436,8 +1434,8 @@ void sinterGenericCommand(client *c, robj **setkeys, signalModifiedKey(c,c->db,dstkey); notifyKeyspaceEvent(NOTIFY_GENERIC,"del",dstkey,c->db->id); } + decrRefCount(dstset); } - decrRefCount(dstset); } else { setDeferredSetLen(c,replylen,cardinality); } @@ -1455,7 +1453,7 @@ void smembersCommand(client *c) { char *str; size_t len = 0; int64_t intobj = 0; - robj *setobj = lookupKeyRead(c->db, c->argv[1]); + kvobj *setobj = lookupKeyRead(c->db, c->argv[1]); if (checkType(c,setobj,OBJ_SET)) return; if (!setobj) { addReply(c, shared.emptyset[c->resp]); @@ -1531,7 +1529,7 @@ void sunionDiffGenericCommand(client *c, robj **setkeys, int setnum, int sameset = 0; for (j = 0; j < setnum; j++) { - robj *setobj = lookupKeyRead(c->db, setkeys[j]); + kvobj *setobj = lookupKeyRead(c->db, setkeys[j]); if (!setobj) { sets[j] = NULL; continue; @@ -1689,7 +1687,7 @@ void sunionDiffGenericCommand(client *c, robj **setkeys, int setnum, /* If we have a target key where to store the resulting set * create this key with the result set inside */ if (setTypeSize(dstset) > 0) { - setKey(c,c->db,dstkey,dstset,0); + setKey(c, c->db, dstkey, &dstset, 0); addReplyLongLong(c,setTypeSize(dstset)); notifyKeyspaceEvent(NOTIFY_SET, op == SET_OP_UNION ? "sunionstore" : "sdiffstore", @@ -1702,8 +1700,8 @@ void sunionDiffGenericCommand(client *c, robj **setkeys, int setnum, signalModifiedKey(c,c->db,dstkey); notifyKeyspaceEvent(NOTIFY_GENERIC,"del",dstkey,c->db->id); } + decrRefCount(dstset); } - decrRefCount(dstset); } zfree(sets); } @@ -1729,7 +1727,7 @@ void sdiffstoreCommand(client *c) { } void sscanCommand(client *c) { - robj *set; + kvobj *set; unsigned long long cursor; if (parseScanCursorOrReply(c,c->argv[2],&cursor) == C_ERR) return; diff --git a/src/t_stream.c b/src/t_stream.c index a242b02ca..83ba6ba38 100644 --- a/src/t_stream.c +++ b/src/t_stream.c @@ -1855,17 +1855,18 @@ size_t streamReplyWithRangeFromConsumerPEL(client *c, stream *s, streamID *start /* Look the stream at 'key' and return the corresponding stream object. * The function creates a key setting it to an empty stream if needed. */ -robj *streamTypeLookupWriteOrCreate(client *c, robj *key, int no_create) { - robj *o = lookupKeyWrite(c->db,key); - if (checkType(c,o,OBJ_STREAM)) return NULL; - if (o == NULL) { - if (no_create) { - addReplyNull(c); - return NULL; - } - o = createStreamObject(); - dbAdd(c->db,key,o); +kvobj *streamTypeLookupWriteOrCreate(client *c, robj *key, int no_create) { + dictEntryLink link; + kvobj *kv = lookupKeyWriteWithLink(c->db,key, &link); + if (checkType(c, kv, OBJ_STREAM)) return NULL; + if (kv != NULL) return kv; + + if (no_create) { + addReplyNull(c); + return NULL; } + robj *o = createStreamObject(); + dbAddByLink(c->db, key, &o, &link); return o; } @@ -2023,10 +2024,10 @@ void xaddCommand(client *c) { } /* Lookup the stream at key. */ - robj *o; + kvobj *kv; stream *s; - if ((o = streamTypeLookupWriteOrCreate(c,c->argv[1],parsed_args.no_mkstream)) == NULL) return; - s = o->ptr; + if ((kv = streamTypeLookupWriteOrCreate(c,c->argv[1],parsed_args.no_mkstream)) == NULL) return; + s = kv->ptr; /* Return ASAP if the stream has reached the last possible ID */ if (s->last_id.ms == UINT64_MAX && s->last_id.seq == UINT64_MAX) { @@ -2096,7 +2097,7 @@ void xaddCommand(client *c) { * will match anything from 1-1 and 1-UINT64_MAX. */ void xrangeGenericCommand(client *c, int rev) { - robj *o; + kvobj *kv; stream *s; streamID startid, endid; long long count = -1; @@ -2135,10 +2136,10 @@ void xrangeGenericCommand(client *c, int rev) { } /* Return the specified range to the user. */ - if ((o = lookupKeyReadOrReply(c,c->argv[1],shared.emptyarray)) == NULL || - checkType(c,o,OBJ_STREAM)) return; + if ((kv = lookupKeyReadOrReply(c, c->argv[1], shared.emptyarray)) == NULL || + checkType(c, kv, OBJ_STREAM)) return; - s = o->ptr; + s = kv->ptr; if (count == 0) { addReplyNullArray(c); @@ -2160,10 +2161,10 @@ void xrevrangeCommand(client *c) { /* XLEN key*/ void xlenCommand(client *c) { - robj *o; - if ((o = lookupKeyReadOrReply(c,c->argv[1],shared.czero)) == NULL - || checkType(c,o,OBJ_STREAM)) return; - stream *s = o->ptr; + kvobj *kv; + if ((kv = lookupKeyReadOrReply(c, c->argv[1], shared.czero)) == NULL + || checkType(c, kv, OBJ_STREAM)) return; + stream *s = kv->ptr; addReplyLongLong(c,s->length); } @@ -2259,7 +2260,7 @@ void xreadCommand(client *c) { * starting from now. */ int id_idx = i - streams_arg - streams_count; robj *key = c->argv[i-streams_count]; - robj *o = lookupKeyRead(c->db,key); + kvobj *o = lookupKeyRead(c->db, key); if (checkType(c,o,OBJ_STREAM)) goto cleanup; streamCG *group = NULL; @@ -2338,7 +2339,7 @@ void xreadCommand(client *c) { size_t arraylen = 0; void *arraylen_ptr = NULL; for (int i = 0; i < streams_count; i++) { - robj *o = lookupKeyRead(c->db,c->argv[streams_arg+i]); + kvobj *o = lookupKeyRead(c->db, c->argv[streams_arg + i]); if (o == NULL) continue; stream *s = o->ptr; streamID *gt = ids+i; /* ID must be greater than this. */ @@ -2706,7 +2707,7 @@ NULL if (s == NULL) { serverAssert(mkstream); o = createStreamObject(); - dbAdd(c->db,c->argv[2],o); + dbAdd(c->db, c->argv[2], &o); s = o->ptr; signalModifiedKey(c,c->db,c->argv[2]); } @@ -2804,9 +2805,9 @@ void xsetidCommand(client *c) { } } - robj *o = lookupKeyWriteOrReply(c,c->argv[1],shared.nokeyerr); - if (o == NULL || checkType(c,o,OBJ_STREAM)) return; - stream *s = o->ptr; + kvobj *kv = lookupKeyWriteOrReply(c, c->argv[1], shared.nokeyerr); + if (kv == NULL || checkType(c, kv, OBJ_STREAM)) return; + stream *s = kv->ptr; if (streamCompareID(&id,&s->max_deleted_entry_id) < 0) { addReplyError(c,"The ID specified in XSETID is smaller than current max_deleted_entry_id"); @@ -2852,14 +2853,14 @@ void xsetidCommand(client *c) { */ void xackCommand(client *c) { streamCG *group = NULL; - robj *o = lookupKeyRead(c->db,c->argv[1]); - if (o) { - if (checkType(c,o,OBJ_STREAM)) return; /* Type error. */ - group = streamLookupCG(o->ptr,c->argv[2]->ptr); + kvobj *kv = lookupKeyRead(c->db, c->argv[1]); + if (kv) { + if (checkType(c, kv, OBJ_STREAM)) return; /* Type error. */ + group = streamLookupCG(kv->ptr, c->argv[2]->ptr); } /* No key or group? Nothing to ack. */ - if (o == NULL || group == NULL) { + if (kv == NULL || group == NULL) { addReply(c,shared.czero); return; } @@ -2969,12 +2970,12 @@ void xpendingCommand(client *c) { } /* Lookup the key and the group inside the stream. */ - robj *o = lookupKeyRead(c->db,c->argv[1]); + kvobj *kv = lookupKeyRead(c->db, c->argv[1]); streamCG *group; - if (checkType(c,o,OBJ_STREAM)) return; - if (o == NULL || - (group = streamLookupCG(o->ptr,groupname->ptr)) == NULL) + if (checkType(c, kv, OBJ_STREAM)) return; + if (kv == NULL || + (group = streamLookupCG(kv->ptr, groupname->ptr)) == NULL) { addReplyErrorFormat(c, "-NOGROUP No such key '%s' or consumer " "group '%s'", @@ -3152,7 +3153,7 @@ void xpendingCommand(client *c) { * what messages it is now in charge of. */ void xclaimCommand(client *c) { streamCG *group = NULL; - robj *o = lookupKeyRead(c->db,c->argv[1]); + kvobj *o = lookupKeyRead(c->db,c->argv[1]); long long minidle; /* Minimum idle time argument. */ long long retrycount = -1; /* -1 means RETRYCOUNT option not given. */ mstime_t deliverytime = -1; /* -1 means IDLE/TIME options not given. */ @@ -3373,7 +3374,7 @@ cleanup: * what messages it is now in charge of. */ void xautoclaimCommand(client *c) { streamCG *group = NULL; - robj *o = lookupKeyRead(c->db,c->argv[1]); + kvobj *o = lookupKeyRead(c->db,c->argv[1]); long long minidle; /* Minimum idle time argument, in milliseconds. */ long count = 100; /* Maximum entries to claim. */ const unsigned attempts_factor = 10; @@ -3551,11 +3552,9 @@ void xautoclaimCommand(client *c) { * of items actually deleted, that may be different from the number * of IDs passed in case certain IDs do not exist. */ void xdelCommand(client *c) { - robj *o; - - if ((o = lookupKeyWriteOrReply(c,c->argv[1],shared.czero)) == NULL - || checkType(c,o,OBJ_STREAM)) return; - stream *s = o->ptr; + kvobj *kv = lookupKeyWriteOrReply(c, c->argv[1], shared.czero); + if (kv == NULL || checkType(c, kv, OBJ_STREAM)) return; + stream *s = kv->ptr; /* We need to sanity check the IDs passed to start. Even if not * a big issue, it is not great that the command is only partially @@ -3633,8 +3632,6 @@ cleanup: * Has meaning only if `~` was provided. */ void xtrimCommand(client *c) { - robj *o; - /* Argument parsing. */ streamAddTrimArgs parsed_args; if (streamParseAddOrTrimArgsOrReply(c, &parsed_args, 0) < 0) @@ -3642,9 +3639,9 @@ void xtrimCommand(client *c) { /* If the key does not exist, we are ok returning zero, that is, the * number of elements removed from the stream. */ - if ((o = lookupKeyWriteOrReply(c,c->argv[1],shared.czero)) == NULL - || checkType(c,o,OBJ_STREAM)) return; - stream *s = o->ptr; + kvobj *kv = lookupKeyWriteOrReply(c, c->argv[1], shared.czero); + if (kv == NULL || checkType(c, kv, OBJ_STREAM)) return; + stream *s = kv->ptr; /* Perform the trimming. */ int64_t deleted = streamTrim(s, &parsed_args); @@ -3904,9 +3901,9 @@ NULL key = c->argv[2]; /* Lookup the key now, this is common for all the subcommands but HELP. */ - robj *o = lookupKeyReadOrReply(c,key,shared.nokeyerr); - if (o == NULL || checkType(c,o,OBJ_STREAM)) return; - s = o->ptr; + kvobj *kv = lookupKeyReadOrReply(c, key, shared.nokeyerr); + if (kv == NULL || checkType(c, kv, OBJ_STREAM)) return; + s = kv->ptr; /* Dispatch the different subcommands. */ if (!strcasecmp(opt,"CONSUMERS") && c->argc == 4) { diff --git a/src/t_string.c b/src/t_string.c index 18396b834..7f747e18b 100644 --- a/src/t_string.c +++ b/src/t_string.c @@ -61,7 +61,18 @@ static int checkStringLength(client *c, long long size, long long append) { /* Forward declaration */ static int getExpireMillisecondsOrReply(client *c, robj *expire, int flags, int unit, long long *milliseconds); -void setGenericCommand(client *c, int flags, robj *key, robj *val, robj *expire, int unit, robj *ok_reply, robj *abort_reply) { +/* Generic SET command family (SET, SETEX, PSETEX, SETNX) + * + * Arguments: + * valref: A pointer to the robj to be set. This argument may be updated by the function. + * The object is expected to have a refcount of 1, allowing its ownership to be + * transferred directly to the database to avoid making a copy. If needed, the + * function will replace *valref with a new allocation and increment its refcount + * so that both the database and the caller maintain valid references. + */ +void setGenericCommand(client *c, int flags, robj *key, robj **valref, robj *expire, + int unit, robj *ok_reply, robj *abort_reply) +{ long long milliseconds = 0; /* initialized to avoid any harmless warning */ int found = 0; int setkey_flags = 0; @@ -74,8 +85,8 @@ void setGenericCommand(client *c, int flags, robj *key, robj *val, robj *expire, if (getGenericCommand(c) == C_ERR) return; } - dictEntry *de = NULL; - found = (lookupKeyWriteWithDictEntry(c->db,key,&de) != NULL); + dictEntryLink link = NULL; + found = (lookupKeyWriteWithLink(c->db,key,&link) != NULL); if ((flags & OBJ_SET_NX && found) || (flags & OBJ_SET_XX && !found)) @@ -90,17 +101,24 @@ void setGenericCommand(client *c, int flags, robj *key, robj *val, robj *expire, setkey_flags |= ((flags & OBJ_KEEPTTL) || expire) ? SETKEY_KEEPTTL : 0; setkey_flags |= found ? SETKEY_ALREADY_EXIST : SETKEY_DOESNT_EXIST; - setKeyWithDictEntry(c,c->db,key,val,setkey_flags,de); + setKeyByLink(c, c->db, key, valref, setkey_flags, &link); + /* If there's an expiration, setExpireByLink may reallocate the object. + * We must update valref to reflect the new object if that happens. */ + if (expire) *valref = setExpireByLink(c, c->db, key->ptr, milliseconds, link); + /* The client still holds a reference to the original object via c->argv[i], + * and will call decrRefCount() at the end of call(). We increment the refcount + * from 1 to 2 to ensure both DB and client have valid references. */ + incrRefCount(*valref); /* 1->2 */ + server.dirty++; notifyKeyspaceEvent(NOTIFY_STRING,"set",key,c->db->id); if (expire) { - setExpireWithDictEntry(c,c->db,key,milliseconds,de); /* Propagate as SET Key Value PXAT millisecond-timestamp if there is * EX/PX/EXAT flag. */ if (!(flags & OBJ_PXAT)) { robj *milliseconds_obj = createStringObjectFromLongLong(milliseconds); - rewriteClientCommandVector(c, 5, shared.set, key, val, shared.pxat, milliseconds_obj); + rewriteClientCommandVector(c, 5, shared.set, key, *valref, shared.pxat, milliseconds_obj); decrRefCount(milliseconds_obj); } notifyKeyspaceEvent(NOTIFY_GENERIC,"expire",key,c->db->id); @@ -283,28 +301,28 @@ void setCommand(client *c) { } c->argv[2] = tryObjectEncoding(c->argv[2]); - setGenericCommand(c,flags,c->argv[1],c->argv[2],expire,unit,NULL,NULL); + setGenericCommand(c,flags,c->argv[1],&(c->argv[2]),expire,unit,NULL,NULL); } void setnxCommand(client *c) { c->argv[2] = tryObjectEncoding(c->argv[2]); - setGenericCommand(c,OBJ_SET_NX,c->argv[1],c->argv[2],NULL,0,shared.cone,shared.czero); + setGenericCommand(c, OBJ_SET_NX, c->argv[1], &(c->argv[2]), NULL, 0, shared.cone, shared.czero); } void setexCommand(client *c) { c->argv[3] = tryObjectEncoding(c->argv[3]); - setGenericCommand(c,OBJ_EX,c->argv[1],c->argv[3],c->argv[2],UNIT_SECONDS,NULL,NULL); + setGenericCommand(c, OBJ_EX, c->argv[1], &(c->argv[3]), c->argv[2], UNIT_SECONDS, NULL, NULL); } void psetexCommand(client *c) { c->argv[3] = tryObjectEncoding(c->argv[3]); - setGenericCommand(c,OBJ_PX,c->argv[1],c->argv[3],c->argv[2],UNIT_MILLISECONDS,NULL,NULL); + setGenericCommand(c, OBJ_PX, c->argv[1], &(c->argv[3]), c->argv[2], UNIT_MILLISECONDS, NULL, NULL); } int getGenericCommand(client *c) { - robj *o; + kvobj *o; - if ((o = lookupKeyReadOrReply(c,c->argv[1],shared.null[c->resp])) == NULL) + if ((o = lookupKeyReadOrReply(c, c->argv[1], shared.null[c->resp])) == NULL) return C_OK; if (checkType(c,o,OBJ_STRING)) { @@ -348,7 +366,7 @@ void getexCommand(client *c) { return; } - robj *o; + kvobj *o; if ((o = lookupKeyReadOrReply(c,c->argv[1],shared.null[c->resp])) == NULL) return; @@ -412,7 +430,8 @@ void getdelCommand(client *c) { void getsetCommand(client *c) { if (getGenericCommand(c) == C_ERR) return; c->argv[2] = tryObjectEncoding(c->argv[2]); - setKey(c,c->db,c->argv[1],c->argv[2],0); + setKey(c, c->db, c->argv[1], &c->argv[2], 0); + incrRefCount(c->argv[2]); notifyKeyspaceEvent(NOTIFY_STRING,"set",c->argv[1],c->db->id); server.dirty++; @@ -422,7 +441,6 @@ void getsetCommand(client *c) { void setrangeCommand(client *c) { int64_t oldLen = -1, newLen; - robj *o; long offset; sds value = c->argv[3]->ptr; const size_t value_len = sdslen(value); @@ -435,9 +453,9 @@ void setrangeCommand(client *c) { return; } - dictEntry *de; - o = lookupKeyWriteWithDictEntry(c->db,c->argv[1],&de); - if (o == NULL) { + dictEntryLink link; + kvobj *kv = lookupKeyWriteWithLink(c->db, c->argv[1], &link); + if (kv == NULL) { /* Return 0 when setting nothing on a non-existing string */ if (value_len == 0) { addReply(c,shared.czero); @@ -448,15 +466,15 @@ void setrangeCommand(client *c) { if (checkStringLength(c,offset,value_len) != C_OK) return; - o = createObject(OBJ_STRING,sdsnewlen(NULL, offset+value_len)); - dbAdd(c->db,c->argv[1],o); + robj *o = createObject(OBJ_STRING,sdsnewlen(NULL, offset+value_len)); + kv = dbAddByLink(c->db, c->argv[1], &o, &link); } else { /* Key exists, check type */ - if (checkType(c,o,OBJ_STRING)) + if (checkType(c,kv,OBJ_STRING)) return; /* Return existing string length when setting nothing */ - oldLen = stringObjectLen(o); + oldLen = stringObjectLen(kv); if (value_len == 0) { addReplyLongLong(c, oldLen); return; @@ -467,25 +485,25 @@ void setrangeCommand(client *c) { return; /* Create a copy when the object is shared or encoded. */ - o = dbUnshareStringValueWithDictEntry(c->db,c->argv[1],o,de); + kv = dbUnshareStringValueByLink(c->db, c->argv[1], kv, link); } if (value_len > 0) { - o->ptr = sdsgrowzero(o->ptr,offset+value_len); - memcpy((char*)o->ptr+offset,value,value_len); + kv->ptr = sdsgrowzero(kv->ptr,offset+value_len); + memcpy((char*)kv->ptr+offset,value,value_len); signalModifiedKey(c,c->db,c->argv[1]); notifyKeyspaceEvent(NOTIFY_STRING, "setrange",c->argv[1],c->db->id); server.dirty++; } - newLen = sdslen(o->ptr); + newLen = sdslen(kv->ptr); updateKeysizesHist(c->db,getKeySlot(c->argv[1]->ptr),OBJ_STRING,oldLen,newLen); addReplyLongLong(c,newLen); } void getrangeCommand(client *c) { - robj *o; + kvobj *o; long long start, end; char *str, llbuf[32]; size_t strlen; @@ -530,7 +548,7 @@ void mgetCommand(client *c) { addReplyArrayLen(c,c->argc-1); for (j = 1; j < c->argc; j++) { - robj *o = lookupKeyRead(c->db,c->argv[j]); + kvobj *o = lookupKeyRead(c->db, c->argv[j]); if (o == NULL) { addReplyNull(c); } else { @@ -562,14 +580,12 @@ void msetGenericCommand(client *c, int nx) { } } - int setkey_flags = nx ? SETKEY_DOESNT_EXIST : 0; for (j = 1; j < c->argc; j += 2) { c->argv[j+1] = tryObjectEncoding(c->argv[j+1]); - setKey(c, c->db, c->argv[j], c->argv[j + 1], setkey_flags); + /* if 'NX', no need set flags SETKEY_DOESNT_EXIST. Already verified earlier! */ + setKey(c, c->db, c->argv[j], &(c->argv[j+1]) , 0 /*flags*/); + incrRefCount(c->argv[j+1]); /* refcnt not incr by setKey() */ notifyKeyspaceEvent(NOTIFY_STRING,"set",c->argv[j],c->db->id); - /* In MSETNX, It could be that we're overriding the same key, we can't be sure it doesn't exist. */ - if (nx) - setkey_flags = SETKEY_ADD_OR_UPDATE; } server.dirty += (c->argc-1)/2; addReply(c, nx ? shared.cone : shared.ok); @@ -585,9 +601,9 @@ void msetnxCommand(client *c) { void incrDecrCommand(client *c, long long incr) { long long value, oldvalue; - robj *o, *new; - dictEntry *de; - o = lookupKeyWriteWithDictEntry(c->db,c->argv[1],&de); + robj *new; + dictEntryLink link; + kvobj *o = lookupKeyWriteWithLink(c->db, c->argv[1], &link); if (checkType(c,o,OBJ_STRING)) return; if (getLongLongFromObjectOrReply(c,o,&value,NULL) != C_OK) return; @@ -600,7 +616,6 @@ void incrDecrCommand(client *c, long long incr) { value += incr; if (o && o->refcount == 1 && o->encoding == OBJ_ENCODING_INT && - (value < 0 || value >= OBJ_SHARED_INTEGERS) && value >= LONG_MIN && value <= LONG_MAX) { new = o; @@ -613,10 +628,10 @@ void incrDecrCommand(client *c, long long incr) { new = createStringObjectFromLongLongForValue(value); if (o) { /* replace value in db and also update keysizes hist */ - dbReplaceValueWithDictEntry(c->db,c->argv[1],new,de); + dbReplaceValueWithLink(c->db, c->argv[1], &new, link); } else { /* Add new key to db and also update keysizes hist */ - dbAdd(c->db,c->argv[1],new); + dbAddByLink(c->db, c->argv[1], &new, &link); } } addReplyLongLongFromStr(c,new); @@ -654,10 +669,9 @@ void decrbyCommand(client *c) { void incrbyfloatCommand(client *c) { long double incr, value; - robj *o, *new; - dictEntry *de; - o = lookupKeyWriteWithDictEntry(c->db,c->argv[1],&de); + dictEntryLink link; + kvobj *o = lookupKeyWriteWithLink(c->db,c->argv[1],&link); if (checkType(c,o,OBJ_STRING)) return; if (getLongDoubleFromObjectOrReply(c,o,&value,NULL) != C_OK || getLongDoubleFromObjectOrReply(c,c->argv[2],&incr,NULL) != C_OK) @@ -668,11 +682,11 @@ void incrbyfloatCommand(client *c) { addReplyError(c,"increment would produce NaN or Infinity"); return; } - new = createStringObjectFromLongDouble(value,1); + robj *new = createStringObjectFromLongDouble(value,1); if (o) - dbReplaceValueWithDictEntry(c->db,c->argv[1],new,de); + dbReplaceValueWithLink(c->db, c->argv[1], &new, link); else - dbAdd(c->db,c->argv[1],new); + dbAddByLink(c->db, c->argv[1], &new, &link); signalModifiedKey(c,c->db,c->argv[1]); notifyKeyspaceEvent(NOTIFY_STRING,"incrbyfloat",c->argv[1],c->db->id); server.dirty++; @@ -688,14 +702,15 @@ void incrbyfloatCommand(client *c) { void appendCommand(client *c) { size_t totlen; - robj *o, *append; + robj *append; + kvobj *o; - dictEntry *de; - o = lookupKeyWriteWithDictEntry(c->db,c->argv[1],&de); + dictEntryLink link; + o = lookupKeyWriteWithLink(c->db,c->argv[1],&link); if (o == NULL) { /* Create the key */ c->argv[2] = tryObjectEncoding(c->argv[2]); - dbAdd(c->db,c->argv[1],c->argv[2]); + dbAddByLink(c->db, c->argv[1], &c->argv[2], &link); incrRefCount(c->argv[2]); totlen = stringObjectLen(c->argv[2]); } else { @@ -710,7 +725,7 @@ void appendCommand(client *c) { return; /* Append the value */ - o = dbUnshareStringValueWithDictEntry(c->db,c->argv[1],o,de); + o = dbUnshareStringValueByLink(c->db,c->argv[1],o,link); o->ptr = sdscatlen(o->ptr,append->ptr,append_len); totlen = sdslen(o->ptr); int64_t oldlen = totlen - append_len; @@ -719,15 +734,15 @@ void appendCommand(client *c) { signalModifiedKey(c,c->db,c->argv[1]); notifyKeyspaceEvent(NOTIFY_STRING,"append",c->argv[1],c->db->id); server.dirty++; - + addReplyLongLong(c,totlen); } void strlenCommand(client *c) { - robj *o; - if ((o = lookupKeyReadOrReply(c,c->argv[1],shared.czero)) == NULL || - checkType(c,o,OBJ_STRING)) return; - addReplyLongLong(c,stringObjectLen(o)); + kvobj *kv; + if ((kv = lookupKeyReadOrReply(c, c->argv[1], shared.czero)) == NULL || + checkType(c, kv, OBJ_STRING)) return; + addReplyLongLong(c,stringObjectLen(kv)); } /* LCS key1 key2 [LEN] [IDX] [MINMATCHLEN ] [WITHMATCHLEN] */ @@ -736,10 +751,9 @@ void lcsCommand(client *c) { long long minmatchlen = 0; sds a = NULL, b = NULL; int getlen = 0, getidx = 0, withmatchlen = 0; - robj *obja = NULL, *objb = NULL; - obja = lookupKeyRead(c->db,c->argv[1]); - objb = lookupKeyRead(c->db,c->argv[2]); + kvobj *obja = lookupKeyRead(c->db, c->argv[1]); + kvobj *objb = lookupKeyRead(c->db, c->argv[2]); if ((obja && obja->type != OBJ_STRING) || (objb && objb->type != OBJ_STRING)) { diff --git a/src/t_zset.c b/src/t_zset.c index d3cb7d67b..3f08eab6e 100644 --- a/src/t_zset.c +++ b/src/t_zset.c @@ -1838,8 +1838,8 @@ void zaddGenericCommand(client *c, int flags) { if (checkType(c,zobj,OBJ_ZSET)) goto cleanup; if (zobj == NULL) { if (xx) goto reply_to_client; /* No key + XX option: nothing to do. */ - zobj = zsetTypeCreate(elements, sdslen(c->argv[scoreidx+1]->ptr)); - dbAdd(c->db,key,zobj); + robj *o = zsetTypeCreate(elements, sdslen(c->argv[scoreidx + 1]->ptr)); + zobj = dbAdd(c->db,key,&o); } else { zsetTypeMaybeConvert(zobj, elements); } @@ -1893,15 +1893,14 @@ void zincrbyCommand(client *c) { void zremCommand(client *c) { robj *key = c->argv[1]; - robj *zobj; int deleted = 0, keyremoved = 0, j; - if ((zobj = lookupKeyWriteOrReply(c,key,shared.czero)) == NULL || - checkType(c,zobj,OBJ_ZSET)) return; + kvobj *zobj = lookupKeyWriteOrReply(c, key, shared.czero); + if (zobj == NULL || checkType(c,zobj,OBJ_ZSET)) return; int64_t oldlen = (int64_t) zsetLength(zobj); for (j = 2; j < c->argc; j++) { - if (zsetDel(zobj,c->argv[j]->ptr)) deleted++; + if (zsetDel(zobj, c->argv[j]->ptr)) deleted++; if (zsetLength(zobj) == 0) { /* Del key but don't update KEYSIZES. Else it will decr wrong bin in histogram */ dbDeleteSkipKeysizesUpdate(c->db, key); @@ -1935,7 +1934,6 @@ typedef enum { /* Implements ZREMRANGEBYRANK, ZREMRANGEBYSCORE, ZREMRANGEBYLEX commands. */ void zremrangeGenericCommand(client *c, zrange_type rangetype) { robj *key = c->argv[1]; - robj *zobj; int keyremoved = 0; unsigned long deleted = 0; zrangespec range; @@ -1966,8 +1964,8 @@ void zremrangeGenericCommand(client *c, zrange_type rangetype) { } /* Step 2: Lookup & range sanity checks if needed. */ - if ((zobj = lookupKeyWriteOrReply(c,key,shared.czero)) == NULL || - checkType(c,zobj,OBJ_ZSET)) goto cleanup; + kvobj *zobj = lookupKeyWriteOrReply(c, key, shared.czero); + if (zobj == NULL || checkType(c, zobj, OBJ_ZSET)) goto cleanup; if (rangetype == ZRANGE_RANK) { /* Sanitize indexes. */ @@ -2666,7 +2664,7 @@ void zunionInterDiffGenericCommand(client *c, robj *dstkey, int numkeysIndex, in /* read keys to be used for input */ for (i = 0, j = numkeysIndex+1; i < setnum; i++, j++) { - robj *obj = lookupKeyRead(c->db, c->argv[j]); + kvobj *obj = lookupKeyRead(c->db, c->argv[j]); if (obj != NULL) { if (obj->type != OBJ_ZSET && obj->type != OBJ_SET) { zfree(src); @@ -2875,7 +2873,7 @@ void zunionInterDiffGenericCommand(client *c, robj *dstkey, int numkeysIndex, in if (dstkey) { if (dstzset->zsl->length) { zsetConvertToListpackIfNeeded(dstobj, maxelelen, totelelen); - setKey(c, c->db, dstkey, dstobj, 0); + setKey(c, c->db, dstkey, &dstobj, 0); addReplyLongLong(c, zsetLength(dstobj)); notifyKeyspaceEvent(NOTIFY_ZSET, (op == SET_OP_UNION) ? "zunionstore" : @@ -2889,8 +2887,8 @@ void zunionInterDiffGenericCommand(client *c, robj *dstkey, int numkeysIndex, in notifyKeyspaceEvent(NOTIFY_GENERIC, "del", dstkey, c->db->id); server.dirty++; } + decrRefCount(dstobj); } - decrRefCount(dstobj); } else if (cardinality_only) { addReplyLongLong(c, cardinality); } else { @@ -3087,7 +3085,7 @@ static void zrangeResultEmitLongLongForStore(zrange_result_handler *handler, static void zrangeResultFinalizeStore(zrange_result_handler *handler, size_t result_count) { if (result_count) { - setKey(handler->client, handler->client->db, handler->dstkey, handler->dstobj, 0); + setKey(handler->client, handler->client->db, handler->dstkey, &handler->dstobj, 0); addReplyLongLong(handler->client, result_count); notifyKeyspaceEvent(NOTIFY_ZSET, "zrangestore", handler->dstkey, handler->client->db->id); server.dirty++; @@ -3098,8 +3096,8 @@ static void zrangeResultFinalizeStore(zrange_result_handler *handler, size_t res notifyKeyspaceEvent(NOTIFY_GENERIC, "del", handler->dstkey, handler->client->db->id); server.dirty++; } + decrRefCount(handler->dstobj); } - decrRefCount(handler->dstobj); } /* Initialize the consumer interface type with the requested type. */ @@ -3373,7 +3371,7 @@ void zrevrangebyscoreCommand(client *c) { void zcountCommand(client *c) { robj *key = c->argv[1]; - robj *zobj; + kvobj *zobj; zrangespec range; unsigned long count = 0; @@ -3450,7 +3448,7 @@ void zcountCommand(client *c) { void zlexcountCommand(client *c) { robj *key = c->argv[1]; - robj *zobj; + kvobj *zobj; zlexrangespec range; unsigned long count = 0; @@ -3657,7 +3655,6 @@ void zrangeGenericCommand(zrange_result_handler *handler, int argc_start, int st { client *c = handler->client; robj *key = c->argv[argc_start]; - robj *zobj; zrangespec range; zlexrangespec lexrange; int minidx = argc_start + 1; @@ -3759,7 +3756,7 @@ void zrangeGenericCommand(zrange_result_handler *handler, int argc_start, int st } /* Step 3: Lookup the key and get the range. */ - zobj = lookupKeyRead(c->db, key); + kvobj *zobj = lookupKeyRead(c->db, key); if (zobj == NULL) { if (store) { handler->beginResultEmission(handler, -1); @@ -3802,7 +3799,7 @@ cleanup: void zcardCommand(client *c) { robj *key = c->argv[1]; - robj *zobj; + kvobj *zobj; if ((zobj = lookupKeyReadOrReply(c,key,shared.czero)) == NULL || checkType(c,zobj,OBJ_ZSET)) return; @@ -3812,7 +3809,7 @@ void zcardCommand(client *c) { void zscoreCommand(client *c) { robj *key = c->argv[1]; - robj *zobj; + kvobj *zobj; double score; if ((zobj = lookupKeyReadOrReply(c,key,shared.null[c->resp])) == NULL || @@ -3827,9 +3824,8 @@ void zscoreCommand(client *c) { void zmscoreCommand(client *c) { robj *key = c->argv[1]; - robj *zobj; double score; - zobj = lookupKeyRead(c->db,key); + kvobj *zobj = lookupKeyRead(c->db, key); if (checkType(c,zobj,OBJ_ZSET)) return; addReplyArrayLen(c,c->argc - 2); @@ -3846,7 +3842,7 @@ void zmscoreCommand(client *c) { void zrankGenericCommand(client *c, int reverse) { robj *key = c->argv[1]; robj *ele = c->argv[2]; - robj *zobj; + kvobj *zobj; robj* reply; long rank; int opt_withscore = 0; @@ -3896,7 +3892,7 @@ void zrevrankCommand(client *c) { } void zscanCommand(client *c) { - robj *o; + kvobj *o; unsigned long long cursor; if (parseScanCursorOrReply(c,c->argv[2],&cursor) == C_ERR) return; @@ -4201,7 +4197,7 @@ static void zrandmemberReplyWithListpack(client *c, unsigned int count, listpack void zrandmemberWithCountCommand(client *c, long l, int withscores) { unsigned long count, size; int uniq = 1; - robj *zsetobj; + kvobj *zsetobj; if ((zsetobj = lookupKeyReadOrReply(c, c->argv[1], shared.emptyarray)) == NULL || checkType(c, zsetobj, OBJ_ZSET)) return; @@ -4406,7 +4402,7 @@ void zrandmemberWithCountCommand(client *c, long l, int withscores) { void zrandmemberCommand(client *c) { long l; int withscores = 0; - robj *zset; + kvobj *zset; listpackEntry ele; if (c->argc >= 3) { diff --git a/tests/integration/replication-rdbchannel.tcl b/tests/integration/replication-rdbchannel.tcl index 52637a4af..5d7836bb0 100644 --- a/tests/integration/replication-rdbchannel.tcl +++ b/tests/integration/replication-rdbchannel.tcl @@ -296,12 +296,15 @@ start_server {tags {"repl external:skip"}} { puts "peak_master_slave_buf_size $peak_master_slave_buf_size" puts "peak_replica_buf_size $peak_replica_buf_size" } - # memory on the master is less than 1mb - assert_lessthan [expr $peak_master_used_mem - $prev_used - $backlog_size] 1000000 - assert_lessthan $peak_master_rpl_buf [expr {$backlog_size + 1000000}] - assert_lessthan $peak_master_slave_buf_size 1000000 - # buffers in the replica are more than 10mb - assert_morethan $peak_replica_buf_size 10000000 + # Valgrind contribute to flakiness of this test + if {!$::valgrind} { + # memory on the master is less than 3mb (TODO: Fine tune thresholds) + assert_lessthan [expr $peak_master_used_mem - $prev_used - $backlog_size] 3000000 + assert_lessthan $peak_master_rpl_buf [expr {$backlog_size + 1000000}] + assert_lessthan $peak_master_slave_buf_size 1000000 + # buffers in the replica are more than 10mb + assert_morethan $peak_replica_buf_size 10000000 + } stop_write_load $load_handle } diff --git a/tests/unit/info.tcl b/tests/unit/info.tcl index 092c17cd3..c930a348e 100644 --- a/tests/unit/info.tcl +++ b/tests/unit/info.tcl @@ -504,35 +504,45 @@ start_server {tags {"info" "external:skip"}} { start_server {tags {"info" "external:skip"}} { test {memory: database and pubsub overhead and rehashing dict count} { r flushall + + # Better not set ht0_size to 4 since there is a probability that all + # keys will end up in the same bucket and rehashing will ended instantly. + set ht0_size [expr 1 << 3] + # ht1 size is twice the size of ht0 + set ht1_size [expr $ht0_size << 1] + + populate [expr $ht0_size - 1] + + # Verify rehashing is not ongoing + wait_for_condition 100 10 { + [dict get [r memory stats] db.dict.rehashing.count] == 0 + } else { + fail "Rehashing did not finish in time" + } + + # Verify the info reflects steady state set info_mem [r info memory] set mem_stats [r memory stats] assert_equal [getInfoProperty $info_mem mem_overhead_db_hashtable_rehashing] {0} - assert_equal [dict get $mem_stats overhead.db.hashtable.lut] {0} + assert_equal [dict get $mem_stats overhead.db.hashtable.lut] [expr $ht0_size * 8] assert_equal [dict get $mem_stats overhead.db.hashtable.rehashing] {0} assert_equal [dict get $mem_stats db.dict.rehashing.count] {0} - # Initial dict expand is not rehashing - r set a b - set info_mem [r info memory] - set mem_stats [r memory stats] - assert_equal [getInfoProperty $info_mem mem_overhead_db_hashtable_rehashing] {0} - assert_range [dict get $mem_stats overhead.db.hashtable.lut] 1 64 - assert_equal [dict get $mem_stats overhead.db.hashtable.rehashing] {0} - assert_equal [dict get $mem_stats db.dict.rehashing.count] {0} - # set 4 more keys to trigger rehashing + + # Set 2 more keys to trigger rehashing # get the info within a transaction to make sure the rehashing is not completed r multi - r set b c - r set c d - r set d e - r set e f + r set this_will_reach_max_load_factor 1 + r set this_must_be_rehashed 1 r info memory r memory stats set res [r exec] - set info_mem [lindex $res 4] - set mem_stats [lindex $res 5] - assert_range [getInfoProperty $info_mem mem_overhead_db_hashtable_rehashing] 1 64 - assert_range [dict get $mem_stats overhead.db.hashtable.lut] 1 192 - assert_range [dict get $mem_stats overhead.db.hashtable.rehashing] 1 64 + set info_mem [lindex $res 2] + set mem_stats [lindex $res 3] + + # Verify the info reflects rehashing state + assert_range [getInfoProperty $info_mem mem_overhead_db_hashtable_rehashing] 1 [expr $ht0_size * 8] + assert_equal [dict get $mem_stats overhead.db.hashtable.lut] [expr ($ht0_size + $ht1_size) * 8 ] + assert_equal [dict get $mem_stats overhead.db.hashtable.rehashing] [expr $ht0_size * 8] assert_equal [dict get $mem_stats db.dict.rehashing.count] {1} } } diff --git a/tests/unit/maxmemory.tcl b/tests/unit/maxmemory.tcl index 966ac4487..bbacfdcd0 100644 --- a/tests/unit/maxmemory.tcl +++ b/tests/unit/maxmemory.tcl @@ -148,29 +148,6 @@ start_server {tags {"maxmemory" "external:skip"}} { } start_server {tags {"maxmemory external:skip"}} { - test "Without maxmemory small integers are shared" { - r config set maxmemory 0 - r set a 1 - assert_refcount_morethan a 1 - } - - test "With maxmemory and non-LRU policy integers are still shared" { - r config set maxmemory 1073741824 - r config set maxmemory-policy allkeys-random - r set a 1 - assert_refcount_morethan a 1 - } - - test "With maxmemory and LRU policy integers are not shared" { - r config set maxmemory 1073741824 - r config set maxmemory-policy allkeys-lru - r set a 1 - r config set maxmemory-policy volatile-lru - r set b 1 - assert_refcount 1 a - assert_refcount 1 b - r config set maxmemory 0 - } foreach policy { allkeys-random allkeys-lru allkeys-lfu volatile-lru volatile-lfu volatile-random volatile-ttl diff --git a/tests/unit/memefficiency.tcl b/tests/unit/memefficiency.tcl index 3e4793a73..cdf99ace1 100644 --- a/tests/unit/memefficiency.tcl +++ b/tests/unit/memefficiency.tcl @@ -327,7 +327,7 @@ run_solo {defrag} { r set "{bigstream}smallitem" val - set expected_frag 1.5 + set expected_frag 1.49 if {$::accurate} { # scale the hash to 1m fields in order to have a measurable the latency for {set j 10000} {$j < 1000000} {incr j} { @@ -683,7 +683,7 @@ run_solo {defrag} { puts "frag [s allocator_frag_ratio]" puts "frag_bytes [s allocator_frag_bytes]" } - assert_morethan [s allocator_frag_ratio] 1.4 + assert_morethan [s allocator_frag_ratio] 1.35 catch {r config set activedefrag yes} e if {[r config get activedefrag] eq "activedefrag yes"} { diff --git a/tests/unit/type/hash-field-expire.tcl b/tests/unit/type/hash-field-expire.tcl index d1afb9c15..87abe2a46 100644 --- a/tests/unit/type/hash-field-expire.tcl +++ b/tests/unit/type/hash-field-expire.tcl @@ -1446,6 +1446,36 @@ start_server {tags {"external:skip needs:debug"}} { r config set hash-max-listpack-value 64 } + + test {Test HEXPIRE coexists with EXPIRE} { + # Verify HEXPIRE & EXPIRE coexists. When setting EXPIRE a new kvobj might be + # created whereas the old one can be ref by hash field expiration DS. + # Take care to set hexpire before expire. Verify all combinations of + # which expired first. + # Another point to verify is that whether hexpire deletes the last field + # and in turn the key (See f2). + foreach etime {10 1000} htime {10 1000} f2 {0 1} { + r del myhash + r hset myhash f1 v1 + if {$f2} { r hset myhash f2 v2 } + r hpexpire myhash $etime FIELDS 1 f1 + r pexpire myhash $htime + after 20 + # If EXPIRE is shorter, it should delete the key. + if {$etime == 10} { + assert_equal [r httl myhash FIELDS 1 f1] $T_NO_FIELD + assert_equal [r exists myhash] 0 + } else { + if {$htime == 10} { + assert_equal [r httl myhash FIELDS 1 f1] $T_NO_FIELD + assert_range [r pttl myhash] 500 1000 + } else { + assert_range [r httl myhash FIELDS 1 f1] 1 1000 + assert_range [r pttl myhash] 500 1000 + } + } + } + } } start_server {tags {"external:skip needs:debug"}} { diff --git a/tests/unit/type/incr.tcl b/tests/unit/type/incr.tcl index 4bc130bcb..2264d788b 100644 --- a/tests/unit/type/incr.tcl +++ b/tests/unit/type/incr.tcl @@ -75,13 +75,13 @@ start_server {tags {"incr"}} { assert_equal {-1} [r decrby key_not_exist 1] } - test {INCR uses shared objects in the 0-9999 range} { + test {INCR does not use shared objects} { r set foo -1 r incr foo - assert_refcount_morethan foo 1 + assert_refcount 1 foo r set foo 9998 r incr foo - assert_refcount_morethan foo 1 + assert_refcount 1 foo r incr foo assert_refcount 1 foo } diff --git a/tests/unit/type/string.tcl b/tests/unit/type/string.tcl index 94702ec3d..d00528052 100644 --- a/tests/unit/type/string.tcl +++ b/tests/unit/type/string.tcl @@ -671,4 +671,37 @@ if {[string match {*jemalloc*} [s mem_allocator]]} { assert_encoding "int" bar lappend res [r get bar] } {12 12} + + if {[string match {*jemalloc*} [s mem_allocator]]} { + test {Check MEMORY USAGE for embedded key strings with jemalloc} { + + proc expected_mem {key val with_expire exp_mem_usage exp_debug_sdslen} { + r del $key + r set $key $val + if {$with_expire} { r expire $key 5678315 } + assert_equal $exp_mem_usage [r memory usage $key] + assert_equal $exp_debug_sdslen [r debug sdslen $key] + } + + if {[s arch_bits] == 64} { + # 16 (kvobj) + 1 (key-hdr-size) + 1 (sdshdr5) + 4 (key) + 1 (\0) + 3 (sdshdr8) + 5 (val) + 1 (\0) = 32bytes + expected_mem x234 y2345 0 32 "key_sds_len:4, key_sds_avail:0, key_zmalloc: 32, val_sds_len:5, val_sds_avail:0, val_zmalloc: 0" + # 16 (kvobj) + 1 (key-hdr-size) + 1 (sdshdr5) + 4 (key) + 1 (\0) + 3 (sdshdr8) + 6 (val) + 1 (\0) = 33bytes + expected_mem x234 y23456 0 40 "key_sds_len:4, key_sds_avail:0, key_zmalloc: 40, val_sds_len:6, val_sds_avail:7, val_zmalloc: 0" + # 16 (kvobj) + 1 (key-hdr-size) + 1 (sdshdr5) + 4 (key) + 1 (\0) + 3 (sdshdr8) + 13 (val) + 1 (\0) = 40bytes + expected_mem x234 y234561234567 0 40 "key_sds_len:4, key_sds_avail:0, key_zmalloc: 40, val_sds_len:13, val_sds_avail:0, val_zmalloc: 0" + # 16 (kvobj) + 8 (expiry) + 1 (key-hdr-size) + 1 (sdshdr5) + 4 (key) + 1 (\0) + 3 (sdshdr8) + 13 (val) + 1 (\0) = 48bytes + expected_mem x234 y234561234567 1 48 "key_sds_len:4, key_sds_avail:0, key_zmalloc: 48, val_sds_len:13, val_sds_avail:0, val_zmalloc: 0" + } else { + # 12 (kvobj) + 1 (key-hdr-size) + 1 (sdshdr5) + 4 (key) + 1 (\0) + 3 (sdshdr8) + 9 (val) + 1 (\0) = 32bytes + expected_mem x234 y23456789 0 32 "key_sds_len:4, key_sds_avail:0, key_zmalloc: 32, val_sds_len:9, val_sds_avail:0, val_zmalloc: 0" + # 12 (kvobj) + 1 (key-hdr-size) + 1 (sdshdr5) + 4 (key) + 1 (\0) + 3 (sdshdr8) + 10 (val) + 1 (\0) = 33bytes + expected_mem x234 y234567890 0 40 "key_sds_len:4, key_sds_avail:0, key_zmalloc: 40, val_sds_len:10, val_sds_avail:7, val_zmalloc: 0" + # 12 (kvobj) + 1 (key-hdr-size) + 1 (sdshdr5) + 4 (key) + 1 (\0) + 3 (sdshdr8) + 17 (val) + 1 (\0) = 40bytes + expected_mem x234 y2345678901234567 0 40 "key_sds_len:4, key_sds_avail:0, key_zmalloc: 40, val_sds_len:17, val_sds_avail:0, val_zmalloc: 0" + # 12 (kvobj) + 8 (expiry) + 1 (key-hdr-size) + 1 (sdshdr5) + 4 (key) + 1 (\0) + 3 (sdshdr8) + 17 (val) + 1 (\0) = 48bytes + expected_mem x234 y2345678901234567 1 48 "key_sds_len:4, key_sds_avail:0, key_zmalloc: 48, val_sds_len:17, val_sds_avail:0, val_zmalloc: 0" + } + } {} {needs:debug} + } }