summaryrefslogtreecommitdiffstats
path: root/xlators/performance/quick-read
diff options
context:
space:
mode:
authorBrian Foster <bfoster@redhat.com>2012-05-03 10:09:27 -0400
committerAnand Avati <avati@redhat.com>2012-05-08 15:34:01 -0700
commitd6244e238976e45ccb8c64ab4a0fe443fe44c952 (patch)
tree5a76de353d5c7e83319b39fd103c51487afcf714 /xlators/performance/quick-read
parent11a19ce031932640584f8bc207274f4e08d97c5f (diff)
quick-read, md-cache: selectively cache xattr data to conserve memory
The md-cache translator can cache items for a long period of time and thus is sensitive to memory usage associated with the items it references. This implements two changes to help conserve memory: - quick-read - Migrate the file content data to a local dict and delete from the xdata dict that passes through the stack. - md-cache - Create a local dict to selectively store the xattr data md-cache is interested in. This includes a slight optimization to not allocate an empty dict in the case where we have not received any xattr's of interest. I've tested both changes independently and together by running a readdirp test against several compiled source trees (~340k files). The base test results in a 7.7GB RSS on the client. The quick-read modification cuts RSS down to 4.1GB, which is still large due to md-cache unintelligently caching a large number of empty dictionaries. The combined modification cuts RSS down to 462MB on the same workload. NOTE: Theoretically the md-cache change should supercede the quick-read change, but practically I save an extra 150MB or so with both. I already had the change and consider it an improvement that quick-read clean up after itself. BUG: 812876 Change-Id: Id59734d12dd6476b0e32480939e633448adb6884 Signed-off-by: Brian Foster <bfoster@redhat.com> Reviewed-on: http://review.gluster.com/3268 Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Raghavendra G <raghavendra@gluster.com> Reviewed-by: Amar Tumballi <amarts@redhat.com> Reviewed-by: Jeff Darcy <jdarcy@redhat.com>
Diffstat (limited to 'xlators/performance/quick-read')
-rw-r--r--xlators/performance/quick-read/src/quick-read.c21
1 files changed, 20 insertions, 1 deletions
diff --git a/xlators/performance/quick-read/src/quick-read.c b/xlators/performance/quick-read/src/quick-read.c
index beddfd878a8..000dcb45400 100644
--- a/xlators/performance/quick-read/src/quick-read.c
+++ b/xlators/performance/quick-read/src/quick-read.c
@@ -429,13 +429,32 @@ qr_lookup_cbk (call_frame_t *frame, void *cookie, xlator_t *this,
}
}
+ /*
+ * Create our own internal dict and migrate the file content
+ * over to it so it isn't floating around in other translator
+ * caches.
+ */
if (qr_inode->xattr) {
dict_unref (qr_inode->xattr);
qr_inode->xattr = NULL;
table->cache_used -= qr_inode->stbuf.ia_size;
}
- qr_inode->xattr = dict_ref (xdata);
+ qr_inode->xattr = dict_new();
+ if (!qr_inode->xattr) {
+ op_ret = -1;
+ op_errno = ENOMEM;
+ goto unlock;
+ }
+
+ if (dict_set(qr_inode->xattr, GF_CONTENT_KEY, content) < 0) {
+ op_ret = -1;
+ op_errno = ENOMEM;
+ goto unlock;
+ }
+
+ dict_del(xdata, GF_CONTENT_KEY);
+
qr_inode->stbuf = *buf;
table->cache_used += buf->ia_size;