From ae47befebeda2de5fd2d706090cbacf4ef60c785 Mon Sep 17 00:00:00 2001 From: Jeff Darcy Date: Fri, 14 Oct 2016 10:04:07 -0400 Subject: libglusterfs: make memory pools more thread-friendly Early multiplexing tests revealed *massive* contention on certain pools' global locks - especially for dictionaries and secondarily for call stubs. For the thread counts that multiplexing can create, a more lock-free solution is clearly needed. Also, the current mem-pool implementation does a poor job releasing memory back to the system, artificially inflating memory usage to match whatever the worst case was since the process started. This is bad in general, but especially so for multiplexing where there are more pools and a major point of the whole exercise is to reduce memory consumption. The basic ideas for the new design are these There is one pool, globally, for each power-of-two size range. Every attempt to create a new pool within this range will instead add a reference to the existing pool. Instead of adding pools for each translator within each multiplexed brick (potentially infinite and quite possibly thousands), we allocate one set of size-based pools per *thread* (hundreds at worst). Each per-thread pool is divided into hot and cold lists. Every allocation first attempts to use the hot list, then the cold list. When objects are freed, they always go on the hot list. There is one global "pool sweeper" thread, which periodically reclaims everything in each pool's cold list and then "demotes" the current hot list to be the new cold list. For normal allocation activity, only a per-thread lock need be taken, and even that only to guard against very rare contention from the pool sweeper. When threads start and stop, a global lock must be taken to add them to the pool sweeper's list. Lock contention is therefore extremely low, and the hot/cold lists also provide good locality. A more complete explanation (of a similar earlier design) can be found here: http://www.gluster.org/pipermail/gluster-devel/2016-October/051160.html Change-Id: I5bc8a1ba57cfb553998f979a498886e0d006e665 BUG: 1385758 Signed-off-by: Jeff Darcy Reviewed-on: https://review.gluster.org/15645 Reviewed-by: Xavier Hernandez Smoke: Gluster Build System NetBSD-regression: NetBSD Build System CentOS-regression: Gluster Build System Reviewed-by: Shyamsundar Ranganathan --- glusterfsd/src/glusterfsd.c | 7 +++++++ 1 file changed, 7 insertions(+) (limited to 'glusterfsd/src') diff --git a/glusterfsd/src/glusterfsd.c b/glusterfsd/src/glusterfsd.c index 1f7b63e7594..f402246e78e 100644 --- a/glusterfsd/src/glusterfsd.c +++ b/glusterfsd/src/glusterfsd.c @@ -2472,6 +2472,13 @@ main (int argc, char *argv[]) if (ret) goto out; + /* + * If we do this before daemonize, the pool-sweeper thread dies with + * the parent, but we want to do it as soon as possible after that in + * case something else depends on pool allocations. + */ + mem_pools_init (); + #ifdef GF_LINUX_HOST_OS ret = set_oom_score_adj (ctx); if (ret) -- cgit