Objects allocated from a per-thread memory pool keep a reference to it to be able to return the object to the pool when not used anymore. The object holding this reference can have a long life cycle that could survive a glfs_fini() call. This means that it's unsafe to destroy memory pools from glfs_fini(). Another side effect of destroying memory pools from glfs_fini() is that the TLS variable that points to one of those pools cannot be reset for all alive threads. This means that any attempt to allocate memory from those threads will access already free'd memory, which is very dangerous. To fix these issues, mem_pools_fini() doesn't destroy pool lists anymore. They should be destroyed when the library is unloaded or the process is terminated, but this cannot be done right now because gluster doesn't stop other threads before calling exit(), which could cause some races. This patch is the backport of 2 master patches: > Change-Id: Ib189a5510ab6bdac78983c6c65a022e9634b0965 > Fixes: bz#1801684 > Signed-off-by: Xavi Hernandez <xhernandez@redhat.com> > > Change-Id: Id7cfb4407fcf208e28f03a7c3cdc3ef9c1f3bf9b > Fixes: bz#1801684 > Signed-off-by: Xavi Hernandez <xhernandez@redhat.com> Change-Id: Id7cfb4407fcf208e28f03a7c3cdc3ef9c1f3bf9b Fixes: bz#1805668 Signed-off-by: Xavi Hernandez <xhernandez@redhat.com>
Two problems have been identified that caused that gluster's memory
usage were twice higher than required.
1. An off by 1 error caused that all objects allocated from the memory
pools were taken from a pool bigger than required. Since each pool
corresponds to a size equal to a power of two, this was wasting half
of the available memory.
2. The header information used for accounting on each memory object was
not taken into consideration when searching for a suitable memory
pool. It was added later when each individual block was allocated.
This made this space "invisible" to memory accounting.
Credits: Thanks to Nithya Balachandran for identifying this problem and
testing this patch.
>Fixes: bz#1722802
Change-Id: I90e27ad795fe51ca11c13080f62207451f6c138c
>Signed-off-by: Xavi Hernandez <xhernandez@redhat.com>
>(cherry picked from commit 1716a907da1a835b658740f1325033d7ddd44952)
Fixes: bz#1748774
Change-Id: I90e27ad795fe51ca11c13080f62207451f6c138c
Signed-off-by: Xavi Hernandez <xhernandez@redhat.com>
There are many include statements that are not needed. A previous more ambitious attempt failed because of *BSD plafrom (see https://review.gluster.org/#/c/glusterfs/+/21929/ ) Now trying a more conservative reduction. It does not solve all circular deps that we have, but it does reduce some of them. There is just too much to handle reasonably (dht-common.h includes dht-lock.h which includes dht-common.h ...), but it does reduce the overall number of lines of include we need to look at in the future to understand and fix the mess later one. Change-Id: I550cd001bdefb8be0fe67632f783c0ef6bee3f9f updates: bz#1193929 Signed-off-by: Yaniv Kaul <ykaul@redhat.com>
updates bz#1193929 Change-Id: I245c065b209bcce5db939b6a0a934ba6fd393b47 Signed-off-by: Pranith Kumar K <pkarampu@redhat.com>
1. Removed some code that was not needed. It did not really do anything. 2. CALLOC -> MALLOC in one place. Compile-tested only! updates: bz#1193929 Signed-off-by: Yaniv Kaul <ykaul@redhat.com> Change-Id: I4419161e1bb636158e32b5d33044b06f1eef2449
Some interdependencies between logging and memory management functions make it impossible to use the logging framework before initializing memory subsystem because they both depend on Thread Local Storage allocated through pthread_key_create() during initialization. This causes a crash when we try to log something very early in the initialization phase. To prevent this, several dynamically allocated TLS structures have been replaced by static TLS reserved at compile time using '__thread' keyword. This also reduces the number of error sources, making initialization simpler. Updates: bz#1193929 Change-Id: I8ea2e072411e30790d50084b6b7e909c7bb01d50 Signed-off-by: Xavi Hernandez <xhernandez@redhat.com>
Currently GF_ASSERT is done under mem_accounting lock at some places. On a GF_ASSERT failure, gf_msg_callingfn is called which calls gf_malloc internally and it takes the same mem_accounting lock leading to deadlock. This is a temporary fix to avoid any hang issue in master. https://review.gluster.org/#/c/glusterfs/+/22589/ is being worked on in the mean while so that GF_ASSERT can be used under mem_accounting lock. Change-Id: I6d67f23979e7edd2695bdc6aab2997dae4a4060a updates: bz#1700865 Signed-off-by: Susant Palai <spalai@redhat.com>
When a translator stops, memory accounting for that translator is not destroyed (because there could remain memory allocated that references it), but mutexes that coordinate updates of memory accounting were destroyed. This caused incorrect memory accounting and even crashes in debug mode. This patch also fixes some other things: * Reduce the number of atomic operations needed to manage memory accounting. * Correctly account memory when realloc() is used. * Merge two critical sections into one. * Cleaned the code a bit. Change-Id: Id5eaee7338729b9bc52c931815ca3ff1e5a7dcc8 Updates: bz#1659334 Signed-off-by: Xavi Hernandez <xhernandez@redhat.com>
Change-Id: I3bbda719027b45e1289db2e6a718627141bcbdc8 updates: bz#1193929 Signed-off-by: Yaniv Kaul <ykaul@redhat.com>
This reverts commit b87c397091bac6a4a6dec4e45a7671fad4a11770. There seems to be some performance regression with the patch and hence recommended to have it reverted. Updates: #325 Change-Id: Id85d6203173a44fad6cf51d39b3e96f37afcec09