glusterfs.git/libglusterfs/src/glusterfs, branch v6.4

features/shard: Fix block-count accounting upon truncate to lower size

2019-07-03T11:16:10+00:00

Backport of:
> BUG: bz#1705884
> Change-Id: I9128a192e9bf8c3c3a959e96b7400879d03d7c53
> Signed-off-by: Krutika Dhananjay 

The way delta_blocks is computed in shard is incorrect, when a file
is truncated to a lower size. The accounting only considers change
in size of the last of the truncated shards.

FIX:

Get the block-count of each shard just before an unlink at posix in
xdata.  Their summation plus the change in size of last shard
(from an actual truncate) is used to compute delta_blocks which is
used in the xattrop for size update.

Change-Id: I9128a192e9bf8c3c3a959e96b7400879d03d7c53
fixes: bz#1716871
Signed-off-by: Krutika Dhananjay 
(cherry picked from commit 400b66d568ad18fefcb59949d1f8368d487b9a80)

core: fix memory allocation issues

2019-07-03T06:26:55+00:00

Two problems have been identified that caused that gluster's memory
usage were twice higher than required.

1. An off by 1 error caused that all objects allocated from the memory
   pools were taken from a pool bigger than required. Since each pool
   corresponds to a size equal to a power of two, this was wasting half
   of the available memory.

2. The header information used for accounting on each memory object was
   not taken into consideration when searching for a suitable memory
   pool. It was added later when each individual block was allocated.
   This made this space "invisible" to memory accounting.

Credits: Thanks to Nithya Balachandran for identifying this problem and
         testing this patch.

Backport of:
> BUG: 1722802
> Change-Id: I90e27ad795fe51ca11c13080f62207451f6c138c
> Signed-off-by: Xavi Hernandez 

Fixes: bz#1724210
Change-Id: I90e27ad795fe51ca11c13080f62207451f6c138c
Signed-off-by: Xavi Hernandez

core: avoid dynamic TLS allocation when possible

2019-07-03T06:25:48+00:00

Some interdependencies between logging and memory management functions
make it impossible to use the logging framework before initializing
memory subsystem because they both depend on Thread Local Storage
allocated through pthread_key_create() during initialization.

This causes a crash when we try to log something very early in the
initialization phase.

To prevent this, several dynamically allocated TLS structures have
been replaced by static TLS reserved at compile time using '__thread'
keyword. This also reduces the number of error sources, making
initialization simpler.

Backport of:
> BUG: 1193929
> Change-Id: I8ea2e072411e30790d50084b6b7e909c7bb01d50
> Signed-off-by: Xavi Hernandez 

Updates: bz#1724210
Change-Id: I8ea2e072411e30790d50084b6b7e909c7bb01d50
Signed-off-by: Xavi Hernandez

mem-pool: remove dead code.

2019-06-28T10:49:37+00:00

Backport of:
> Change-Id: I3bbda719027b45e1289db2e6a718627141bcbdc8
> BUG: 1193929
> Signed-off-by: Yaniv Kaul 

Change-Id: I3bbda719027b45e1289db2e6a718627141bcbdc8
updates: bz#1724210
Signed-off-by: Yaniv Kaul

core: handle memory accounting correctly

2019-04-23T11:53:02+00:00

When a translator stops, memory accounting for that translator is not
destroyed (because there could remain memory allocated that references
it), but mutexes that coordinate updates of memory accounting were
destroyed. This caused incorrect memory accounting and even crashes in
debug mode.

This patch also fixes some other things:

* Reduce the number of atomic operations needed to manage memory
  accounting.
* Correctly account memory when realloc() is used.
* Merge two critical sections into one.
* Cleaned the code a bit.

Backport of:
> Change-Id: Id5eaee7338729b9bc52c931815ca3ff1e5a7dcc8
> BUG: bz#1659334
> Signed-off-by: Xavi Hernandez 

Change-Id: Id5eaee7338729b9bc52c931815ca3ff1e5a7dcc8
Fixes: bz#1702271
Signed-off-by: Xavi Hernandez

core: Log level changes do not effect on running client process

2019-04-16T10:59:36+00:00

Problem: commit c34e4161f3cb6539ec83a9020f3d27eb4759a975 set log-level
         per xlator during reconfigure only for a brick process not for
         the client process.

Solution: 1) Change per xlator log-level only if brick_mux is enabled.To make sure
             about brick multiplex introudce a flag brick_mux at ctx->cmd_args.

Note: There are two other changes done with this patch
      1) Ignore client-log-level option to attach a brick with
         already running brick if brick_mux is enabled
      2) Add a log to print pid of the running process to make easier
         debugging

> Change-Id: I39e85de778e150d0685cd9a79425ce8b4783f9c9
> Signed-off-by: Mohit Agrawal 
> Fixes: bz#1696046
> (Cherry picked from commit 798aadbe51a9a02dd98a0f861cc239ecf7c8ed57)
> (Reviewed on upstream link https://review.gluster.org/#/c/glusterfs/+/22495/)

Change-Id: If91682830f894ab8f6857f19dcb1797fc15ca64c
Fixes: bz#1699715
Signed-off-by: Mohit Agrawal

logging: Fix GF_LOG_OCCASSIONALLY API

2019-04-16T10:49:22+00:00

GF_LOG_OCCASSIONALLY doesn't log on the first instance rather at every
42nd iterations which isn't effective as in some cases we might not have
the code flow hitting the same log for as many as 42 times and we'd end
up suppressing the log.

Updates: bz#1679904
Change-Id: Iee293281d25a652b64df111d59b13de4efce06fa
Signed-off-by: Atin Mukherjee 
(cherry picked from commit d0d3e10d44366c68fc153e48b229e72a4aa26e61)

core: make compute_cksum function op_version compatible

2019-03-08T14:45:41+00:00

Problem: commit 5a152a changed the mechansim of computing the
checksum. In heterogeneous cluster, peers are running into
rejected state because we have different cksum computation
mechansims in upgraded and non-upgraded nodes.

Solution: add a check for op-version so that all the nodes
in the cluster follow the same mechanism for computing the
cksum.

fixes: bz#1684029

> Change-Id: I1508f000e8c9895588b6011b8b6cc0eda7102193
> BUG: bz#1685120
> Signed-off-by: Sanju Rakonde 
> (cherry picked from commit 073444b693b7a91c42963512e0fdafb57ad46670)

Change-Id: I1508f000e8c9895588b6011b8b6cc0eda7102193

socket: socket event handlers now return void

2019-03-02T11:54:24+00:00

Problem:
Returning any value from socket event handlers to the event sub-system
doesn't make sense since event sub-system cannot handle socket
sub-system errors.

Solution:
Change return type of all socket event handlers to 'void'

mainline:
> Change-Id: I70dc2c57f12b7ea2fae41120f71aa0d7fe0b2b6f
> Fixes: bz#1651246
> Signed-off-by: Milind Changire 
> Reviewed-on: https://review.gluster.org/c/glusterfs/+/22221

Change-Id: I70dc2c57f12b7ea2fae41120f71aa0d7fe0b2b6f
Fixes: bz#1683900
Signed-off-by: Milind Changire 
(cherry picked from commit 776ba851c6ee6c265253d44cf1d6e4e3d4a21772)

mount/fuse: expose auto-invalidation as a mount option

2019-02-02T03:07:35+00:00

Auto invalidation is necessary when same (meta)data is shared/access
across multiple mounts. However, if (meta)data is not shared, all
relevant I/O goes through the cache of single mount and hence is
coherent with (meta)data on bricks always. So, fuse-auto-invalidation
can be disabled for this case which gives a huge performance boost for
workloads that write data and then immediately read the data they just
wrote.

From glusterfs --help,


      --auto-invalidation[=BOOL]   controls whether fuse-kernel can
                             auto-invalidate attribute, dentry and page-cache.
                             Disable this only if same files/directories are
                             not accessed across two different mounts
                             concurrently [default: "on"]


Details on how disabling auto-invalidation helped to reduce pgbench
init times can be found at [1]. Time taken for pgbench init of scale
8000 was 8340s. That will be an improvement of 86% (59280s vs 8340s)
with auto-invalidations turned off along with other
optimizations. Just disabling auto-invalidation contributed 56%
improvement by reducing the total time taken by 33260s.

[1] https://www.spinics.net/lists/gluster-devel/msg25907.html

Change-Id: I0ed730dba9064bd9c576ad1800170a21e100e1ce
Signed-off-by: Raghavendra Gowdappa 
updates: bz#1664934