glusterfs.git/extras, branch v3.12.0

gluster-block: strict-o-direct should be on

2017-08-29T13:36:24+00:00

       Backport of https://review.gluster.org/18120

tcmu-runner is not going to open block with O_SYNC anymore
so writes have a chance of getting cached in write-behind
when that happens, there is a chance that on failover some
data could be stuck in cache and be lost.

 >BUG: 1485962
 >Change-Id: If9835d914821dfc4ff432dc96775677a55d2918f
 >Signed-off-by: Pranith Kumar K 

BUG: 1486122
Change-Id: If9835d914821dfc4ff432dc96775677a55d2918f
Signed-off-by: Pranith Kumar K 
Reviewed-on: https://review.gluster.org/18126
Smoke: Gluster Build System 
Reviewed-by: Krutika Dhananjay 
Tested-by: Krutika Dhananjay 
CentOS-regression: Gluster Build System 
Reviewed-by: Shyamsundar Ranganathan

glusterd: Gluster should keep PID file in correct location

2017-08-12T13:38:30+00:00

Currently Gluster keeps process pid information of all the daemons
and brick processes in Gluster configuration file directory
(ie., /var/lib/glusterd/*).

These pid files should be seperate from configuration files.
Deletion of the configuration file directory might result into serious problems.
Also, /var/run/gluster is the default placeholder directory for pid files.

So, with this fix Gluster will keep all process pid information of all
processes in /var/run/gluster/* directory.

> BUG: 1258561
> Signed-off-by: Gaurav Kumar Garg 
> Signed-off-by: Saravanakumar Arumugam 
> Reviewed-on: https://review.gluster.org/13580
> Tested-by: MOHIT AGRAWAL 
> Smoke: Gluster Build System 
> CentOS-regression: Gluster Build System 
> Reviewed-by: Atin Mukherjee 
> cherry pick from commit 220d406ad13d840e950eef001a2b36f87570058d

BUG: 1480459
Change-Id: Idb09e3fccb6a7355fbac1df31082637c8d7ab5b4
Signed-off-by: Mohit Agrawal 
Reviewed-on: https://review.gluster.org/18023
Reviewed-by: Atin Mukherjee 
Smoke: Gluster Build System 
CentOS-regression: Gluster Build System

scripts: invalid test(1) in extras/S32gluster_enable_shared_storage.sh

2017-08-02T15:10:47+00:00

test(1) man pages says -eq is for INTEGER compares, and = is for
string compares.

Also note the comment that -a and -o are ambiguous and to use
test && test or test || test instead.

This bug has existed since 2015! (yikes)

Found while testing localtime logging and running glusterd in the
foreground.

Change-Id: Ia544f7295e247b981504d085ebc4c533ab60ba84
BUG: 1476819
Signed-off-by: Kaleb S. KEITHLEY 
Reviewed-on: https://review.gluster.org/17926
Smoke: Gluster Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Shyamsundar Ranganathan

extras: Turn eager-lock off for gluster-block

2017-08-02T15:02:43+00:00

With the current implementation of eager-lock FINODELK is taking so much
time that cassandra workload times out and errors out. AFR eager-locking
needs to be changed similar to EC eager-locking to make things work
as expected. In the interim, it is better to turn it off

This is how the profile looks if eager-lock is turned on:
      0.35     628.26 us      64.00 us  129882.00 us          42278    FXATTROP
     17.45   16500.54 us     212.00 us  375829.00 us          79568       WRITE
     81.76  209862.12 us      15.00 us 1992486.00 us          29318    FINODELK

This is how profile looks if eager-lock is turned off:
      1.87     283.71 us      65.00 us  298970.00 us          68346    FXATTROP
      6.33     199.04 us      13.00 us  373428.00 us         330524    FINODELK
     10.37    3151.47 us      53.00 us 1528484.00 us          34172       FSYNC
     81.31    5110.45 us     270.00 us 1519722.00 us         165244       WRITE

 >BUG: 1477404
 >Change-Id: I98026b1ecf30002ddac01be76f375c2e8c0b7838
 >Signed-off-by: Pranith Kumar K 

BUG: 1477405
Change-Id: I98026b1ecf30002ddac01be76f375c2e8c0b7838
Signed-off-by: Pranith Kumar K 
Reviewed-on: https://review.gluster.org/17955
CentOS-regression: Gluster Build System 
Smoke: Gluster Build System 
Reviewed-by: Krutika Dhananjay 
Reviewed-by: Prasanna Kumar Kalever

extras: Disable remote-dio in gluster-block profile

2017-07-31T16:29:58+00:00

gluster-block file is opened with O_DIRECT but because block-profile has
remote-dio enabled, it is leading to high latency for fsync which leads to
failures in cassandra. Disabling remote-dio fixed this issue. We need to change
remote-dio to disabled in gluster-block.

 >BUG: 1474190
 >Change-Id: Ifd845ea9cbdcc08dd6073faca6082682af376ca3
 >Signed-off-by: Pranith Kumar K 
 >Reviewed-on: https://review.gluster.org/17856
 >CentOS-regression: Gluster Build System 
 >Reviewed-by: Krutika Dhananjay 
 >Smoke: Gluster Build System 
 >Reviewed-by: Jeff Darcy 
 >Reviewed-by: Niels de Vos 
 >(cherry picked from commit 0b3fec6924cad5c9f38941550ab4106972efa5cc)

BUG: 1476653
Change-Id: Ifd845ea9cbdcc08dd6073faca6082682af376ca3
Signed-off-by: Pranith Kumar K 
Reviewed-on: https://review.gluster.org/17918
CentOS-regression: Gluster Build System 
Smoke: Gluster Build System 
Reviewed-by: Krutika Dhananjay 
Reviewed-by: Niels de Vos 
Reviewed-by: Shyamsundar Ranganathan

group-gluster-block: Set default shard-block-size to 64MB

2017-07-31T16:29:09+00:00

With 4MB shard size I/O slows down more because of the extra
inodelk/xattrops in replicate. So increasing it to 64MB which
gave better performance than 4MB.

To simulate writes on a preallocated VM-image, fallocate the file and then do
dd with notrunc
do "fallocate -l 1GB" then "dd if=/dev/zero of=file-1GB bs=1MB count=1024 conv=notrunc"

These are the results on my laptop for dd:
With 4MB:
      1.84    1357.37 us      19.00 us   12431.00 us           1188    FINODELK
      2.45     255.08 us      58.00 us    4038.00 us           8428       WRITE
     95.69   78967.76 us      30.00 us 20324240.00 us           1063    FXATTROP

With 64MB:
      0.13      59.36 us      15.00 us     814.00 us            657    FINODELK
      6.02     225.53 us      69.00 us    6556.00 us           8205       WRITE
     93.82  103015.12 us      32.00 us 13046368.00 us            280    FXATTROP

 >BUG: 1475605
 >Change-Id: I4ed5441409df639e38c731ba0d140fe92902f25f
 >Signed-off-by: Pranith Kumar K 
 >Reviewed-on: https://review.gluster.org/17887
 >CentOS-regression: Gluster Build System 
 >Smoke: Gluster Build System 
 >Reviewed-by: Krutika Dhananjay 
 >(cherry picked from commit abfbc3eb821e144ddbfdc5d7da401557b52beaf1)

BUG: 1476654
Change-Id: I4ed5441409df639e38c731ba0d140fe92902f25f
Signed-off-by: Pranith Kumar K 
Reviewed-on: https://review.gluster.org/17919
CentOS-regression: Gluster Build System 
Smoke: Gluster Build System 
Reviewed-by: Krutika Dhananjay 
Reviewed-by: Niels de Vos 
Reviewed-by: Shyamsundar Ranganathan

extras: Change Makefile generation in generate_xlator.py

2017-07-18T16:44:27+00:00

Makefile generation should include default LD_FLAGS and also include rpc
related paths in include path

Change-Id: I45e1c97b96f08bbfe4663384f4873726febef9f6
Signed-off-by: Pranith Kumar K 
Reviewed-on: https://review.gluster.org/17811
Reviewed-by: Niels de Vos 
Smoke: Gluster Build System 
CentOS-regression: Gluster Build System

systemd/glusterfssharedstorage : remove dependency for var-run-gluster-shared_storage

2017-07-17T11:41:00+00:00

Currently the script used by glusterfssharedstorage have dependency over
var-run-gluster-shared_storage. But this service will be present only if
node has rebooted. Also in reboot scenario , there is a chance that this
service can be executed before creating var-run-gluster-shared_storage.
In that case glusterfssharedstorage will get succeed even without mounting
the shared storage

Also the type of glusterfssharedstorage changed to "forking" so that it can
be active(instead of dead) after the successful start.

Change-Id: I1c02cc64946e534d845aa7ec7b72644bbe4d26f9
BUG: 1452527
Signed-off-by: Jiffin Tony Thottan 
Reviewed-on: https://review.gluster.org/17658
Smoke: Gluster Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: soumya k 
Reviewed-by: Kaleb KEITHLEY

extras: Enable stat-prefetch in virt profile

2017-07-10T07:53:36+00:00

In the internal testing that was done, stat-prefetch did help
reduce the number of stats coming from qemu hitting the disk,
and thereby improved performance.

Change-Id: Icf1ce62ecf4e96b97e1946a77b30434157a7786a
BUG: 1468191
Signed-off-by: Krutika Dhananjay 
Reviewed-on: https://review.gluster.org/17713
CentOS-regression: Gluster Build System 
Smoke: Gluster Build System 
Reviewed-by: Amar Tumballi 
Reviewed-by: Pranith Kumar Karampuri

Scripts to identify quota accouting issues

2017-06-30T08:39:05+00:00

The patch contains 2 scripts:

log_accounting.sh does a du -h on the FS hierarchy and a quota list
on the hierarchy and interleaves the two output. We can then identify
which directory(s) in FS has caused the accounting to go bad and try
to investigate what fops happened on those directories. We can also
limit the set of directories on which we need to set dirty xattr to
correct accounting.
xattr_analysis.py reads all the xattr of a brick and dumps it a human
readable form to ease debugging.

Change-Id: I2155561d10c08dc3ab9e8b09dbd258f0592b4d33
BUG: 1466188
Signed-off-by: Sanoj Unnikrishnan 
Reviewed-on: https://review.gluster.org/17649
CentOS-regression: Gluster Build System 
Smoke: Gluster Build System 
Reviewed-by: Raghavendra G