glusterfs.git/xlators, branch v8.1 afr: add null check for thin-arbiter gfid. 2020-08-21T10:42:33+00:00 Ravishankar N ravishankar@redhat.com 2020-08-19T05:44:25+00:00 f120311737cf681c36423d8551e2e218509cd5f7 Problem: Lookup/creation of thin-arbiter ID file happens in background during mounting. On new volumes, if the ID file creation is in progress, and a FOP fails on data brick, a post-op (xattrop) is attemtped on TA. Since the TA file's gfid is null at this point, the ASSERT checks in protocol/ client causes a crash. Fix: Given that we decided to do Lookup/creation of thin-arbiter in background, fail the other AFR FOPS on TA if the ID file's gfid is null instead of winding it down to protocol/client. Also remove afr_changelog_thin_arbiter_post_op() which seems to be dead code. Updates: #763 Change-Id: I70dc666faf55cc5c8f7cf8e7d36085e4fa399c4d Signed-off-by: Ravishankar N <ravishankar@redhat.com> (cherry picked from commit f9b5074394e3d2f3b6728aab97230ba620879426) 
Problem:
Lookup/creation of thin-arbiter ID file happens in background during
mounting. On new volumes, if the  ID file creation is in progress, and a
FOP fails on data brick, a post-op (xattrop) is attemtped on TA. Since
the TA file's gfid is null at this point, the ASSERT checks in protocol/
client causes a crash.

Fix:
Given that we decided to do Lookup/creation of thin-arbiter in
background, fail the other AFR FOPS on TA if the ID file's gfid is null
instead of winding it down to protocol/client.

Also remove afr_changelog_thin_arbiter_post_op() which seems to be dead
code.

Updates: #763
Change-Id: I70dc666faf55cc5c8f7cf8e7d36085e4fa399c4d
Signed-off-by: Ravishankar N <ravishankar@redhat.com>
(cherry picked from commit f9b5074394e3d2f3b6728aab97230ba620879426)
open-behind: fix call_frame leak 2020-08-21T10:39:37+00:00 Xavi Hernandez xhernandez@redhat.com 2020-08-19T21:27:38+00:00 4398db9d70f34e9a8af88fe3de564a906db7b182 When an open was delayed, a copy of the frame was created because the current frame was used to unwind the "fake" open. When the open was actually sent, the frame was correctly destroyed. However if the file was closed before needing to send the open, the frame was not destroyed. This patch correctly destroys the frame in all cases. Change-Id: I8c00fc7f15545c240e8151305d9e4cf06d653926 Signed-off-by: Xavi Hernandez <xhernandez@redhat.com> Fixes: #1440 
When an open was delayed, a copy of the frame was created because the
current frame was used to unwind the "fake" open. When the open was
actually sent, the frame was correctly destroyed. However if the file
was closed before needing to send the open, the frame was not destroyed.

This patch correctly destroys the frame in all cases.

Change-Id: I8c00fc7f15545c240e8151305d9e4cf06d653926
Signed-off-by: Xavi Hernandez <xhernandez@redhat.com>
Fixes: #1440
posix: Implement a janitor thread to close fd 2020-08-21T10:38:07+00:00 Mohit Agrawal moagrawa@redhat.com 2020-07-27T12:38:00+00:00 e173c5b0ee32c210a7d36f03f1847c42218a62e5 Problem: In the commit fb20713b380e1df8d7f9e9df96563be2f9144fd6 we use syntask to close fd but we have found the patch is reducing the performance Solution: Use janitor thread to close fd's and save the pfd ctx into ctx janitor list and also save the posix_xlator into pfd object to avoid the race condition during cleanup in brick_mux environment Change-Id: Ifb3d18a854b267333a3a9e39845bfefb83fbc092 Fixes: #1396 Signed-off-by: Mohit Agrawal <moagrawa@redhat.com> (cherry picked from commit 41b9616435cbdf671805856e487e373060c9455b) 
Problem: In the commit fb20713b380e1df8d7f9e9df96563be2f9144fd6 we use
         syntask to close fd but we have found the patch is reducing the
         performance

Solution: Use janitor thread to close fd's and save the pfd ctx into
          ctx janitor list and also save the posix_xlator into pfd object to
          avoid the race condition during cleanup in brick_mux environment

Change-Id: Ifb3d18a854b267333a3a9e39845bfefb83fbc092
Fixes: #1396
Signed-off-by: Mohit Agrawal <moagrawa@redhat.com>
(cherry picked from commit 41b9616435cbdf671805856e487e373060c9455b)
features/shard: optimization over shard lookup in case of prealloc 2020-08-20T13:24:35+00:00 Vinayakswami Hariharmath vharihar@redhat.com 2020-08-06T09:09:59+00:00 c5fc58c8cb01753e2fed173c76aea1e9cc333862 Assume that we are preallocating a VM of size 1TB with a shard block size of 64MB then there will be ~16k shards. This creation happens in 2 steps shard_fallocate() path i.e 1. lookup for the shards if any already present and 2. mknod over those shards do not exist. But in case of fresh creation, we dont have to lookup for all shards which are not present as the the file size will be 0. Through this, we can save lookup on all shards which are not present. This optimization is quite useful in the case of preallocating big vm. Also if the file is already present and the call is to extend it to bigger size then we need not to lookup for non- existent shards. Just lookup preexisting shards, populate the inodes and issue mknod on extended size. Fixes: #1425 Change-Id: I60036fe8302c696e0ca80ff11ab0ef5bcdbd7880 Signed-off-by: Vinayakswami Hariharmath <vharihar@redhat.com> (cherry picked from commit 2ede911d07c6dc07a0f729526ab590ace77341ae) 
Assume that we are preallocating a VM of size 1TB with a shard
block size of 64MB then there will be ~16k shards.

This creation happens in 2 steps shard_fallocate() path i.e

1. lookup for the shards if any already present and
2. mknod over those shards do not exist.

But in case of fresh creation, we dont have to lookup for all
shards which are not present as the the file size will be 0.
Through this, we can save lookup on all shards which are not
present. This optimization is quite useful in the case of
preallocating big vm.

Also if the file is already present and the call is to
extend it to bigger size then we need not to lookup for non-
existent shards. Just lookup preexisting shards, populate
the inodes and issue mknod on extended size.

Fixes: #1425
Change-Id: I60036fe8302c696e0ca80ff11ab0ef5bcdbd7880
Signed-off-by: Vinayakswami Hariharmath <vharihar@redhat.com>
(cherry picked from commit 2ede911d07c6dc07a0f729526ab590ace77341ae)
cluster/ec: Remove stale entries from indices/xattrop folder 2020-08-20T13:21:58+00:00 Ashish Pandey aspandey@redhat.com 2020-07-23T05:37:32+00:00 7b372cdeed876e68293620c25c6821324068fb54 Problem: If a gfid is present in indices/xattrop folder while the file/dir is actaully healthy and all the xattrs are healthy, it causes lot of lookups by shd on an entry which does not need to be healed. This whole process eats up lot of CPU usage without doing meaningful work. Solution: Set trusted.ec.dirty xattr of the entry so that actual heal process happens and at the end of it, during unset of dirty, gfid enrty from indices/xattrop will be removed. Change-Id: Ib1b9377d8dda384bba49523e9ff6ba9f0699cc1b Fixes: #1385 Signed-off-by: Ashish Pandey <aspandey@redhat.com> (cherry picked from commit ba1b0a471dec968633f89c7f790b099fb4ad700d) 
Problem:
If a gfid is present in indices/xattrop folder while
the file/dir is actaully healthy and all the xattrs are healthy,
it causes lot of lookups by shd on an entry which does not need
to be healed.
This whole process eats up lot of CPU usage without doing meaningful
work.

Solution:
Set trusted.ec.dirty xattr of the entry so that actual heal process
happens and at the end of it, during unset of dirty, gfid enrty from
indices/xattrop will be removed.

Change-Id: Ib1b9377d8dda384bba49523e9ff6ba9f0699cc1b
Fixes: #1385
Signed-off-by: Ashish Pandey <aspandey@redhat.com>
(cherry picked from commit ba1b0a471dec968633f89c7f790b099fb4ad700d)
glusterd: Increase buffer length to save multiple hostnames in peer file 2020-08-19T18:11:28+00:00 Mohit Agrawal moagrawa@redhat.com 2020-08-01T03:58:47+00:00 7ff51badda5cbcbaa17f729d1e4ab715c462396a Problem: At the time of handling friend update request glusterd updates peer file and if DNS has returned multiple hostnames for the same IP, glusterd saves all hostnames in peer file.In commit 1fa089e7a2b180e0bdcc1e7e09a63934a2a0c0ef We changed the approach to save all key value pairs in single shot. In case of a buffer is not having space to store the hostnames glusterd writes partial hostname in peer file. Solution: To avoid the failure increase the buffer length Change-Id: Iee969d165333e9c5ba69431d474c541b8f12d442 Fixes: #1407 Signed-off-by: Mohit Agrawal <moagrawa@redhat.com> (cherry picked from commit 6e8e73a06d71382f8f6e3cd83fe72692d19e66ba) 
Problem: At the time of handling friend update request glusterd updates peer
         file and if DNS has returned multiple hostnames for the same IP, glusterd
         saves all hostnames in peer file.In commit 1fa089e7a2b180e0bdcc1e7e09a63934a2a0c0ef
         We changed the approach to save all key value pairs in single shot.
         In case of a buffer is not having space to store the hostnames glusterd
         writes partial hostname in peer file.

Solution: To avoid the failure increase the buffer length

Change-Id: Iee969d165333e9c5ba69431d474c541b8f12d442
Fixes: #1407
Signed-off-by: Mohit Agrawal <moagrawa@redhat.com>
(cherry picked from commit 6e8e73a06d71382f8f6e3cd83fe72692d19e66ba)
cluster/ec: Improve detection of new heals 2020-08-19T18:00:31+00:00 Xavi Hernandez xhernandez@redhat.com 2020-07-02T16:08:52+00:00 269ece312c9fd890c74c46e79de70efe1720752c When EC successfully healed a directory it assumed that maybe other entries inside that directory could have been created, which could require additional heal cycles. For this reason, when the heal happened as part of one index heal iteration, it triggered a new iteration. The problem happened when the directory was healthy, so no new entries were added, but its index entry was not removed for some reason. In this case self-heal started and endless loop healing the same directory continuously, cause high CPU utilization. This patch improves detection of new files added to the heal index so that a new index heal iteration is only triggered if there is new work to do. Change-Id: I2355742b85fbfa6de758bccc5d2e1a283c82b53f Fixes: #1354 Signed-off-by: Xavi Hernandez <xhernandez@redhat.com> 
When EC successfully healed a directory it assumed that maybe other
entries inside that directory could have been created, which could
require additional heal cycles. For this reason, when the heal happened
as part of one index heal iteration, it triggered a new iteration.

The problem happened when the directory was healthy, so no new entries
were added, but its index entry was not removed for some reason. In
this case self-heal started and endless loop healing the same directory
continuously, cause high CPU utilization.

This patch improves detection of new files added to the heal index so
that a new index heal iteration is only triggered if there is new work
to do.

Change-Id: I2355742b85fbfa6de758bccc5d2e1a283c82b53f
Fixes: #1354
Signed-off-by: Xavi Hernandez <xhernandez@redhat.com>
features/shard: Convert shard block indices to uint64 2020-08-19T17:58:23+00:00 Krutika Dhananjay kdhananj@redhat.com 2019-09-12T05:37:10+00:00 19dc7b37fc9f6d6037958e5dd3c0c6a4a993e2af This patch fixes a crash in FOPs that operate on really large sharded files where number of participant shards could sometimes exceed signed int32 max. The patch also adds GF_ASSERTs to ensure that number of participating shards is always greater than 0 for files that do have more than one shard. Change-Id: I354de58796f350eb1aa42fcdf8092ca2e69ccbb6 Fixes: #1348 Signed-off-by: Krutika Dhananjay <kdhananj@redhat.com> (cherry picked from commit cdf01cc47eb2efb427b5855732d9607eec2abc8a) 
This patch fixes a crash in FOPs that operate on really large sharded
files where number of participant shards could sometimes exceed
signed int32 max.

The patch also adds GF_ASSERTs to ensure that number of participating
shards is always greater than 0 for files that do have more than one
shard.

Change-Id: I354de58796f350eb1aa42fcdf8092ca2e69ccbb6
Fixes: #1348
Signed-off-by: Krutika Dhananjay <kdhananj@redhat.com>
(cherry picked from commit cdf01cc47eb2efb427b5855732d9607eec2abc8a)
features/shard: Use fd lookup post file open 2020-08-19T17:55:13+00:00 Vinayakswami Hariharmath vharihar@redhat.com 2020-06-03T13:28:56+00:00 96e7ff7396a8e18ca69d5198be0a8d29bcc37129 Issue: When a process has the open fd and the same file is unlinked in middle of the operations, then file based lookup fails with ENOENT or stale file Solution: When the file already open and fd is available, use fstat to get the file attributes Change-Id: I0e83aee9f11b616dcfe13769ebfcda6742e4e0f4 Fixes: #1281 Signed-off-by: Vinayakswami Hariharmath <vharihar@redhat.com> (cherry picked from commit 71dd19f710b81136f318b3a95ae430971198ee70) 
Issue:
When a process has the open fd and the same file is
unlinked in middle of the operations, then file based
lookup fails with ENOENT or stale file

Solution:
When the file already open and fd is available, use fstat
to get the file attributes

Change-Id: I0e83aee9f11b616dcfe13769ebfcda6742e4e0f4
Fixes: #1281
Signed-off-by: Vinayakswami Hariharmath <vharihar@redhat.com>
(cherry picked from commit 71dd19f710b81136f318b3a95ae430971198ee70)
cluster/afr: Delay post-op for fsync 2020-07-28T13:21:05+00:00 Pranith Kumar K pkarampu@redhat.com 2020-05-29T08:54:53+00:00 e8aedcd40f9f24e5b821e1539275e40ebfccca94 Problem: AFR doesn't delay post-op for fsync fop. For fsync heavy workloads this leads to un-necessary fxattrop/finodelk for every fsync leading to bad performance. Fix: Have delayed post-op for fsync. Add special flag in xdata to indicate that afr shouldn't delay post-op in cases where either the process will terminate or graph-switch would happen. Otherwise it leads to un-necessary heals when the graph-switch/process-termination happens before delayed-post-op completes. Fixes: #1253 Change-Id: I531940d13269a111c49e0510d49514dc169f4577 Signed-off-by: Pranith Kumar K <pkarampu@redhat.com> 
Problem:
AFR doesn't delay post-op for fsync fop. For fsync heavy workloads
this leads to un-necessary fxattrop/finodelk for every fsync leading
to bad performance.

Fix:
Have delayed post-op for fsync. Add special flag in xdata to indicate
that afr shouldn't delay post-op in cases where either the
process will terminate or graph-switch would happen. Otherwise it leads
to un-necessary heals when the graph-switch/process-termination
happens before delayed-post-op completes.

Fixes: #1253
Change-Id: I531940d13269a111c49e0510d49514dc169f4577
Signed-off-by: Pranith Kumar K <pkarampu@redhat.com>