<feed xmlns='http://www.w3.org/2005/Atom'>
<title>glusterfs.git/xlators/performance/write-behind, branch v5.12</title>
<subtitle></subtitle>
<link rel='alternate' type='text/html' href='http://git.gluster.org/cgit/glusterfs.git/'/>
<entry>
<title>performance/write-behind: remove request from wip list in wb_writev_cbk</title>
<updated>2019-05-08T14:07:56+00:00</updated>
<author>
<name>Raghavendra G</name>
<email>rgowdapp@redhat.com</email>
</author>
<published>2019-05-03T04:44:48+00:00</published>
<link rel='alternate' type='text/html' href='http://git.gluster.org/cgit/glusterfs.git/commit/?id=09cb7406c28052c4643b2cb6e0ab986331b9576b'/>
<id>09cb7406c28052c4643b2cb6e0ab986331b9576b</id>
<content type='text'>
There is a race in the way O_DIRECT writes are handled. Assume two
overlapping write requests w1 and w2.

* w1 is issued and is in wb_inode-&gt;wip queue as the response is still
  pending from bricks. Also wb_request_unref in wb_do_winds is not yet
  invoked.

       list_for_each_entry_safe (req, tmp, tasks, winds) {
		list_del_init (&amp;req-&gt;winds);

                if (req-&gt;op_ret == -1) {
			call_unwind_error_keep_stub (req-&gt;stub, req-&gt;op_ret,
		                                     req-&gt;op_errno);
                } else {
                        call_resume_keep_stub (req-&gt;stub);
		}

                wb_request_unref (req);
        }

* w2 is issued and wb_process_queue is invoked. w2 is not picked up
  for winding as w1 is still in wb_inode-&gt;wip. w1 is added to todo
  list and wb_writev for w2 returns.

* response to w1 is received and invokes wb_request_unref. Assume
  wb_request_unref in wb_do_winds (see point 1) is not invoked
  yet. Since there is one more refcount, wb_request_unref in
  wb_writev_cbk of w1 doesn't remove w1 from wip.

* wb_process_queue is invoked as part of wb_writev_cbk of w1. But, it
  fails to wind w2 as w1 is still in wip.

* wb_requet_unref is invoked on w1 as part of wb_do_winds. w1 is
  removed from all queues including w1.

* After this point there is no invocation of wb_process_queue unless
  new request is issued from application causing w2 to be hung till
  the next request.

This bug is similar to bz 1626780 and bz 1379655.

Change-Id: Iaa47437613591699d4c8ad18bc0b32de6affcc31
Signed-off-by: Raghavendra G &lt;rgowdapp@redhat.com&gt;
Fixes: bz#1707198
(cherry picked from commit 6454132342c0b549365d92bcf3572ecd914f7fa8)
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
There is a race in the way O_DIRECT writes are handled. Assume two
overlapping write requests w1 and w2.

* w1 is issued and is in wb_inode-&gt;wip queue as the response is still
  pending from bricks. Also wb_request_unref in wb_do_winds is not yet
  invoked.

       list_for_each_entry_safe (req, tmp, tasks, winds) {
		list_del_init (&amp;req-&gt;winds);

                if (req-&gt;op_ret == -1) {
			call_unwind_error_keep_stub (req-&gt;stub, req-&gt;op_ret,
		                                     req-&gt;op_errno);
                } else {
                        call_resume_keep_stub (req-&gt;stub);
		}

                wb_request_unref (req);
        }

* w2 is issued and wb_process_queue is invoked. w2 is not picked up
  for winding as w1 is still in wb_inode-&gt;wip. w1 is added to todo
  list and wb_writev for w2 returns.

* response to w1 is received and invokes wb_request_unref. Assume
  wb_request_unref in wb_do_winds (see point 1) is not invoked
  yet. Since there is one more refcount, wb_request_unref in
  wb_writev_cbk of w1 doesn't remove w1 from wip.

* wb_process_queue is invoked as part of wb_writev_cbk of w1. But, it
  fails to wind w2 as w1 is still in wip.

* wb_requet_unref is invoked on w1 as part of wb_do_winds. w1 is
  removed from all queues including w1.

* After this point there is no invocation of wb_process_queue unless
  new request is issued from application causing w2 to be hung till
  the next request.

This bug is similar to bz 1626780 and bz 1379655.

Change-Id: Iaa47437613591699d4c8ad18bc0b32de6affcc31
Signed-off-by: Raghavendra G &lt;rgowdapp@redhat.com&gt;
Fixes: bz#1707198
(cherry picked from commit 6454132342c0b549365d92bcf3572ecd914f7fa8)
</pre>
</div>
</content>
</entry>
<entry>
<title>performance/write-behind: fix use-after-free in readdirp</title>
<updated>2019-02-22T14:50:00+00:00</updated>
<author>
<name>Raghavendra Gowdappa</name>
<email>rgowdapp@redhat.com</email>
</author>
<published>2019-02-13T11:38:14+00:00</published>
<link rel='alternate' type='text/html' href='http://git.gluster.org/cgit/glusterfs.git/commit/?id=3c1368fda17b71a95edf1f4fe102e15f64820e3e'/>
<id>3c1368fda17b71a95edf1f4fe102e15f64820e3e</id>
<content type='text'>
Two issues were found:
1. in wb_readdirp_cbk, inode should unrefed after wb_inode is
unlocked. Otherwise, inode and hence the context wb_inode can be freed
by the type we try to unlock wb_inode
2. wb_readdirp_mark_end iterates over a list of wb_inodes of children
of a directory. But inodes could've been freed and hence the list
might be corrupted. To fix take a reference on inode before adding it
to invalidate_list of parent.

Change-Id: I911b0e0b2060f7f41ded0b05db11af6f9b7c09c5
Signed-off-by: Raghavendra Gowdappa &lt;rgowdapp@redhat.com&gt;
Updates: bz#1671556
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
Two issues were found:
1. in wb_readdirp_cbk, inode should unrefed after wb_inode is
unlocked. Otherwise, inode and hence the context wb_inode can be freed
by the type we try to unlock wb_inode
2. wb_readdirp_mark_end iterates over a list of wb_inodes of children
of a directory. But inodes could've been freed and hence the list
might be corrupted. To fix take a reference on inode before adding it
to invalidate_list of parent.

Change-Id: I911b0e0b2060f7f41ded0b05db11af6f9b7c09c5
Signed-off-by: Raghavendra Gowdappa &lt;rgowdapp@redhat.com&gt;
Updates: bz#1671556
</pre>
</div>
</content>
</entry>
<entry>
<title>performance/write-behind: handle call-stub leaks</title>
<updated>2019-02-22T14:49:13+00:00</updated>
<author>
<name>Raghavendra Gowdappa</name>
<email>rgowdapp@redhat.com</email>
</author>
<published>2019-02-19T02:34:52+00:00</published>
<link rel='alternate' type='text/html' href='http://git.gluster.org/cgit/glusterfs.git/commit/?id=ceee95dbc3c80d52134ceaee22e17c9f1c861300'/>
<id>ceee95dbc3c80d52134ceaee22e17c9f1c861300</id>
<content type='text'>
Change-Id: I7be9a5f48dcad1b136c479c58b1dca1e0488166d
Signed-off-by: Raghavendra Gowdappa &lt;rgowdapp@redhat.com&gt;
Fixes: bz#1671556
(cherry picked from commit 6175cb10cd5f59f3c7ae4100bc78f359b68ca3e9)
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
Change-Id: I7be9a5f48dcad1b136c479c58b1dca1e0488166d
Signed-off-by: Raghavendra Gowdappa &lt;rgowdapp@redhat.com&gt;
Fixes: bz#1671556
(cherry picked from commit 6175cb10cd5f59f3c7ae4100bc78f359b68ca3e9)
</pre>
</div>
</content>
</entry>
<entry>
<title>all: fix the format string exceptions</title>
<updated>2018-11-09T14:03:02+00:00</updated>
<author>
<name>Amar Tumballi</name>
<email>amarts@redhat.com</email>
</author>
<published>2018-11-06T17:17:41+00:00</published>
<link rel='alternate' type='text/html' href='http://git.gluster.org/cgit/glusterfs.git/commit/?id=83304fedb464fe3f97db662ce3e07bd948b7b7d9'/>
<id>83304fedb464fe3f97db662ce3e07bd948b7b7d9</id>
<content type='text'>
Currently, there are possibilities in few places, where a user-controlled
(like filename, program parameter etc) string can be passed as 'fmt' for
printf(), which can lead to segfault, if the user's string contains '%s',
'%d' in it.

While fixing it, makes sense to make the explicit check for such issues
across the codebase, by making the format call properly.

Fixes: CVE-2018-14661

Fixes: bz#1647666
Change-Id: Ib547293f2d9eb618594cbff0df3b9c800e88bde4
Signed-off-by: Amar Tumballi &lt;amarts@redhat.com&gt;
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
Currently, there are possibilities in few places, where a user-controlled
(like filename, program parameter etc) string can be passed as 'fmt' for
printf(), which can lead to segfault, if the user's string contains '%s',
'%d' in it.

While fixing it, makes sense to make the explicit check for such issues
across the codebase, by making the format call properly.

Fixes: CVE-2018-14661

Fixes: bz#1647666
Change-Id: Ib547293f2d9eb618594cbff0df3b9c800e88bde4
Signed-off-by: Amar Tumballi &lt;amarts@redhat.com&gt;
</pre>
</div>
</content>
</entry>
<entry>
<title>Land part 2 of clang-format changes</title>
<updated>2018-09-12T12:22:45+00:00</updated>
<author>
<name>Gluster Ant</name>
<email>bugzilla-bot@gluster.org</email>
</author>
<published>2018-09-12T12:22:45+00:00</published>
<link rel='alternate' type='text/html' href='http://git.gluster.org/cgit/glusterfs.git/commit/?id=e16868dede6455cab644805af6fe1ac312775e13'/>
<id>e16868dede6455cab644805af6fe1ac312775e13</id>
<content type='text'>
Change-Id: Ia84cc24c8924e6d22d02ac15f611c10e26db99b4
Signed-off-by: Nigel Babu &lt;nigelb@redhat.com&gt;
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
Change-Id: Ia84cc24c8924e6d22d02ac15f611c10e26db99b4
Signed-off-by: Nigel Babu &lt;nigelb@redhat.com&gt;
</pre>
</div>
</content>
</entry>
<entry>
<title>Land clang-format changes</title>
<updated>2018-09-12T11:52:48+00:00</updated>
<author>
<name>Gluster Ant</name>
<email>bugzilla-bot@gluster.org</email>
</author>
<published>2018-09-12T11:52:48+00:00</published>
<link rel='alternate' type='text/html' href='http://git.gluster.org/cgit/glusterfs.git/commit/?id=45a71c0548b6fd2c757aa2e7b7671a1411948894'/>
<id>45a71c0548b6fd2c757aa2e7b7671a1411948894</id>
<content type='text'>
Change-Id: I6f5d8140a06f3c1b2d196849299f8d483028d33b
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
Change-Id: I6f5d8140a06f3c1b2d196849299f8d483028d33b
</pre>
</div>
</content>
</entry>
<entry>
<title>performance/write-behind: remove the request from wip queue in wb_fulfill_request</title>
<updated>2018-09-11T03:29:24+00:00</updated>
<author>
<name>Raghavendra G</name>
<email>rgowdapp@redhat.com</email>
</author>
<published>2018-09-08T14:23:07+00:00</published>
<link rel='alternate' type='text/html' href='http://git.gluster.org/cgit/glusterfs.git/commit/?id=5537a8f050a07d8f1c32138d787331d61695ce38'/>
<id>5537a8f050a07d8f1c32138d787331d61695ce38</id>
<content type='text'>
The bug is very similar to bz 1379655 and the fix too very similar to
commit a8b2a981881221925bb5edfe7bb65b25ad855c04.

Before this patch, a request is removed from wip queue only when ref
count of request hits 0. Though, wb_fulfill_request does an unref,
it need not be the last unref and hence the request may survive in
wip queue till the last unref. Let,

T1: the time at which wb_fulfill_request is invoked
T2: the time at which last unref is done on request

Let's consider a case of T2 &gt; T1. In the time window between T1 and
T2, any other request (waiter) conflicting with request in liability
queue (blocker - basically a write which has been lied) is blocked
from winding. If T2 happens to be when wb_do_unwinds is invoked, no
further processing of request list happens and "waiter" would get
blocked forever. An example imaginary sequence of events is given
below:

1. A write request w1 is picked up for winding in __wb_pick_winds
   and w1 is moved to wip queue. Let's call this
   invocation of wb_process_queue by wb_writev as PQ1. Note w1 is not
   unwound.

2. A dependent write (w2) hits write-behind and is unwound followed by
   a flush (f1) request. Since the liability queue
   of inode is not empty, w2 and f1 are not picked for unwinding. Let's call
   the invocation of wb_process_queue by wb_flush as PQ2. Note that
   invocation of wb_process_queue by w2 doesn't wind w2 instead
   unwinds it after which we hit PQ2

3. PQ2 continues and picks w1 for fulfilling and invokes
   wb_fulfill. As part of successful wb_fulfill_cbk,
   wb_fulfill_request (w1) is invoked. But, w1 is not freed (and hence
   not removed from wip queue) as w1 is not unwound _yet_ and a
   ref remains (PQ1 has not invoked wb_do_unwinds _yet_).

4. wb_fulfill_cbk (triggered by PQ2) invokes a wb_process_queue (let's
   say PQ3). w2 is not picked up for winding in PQ3 as w1 is still in wip
   queue. At this time, PQ2 and PQ3 are complete.

5. PQ1 continues, unwinds w1 and does last unref on w1 and w1 is freed
   (and removed from wip queue). Since PQ1 didn't invoke
   wb_fulfill on any other write requests, there won't be any future
   codepaths that would invoke wb_process_queue and w2 is stuck
   forever. This will prevent f2 too and hence close syscall is hung

With this fix, w1 is removed from liability queue in step 3 above and
PQ3 winds w2 in step 4 (as there are no requests conflicting with w2
in liability queue during execution of PQ3). Once w2 is complete, f1
is resumed.

Change-Id: Ia972fad0858dc4abccdc1227cb4d880f85b3b89b
Signed-off-by: Raghavendra G &lt;rgowdapp@redhat.com&gt;
Fixes: bz#1626787
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
The bug is very similar to bz 1379655 and the fix too very similar to
commit a8b2a981881221925bb5edfe7bb65b25ad855c04.

Before this patch, a request is removed from wip queue only when ref
count of request hits 0. Though, wb_fulfill_request does an unref,
it need not be the last unref and hence the request may survive in
wip queue till the last unref. Let,

T1: the time at which wb_fulfill_request is invoked
T2: the time at which last unref is done on request

Let's consider a case of T2 &gt; T1. In the time window between T1 and
T2, any other request (waiter) conflicting with request in liability
queue (blocker - basically a write which has been lied) is blocked
from winding. If T2 happens to be when wb_do_unwinds is invoked, no
further processing of request list happens and "waiter" would get
blocked forever. An example imaginary sequence of events is given
below:

1. A write request w1 is picked up for winding in __wb_pick_winds
   and w1 is moved to wip queue. Let's call this
   invocation of wb_process_queue by wb_writev as PQ1. Note w1 is not
   unwound.

2. A dependent write (w2) hits write-behind and is unwound followed by
   a flush (f1) request. Since the liability queue
   of inode is not empty, w2 and f1 are not picked for unwinding. Let's call
   the invocation of wb_process_queue by wb_flush as PQ2. Note that
   invocation of wb_process_queue by w2 doesn't wind w2 instead
   unwinds it after which we hit PQ2

3. PQ2 continues and picks w1 for fulfilling and invokes
   wb_fulfill. As part of successful wb_fulfill_cbk,
   wb_fulfill_request (w1) is invoked. But, w1 is not freed (and hence
   not removed from wip queue) as w1 is not unwound _yet_ and a
   ref remains (PQ1 has not invoked wb_do_unwinds _yet_).

4. wb_fulfill_cbk (triggered by PQ2) invokes a wb_process_queue (let's
   say PQ3). w2 is not picked up for winding in PQ3 as w1 is still in wip
   queue. At this time, PQ2 and PQ3 are complete.

5. PQ1 continues, unwinds w1 and does last unref on w1 and w1 is freed
   (and removed from wip queue). Since PQ1 didn't invoke
   wb_fulfill on any other write requests, there won't be any future
   codepaths that would invoke wb_process_queue and w2 is stuck
   forever. This will prevent f2 too and hence close syscall is hung

With this fix, w1 is removed from liability queue in step 3 above and
PQ3 winds w2 in step 4 (as there are no requests conflicting with w2
in liability queue during execution of PQ3). Once w2 is complete, f1
is resumed.

Change-Id: Ia972fad0858dc4abccdc1227cb4d880f85b3b89b
Signed-off-by: Raghavendra G &lt;rgowdapp@redhat.com&gt;
Fixes: bz#1626787
</pre>
</div>
</content>
</entry>
<entry>
<title>performance/write-behind: fix fulfill and readdirp race</title>
<updated>2018-08-23T15:40:57+00:00</updated>
<author>
<name>Raghavendra G</name>
<email>rgowdapp@redhat.com</email>
</author>
<published>2018-08-21T04:14:15+00:00</published>
<link rel='alternate' type='text/html' href='http://git.gluster.org/cgit/glusterfs.git/commit/?id=370f05546eeedab394ca0d333a6ca6637757f1e3'/>
<id>370f05546eeedab394ca0d333a6ca6637757f1e3</id>
<content type='text'>
Current invalidation of stats in wb_readdirp_cbk is prone to races. As
the deleted comment explains,

&lt;snip&gt;
    We cannot guarantee integrity of entry-&gt;d_stat as there are cached
    writes. The stat is most likely stale as it doesn't account the cached
    writes. However, checking for non-empty liability list here is not a
    fool-proof solution as there can be races like,
      1. readdirp is successful on posix
      2. sync of cached write is successful on posix
      3. write-behind received sync response and removed the request from
         liability queue
      4. readdirp response is processed at write-behind.

    In the above scenario, stat for the file is sent back in readdirp
    response but it is stale.
&lt;/snip&gt;

The fix is to mark readdirp sessions (tracked in this patch by
non-zero value of "readdirps" on parent inode) and if fulfill
completes when one or more readdirp sessions are in progress, mark the
inode so that wb_readdirp_cbk doesn't send iatts for that in inode in
readdirp response. Note that wb_readdirp_cbk already checks for
presence of a non-empty liability queue and invalidates iatt. Since
the only way a liability queue can shrink is by fulfilling requests in
liability queue, wb_fulfill_cbk indicates wb_readdirp_cbk that a
potential race could've happened b/w readdirp and fulfill.

Change-Id: I12d167bf450648baa64be1cbe1ca0fddf5379521
Signed-off-by: Raghavendra G &lt;rgowdapp@redhat.com&gt;
updates: bz#1512691
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
Current invalidation of stats in wb_readdirp_cbk is prone to races. As
the deleted comment explains,

&lt;snip&gt;
    We cannot guarantee integrity of entry-&gt;d_stat as there are cached
    writes. The stat is most likely stale as it doesn't account the cached
    writes. However, checking for non-empty liability list here is not a
    fool-proof solution as there can be races like,
      1. readdirp is successful on posix
      2. sync of cached write is successful on posix
      3. write-behind received sync response and removed the request from
         liability queue
      4. readdirp response is processed at write-behind.

    In the above scenario, stat for the file is sent back in readdirp
    response but it is stale.
&lt;/snip&gt;

The fix is to mark readdirp sessions (tracked in this patch by
non-zero value of "readdirps" on parent inode) and if fulfill
completes when one or more readdirp sessions are in progress, mark the
inode so that wb_readdirp_cbk doesn't send iatts for that in inode in
readdirp response. Note that wb_readdirp_cbk already checks for
presence of a non-empty liability queue and invalidates iatt. Since
the only way a liability queue can shrink is by fulfilling requests in
liability queue, wb_fulfill_cbk indicates wb_readdirp_cbk that a
potential race could've happened b/w readdirp and fulfill.

Change-Id: I12d167bf450648baa64be1cbe1ca0fddf5379521
Signed-off-by: Raghavendra G &lt;rgowdapp@redhat.com&gt;
updates: bz#1512691
</pre>
</div>
</content>
</entry>
<entry>
<title>Revert "performance/write-behind: better invalidation in readdirp"</title>
<updated>2018-08-21T11:36:51+00:00</updated>
<author>
<name>Raghavendra G</name>
<email>rgowdapp@redhat.com</email>
</author>
<published>2018-08-20T11:08:41+00:00</published>
<link rel='alternate' type='text/html' href='http://git.gluster.org/cgit/glusterfs.git/commit/?id=0eac8026f880bf8bf7e05530de16a06ba3e8a40c'/>
<id>0eac8026f880bf8bf7e05530de16a06ba3e8a40c</id>
<content type='text'>
This reverts commit 4d3c62e71f3250f10aa0344085a5ec2d45458d5c.

Traversing all children of a directory in wb_readdirp caused
significant performance regression. Hence reverting this patch

Change-Id: I6c3b6cee2dd2aca41d49fe55ecdc6262e7cc5f34
updates: bz#1512691
Signed-off-by: Raghavendra G &lt;rgowdapp@redhat.com&gt;
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
This reverts commit 4d3c62e71f3250f10aa0344085a5ec2d45458d5c.

Traversing all children of a directory in wb_readdirp caused
significant performance regression. Hence reverting this patch

Change-Id: I6c3b6cee2dd2aca41d49fe55ecdc6262e7cc5f34
updates: bz#1512691
Signed-off-by: Raghavendra G &lt;rgowdapp@redhat.com&gt;
</pre>
</div>
</content>
</entry>
<entry>
<title>write-behind: coverity fixes</title>
<updated>2018-08-20T15:49:26+00:00</updated>
<author>
<name>Bhumika Goyal</name>
<email>bgoyal@redhat.com</email>
</author>
<published>2018-08-20T10:10:06+00:00</published>
<link rel='alternate' type='text/html' href='http://git.gluster.org/cgit/glusterfs.git/commit/?id=a3a3b8ec67fa1609e06b1d919675e3df74d531e5'/>
<id>a3a3b8ec67fa1609e06b1d919675e3df74d531e5</id>
<content type='text'>
Fixes CID: 1124360 1291740 1370918

Change-Id: I008c7ade8f9809d040f42f6d3e9af70fff2f3dc6
updates: bz#789278
Signed-off-by: Bhumika Goyal &lt;bgoyal@redhat.com&gt;
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
Fixes CID: 1124360 1291740 1370918

Change-Id: I008c7ade8f9809d040f42f6d3e9af70fff2f3dc6
updates: bz#789278
Signed-off-by: Bhumika Goyal &lt;bgoyal@redhat.com&gt;
</pre>
</div>
</content>
</entry>
</feed>
