| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
There was a bug in write-behind that allowed a previous completed write
to overwrite the overlapping region of data from a future write.
Suppose we want to send three writes (W1, W2 and W3). W1 and W2 are
sequential, and W3 writes at the same offset of W2:
W2.offset = W3.offset = W1.offset + W1.size
Both W1 and W2 are sent in parallel. W3 is only sent after W2 completes.
So W3 should *always* overwrite the overlapping part of W2.
Suppose write-behind processes the requests from 2 concurrent threads:
Thread 1 Thread 2
<received W1>
<received W2>
wb_enqueue_tempted(W1)
/* W1 is assigned gen X */
wb_enqueue_tempted(W2)
/* W2 is assigned gen X */
wb_process_queue()
__wb_preprocess_winds()
/* W1 and W2 are sequential and all
* other requisites are met to merge
* both requests. */
__wb_collapse_small_writes(W1, W2)
__wb_fulfill_request(W2)
__wb_pick_unwinds() -> W2
/* In this case, since the request is
* already fulfilled, wb_inode->gen
* is not updated. */
wb_do_unwinds()
STACK_UNWIND(W2)
/* The application has received the
* result of W2, so it can send W3. */
<received W3>
wb_enqueue_tempted(W3)
/* W3 is assigned gen X */
wb_process_queue()
/* Here we have W1 (which contains
* the conflicting W2) and W3 with
* same gen, so they are interpreted
* as concurrent writes that do not
* conflict. */
__wb_pick_winds() -> W3
wb_do_winds()
STACK_WIND(W3)
wb_process_queue()
/* Eventually W1 will be
* ready to be sent */
__wb_pick_winds() -> W1
__wb_pick_unwinds() -> W1
/* Here wb_inode->gen is
* incremented. */
wb_do_unwinds()
STACK_UNWIND(W1)
wb_do_winds()
STACK_WIND(W1)
So, as we can see, W3 is sent before W1, which shouldn't happen.
The problem is that wb_inode->gen is only incremented for requests that
have not been fulfilled but, after a merge, the request is marked as
fulfilled even though it has not been sent to the brick. This allows
that future requests are assigned to the same generation, which could
be internally reordered.
Solution:
Increment wb_inode->gen before any unwind, even if it's for a fulfilled
request.
Special thanks to Stefan Ring for writing a reproducer that has been
crucial to identify the issue.
Change-Id: Id4ab0f294a09aca9a863ecaeef8856474662ab45
Signed-off-by: Xavi Hernandez <xhernandez@redhat.com>
Fixes: #884
|
| |
|
|
|
| |
Change-Id: Ia84cc24c8924e6d22d02ac15f611c10e26db99b4
Signed-off-by: Nigel Babu <nigelb@redhat.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
There is a race condition which is causing the test to fail. For lack
of bandwidth I am moving this test to BAD, though clearly there is
some issue with codebase.
BUG: 1337777
Change-Id: If4f3eff8a5985f37a4dee65d2df29fa7b6bda7ae
Signed-off-by: Raghavendra G <rgowdapp@redhat.com>
Reviewed-on: http://review.gluster.org/14443
Smoke: Gluster Build System <jenkins@build.gluster.com>
NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org>
CentOS-regression: Gluster Build System <jenkins@build.gluster.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* Don't mark the request with a fake EIO after a short write.
* retry the remaining buffer at least once before unwinding reply to
application. This way we capture correct error from backend (ENOSPC,
EDQUOT etc).
Thanks to "Vijaikumar Mallikarjuna"<vmallika@redhat.com> for the test
script.
Change-Id: I73a18b39b661a7424db1a7855a980469a51da8f9
BUG: 1292020
Signed-off-by: Raghavendra G <rgowdapp@redhat.com>
Reviewed-on: http://review.gluster.org/13438
Smoke: Gluster Build System <jenkins@build.gluster.com>
NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org>
CentOS-regression: Gluster Build System <jenkins@build.gluster.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
1. When sync fails, the cached-write is still preserved unless there
is a flush/fsync waiting on it.
2. When a sync fails and there is a flush/fsync waiting on the
cached-write, the cache is thrown away and no further retries will
be made. In other words flush/fsync act as barriers for all the
previous writes. The behaviour of fsync acting as a barrier is
controlled by an option (see below for details). All previous
writes are either successfully synced to backend or forgotten in
case of an error. Without such barrier fop (especially flush which
is issued prior to a close), we end up retrying for ever even after
fd is closed.
3. If a fop is waiting on cached-write and syncing to backend fails,
the waiting fop is failed.
4. sync failures when no fop is waiting are ignored and are not
propagated to application. For eg.,
a. first attempt of sync of a cached-write w1 fails
b. second attempt of sync of w1 succeeds
If there are no fops dependent on w1 are issued b/w a and b,
application won't know about failure encountered in a.
5. The effect of repeated sync failures is that, there will be no
cache for future writes and they cannot be written behind.
fsync as a barrier and resync of cached writes post fsync failure:
==================================================================
Whether to keep retrying failed syncs post fsync is controlled by an
option "resync-failed-syncs-after-fsync". By default, this option is
set to "off".
If sync of "cached-writes issued before fsync" (to backend) fails,
this option configures whether to retry syncing them after fsync or
forget them. If set to on, cached-writes are retried till a "flush"
fop (or a successful sync) on sync failures. fsync itself is failed
irrespective of the value of this option, when there is a sync failure
of any cached-writes issued before fsync.
Change-Id: I6097c9257bfb9ee5b15616fbe6a0576ae9af369a
Signed-off-by: Raghavendra G <rgowdapp@redhat.com>
BUG: 1279730
Reviewed-on: http://review.gluster.org/12594
|
|
|
There are around 300 regression tests, 250 being in tests/bugs. Running
partial set of tests/bugs is not easy because this is a flat directory
with almost all tests inside.
It would be valuable to make partial test/bugs easier, and allow the use
of mulitple build hosts for a single commit, each running a subset of
the tests for a quicker result.
Additional changes made:
- correct the include path for *.rc shell libraries and *.py utils
- make the testcases pass checkpatch
- arequal-checksum in afr/self-heal.t was never executed, now it is
- include.rc now complains loudly if it fails to find env.rc
Change-Id: I26ffd067e9853d3be1fd63b2f37d8aa0fd1b4fea
BUG: 1178685
Reported-by: Emmanuel Dreyfus <manu@netbsd.org>
Reported-by: Atin Mukherjee <amukherj@redhat.com>
URL: http://www.gluster.org/pipermail/gluster-devel/2014-December/043414.html
Signed-off-by: Niels de Vos <ndevos@redhat.com>
Reviewed-on: http://review.gluster.org/9353
Reviewed-by: Kaleb KEITHLEY <kkeithle@redhat.com>
Reviewed-by: Emmanuel Dreyfus <manu@netbsd.org>
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Vijay Bellur <vbellur@redhat.com>
|
