glusterfs.git/xlators/features/bit-rot, branch v3.7.2

features/bitrot: fix fd leak in truncate (stub)

2015-06-19T08:37:54+00:00

    Backport of http://review.gluster.org/#/c/11077

The need to perform object versioning in the truncate() code path
required an fd to reuse existing versioning infrastructure that's
used by fd based operations (such as writev(), ftruncate(), etc..).

This tempted the use of anonymous fd which was never ever unref()'d
after use resulting in fd and/or memory leak depending on the code
path taken. Versioning resulted in a dangling file descriptor left
open in the filesystem effecting the signing process of a given
object (no release() would be trigerred, hence no signing would be
performed). On the other hand, cases where the object need not be
versioned, the anonymous fd in still ref()'d resulting in memory
leak (NOTE: there's no "dangling" file descriptor in this case).

Change-Id: I29c3d2af9bbc5cd4b8ddf38954080e3c7a44ba61
BUG: 1232179
Signed-off-by: Venky Shankar 
Reviewed-on: http://review.gluster.org/11300
Tested-by: Gluster Build System 
Tested-by: NetBSD Build System 
Reviewed-by: Raghavendra Bhat

features/bitrot: tuanble object signing waiting time value for bitrot

2015-06-17T16:38:59+00:00

 Currently bitrot using 120 second waiting time for object to be signed
 after all fop's released. This signing waiting time value should be tunable.

 Command for changing the signing waiting time will be
 #gluster volume bitrot  signing-time 

Change-Id: I89f3121564c1bbd0825f60aae6147413a2fbd798
BUG: 1231832
Signed-off-by: Gaurav Kumar Garg 
Signed-off-by: Venky Shankar 
Reviewed-on: http://review.gluster.org/11105
(cherry picked from commit 554fa0c1315d0b4b78ba35a2d332d7ac0fd07d48)
Reviewed-on: http://review.gluster.org/11235
Tested-by: Gluster Build System

features/bit-rot-stub: implement mknod fop

2015-06-01T02:33:21+00:00

    Backport of http://review.gluster.org/10790

With the absence of mknod() fop implementation in bitrot stub,
further operations that trigger versioning resulted in crashes
as they expect the inode context to be valid. Therefore, this
patch implements mknod() following similar simantics to fops
such as create().

Furthermore, bitrot stub test C program is fixed to stop lying
and validate obj versions according to the versioning protocol.

Change-Id: If76f252577445d1851d6c13c7e969e864e2183ef
BUG: 1226139
Original-Author: Raghavendra Bhat 
Signed-off-by: Venky Shankar 
Reviewed-on: http://review.gluster.org/10987
Tested-by: NetBSD Build System 
Tested-by: Gluster Build System

features/bitrot: serialize versioning

2015-05-31T12:16:10+00:00

    Backport of http://review.gluster.org/10832

Current signing interface (fsetxattr()) had couple of issues:

One, a signing request (by bitrot daemon) is denied if the version
against which an object is to be signed is unequal to the current
version of the object (cases where another subsequent modification
increments the version). Such request(s) are rejected with EINVAL
sent back to the signer resulting in a bunch of errors (in logs)
reported by bitrot daemon. Although, the object would be eventaully
signed with the version matching the current version, the "lagging"
request should be correctly handled.

Two, more than one signing request could race against each other
with the object getting signed with a version depending on which
request ended up last in the race. Although harmless to some extent,
such a case could end up marking the object's signature as stale
for infinity (if the object is *never* touched) thereby resulting
in scrubber skipping the object during verification.

This patch fixes these issues by ordering signing request(s) and
fixing version comparison checks at the time of signing.

Change-Id: I9fa83dfa3be664ba4db61d7f2edc408f4bde77dd
BUG: 1224650
Signed-off-by: Venky Shankar 
Tested-by: Gluster Build System 
Reviewed-on: http://review.gluster.org/10900
Tested-by: NetBSD Build System

features/bitrot: refactor brick connection logic

2015-05-31T04:15:04+00:00

    Backport of http://review.gluster.org/10763

Brick connection was bloated (and not implemented efficiently) with
calls which were not required to be called under lock. This resulted
in starvation of lock by critical code paths. This eventally did not
scale when the number of bricks per volume increases (add-brick and
the likes).

Also, this patch cleans up some of the weird reconnection logic that
added more to the starvation of resources and cleans up uncontrolled
growing of log files.

Change-Id: I05e737f2a9742944a4a543327d167de2489236a4
BUG: 1226146
Original-Author: Raghavendra Bhat 
Signed-off-by: Venky Shankar 
Reviewed-on: http://review.gluster.org/10986
Tested-by: Gluster Build System 
Tested-by: NetBSD Build System

features/bitrot: reimplement scrubbing frequency

2015-05-31T04:14:10+00:00

This patch reimplments existing scrub-frequency mechanism used
to schedule scrubber runs. Existing mechanism uses periodic
sleeps (waking up periodically on minimum granularity) and
performing a number of tracking checks based on counters and
sleep times. This patch does away with all the nifty counters
and uses timer-wheel to schedule scrub runs.

Scheduling changes are peformed by merely calculating the new
expiry time and calling mod_timer() [mod_timer_pending() in
some cases] making the code more debuggable and easier to
follow. This also introduces "hourly" scrubbing tunable as an
aid for testing scrubbing during development/testing cycle.

One could also implement on-demand scrubbing with ease: by
invoking mod_timer() with an expiry of one (1) second, thereby
scheduling a scrub run the very next second.

Change-Id: I6c7c5f0c6c9f886bf574d88c04cde14b76e60a8b
BUG: 1224647
Signed-off-by: Venky Shankar 
Reviewed-on: http://review.gluster.org/10902
Tested-by: Gluster Build System 
Tested-by: NetBSD Build System

features/bitrot: stub improvements and fixes

2015-05-31T04:12:48+00:00

This patch refactors the signing trigger mechanism used by bitrot
daemon as a "catch up" meachanism to sign files which _missed_
signing on the last run either due to bitrot being disabled and
enabled again or if bitrot is enabled for a volume with existing
data.

Existing implementation relies on overloading writev() to trigger
signing which just by the looks sounded dangerous and I hated it
to the core. This change moves all that business to the setxattr
interface thereby keeping the writev path strictly for client
IO.

Why not use IPC fop to trigger signing?
There's a need to access the object's inode to perform various
maintainance operations. inode is not _directly_ accessible in
the IPC fop (although, it can be found via inode_grep() for the
object's GFID - the inode just needs to be pinned in memory,
which is the case if there's an active fd on the inode). This
patch relies on good old technique of overloading fsetxattr()
to do the job instead of using IPC fop.

There are some pretty nice cleanups along the lines of memory
deallocations, unncessary allocations and redundant ref()ing
of structures (such as fd's) provided by this patch. All in
all - much improved code navigation.

Change-Id: Id93fe90b1618802d1a95a5072517dac342b96cb8
BUG: 1225709
Signed-off-by: Venky Shankar 
Reviewed-on: http://review.gluster.org/10953
Tested-by: Gluster Build System 
Tested-by: NetBSD Build System

features/bit-rot-stub: versioning of objects in write/truncate fop instead of open

2015-05-10T15:14:33+00:00

* This patch brings in the changes where object versioning is done in write and
  truncate fops instead of tracking them in open and create fops. This model
  works for both regular and anonymous fds. It also removes the race associated
  with open calls, create and lookups.

  This patch follows the below method for object versioning and notifications:

  Before sending writev on the fd, increase the ongoing
  version first. This makes anonymous fd write similar to the regular
  fd write by having the ongoing version increased before doing the
  write.

  Do following steps to do versioning:
  1) For anonymous fds set the fd context (so that release is invoked) and add
     the fd context to the list maintained in the inode context.
     For regular fds the above think would have been done in open itself.
  2) Increase the on-disk ongoing version
  3) Increase the in memory ongoing version and mark inode as non-dirty
  3) Once versioning is successfully done send write operation. If
     versioning fails, then fail the write fop.
  5) In writev_cbk mark inode as modified.

> Change-Id: I7104391bbe076d8fc49b68745d2ec29a6e92476c
> BUG: 1207979
> Signed-off-by: Raghavendra Bhat 
> Reviewed-on: http://review.gluster.org/10233
> Tested-by: Gluster Build System 
> Reviewed-by: Vijay Bellur 

Change-Id: I4bb86989b5fab02b9ed2950798b1a80e566f1024
BUG: 1220041
Signed-off-by: Raghavendra Bhat 
Reviewed-on: http://review.gluster.org/10722
Reviewed-by: Gaurav Kumar Garg 
Tested-by: NetBSD Build System
Tested-by: Gluster Build System

features/bitrot: scrubber should crawl based on the scrubber frequency value

2015-05-10T13:03:46+00:00

Currently scrubber is crawling all the files continuously. It should
crawl files based on the scrubber frequency which user have set.

By default scrubber crawling frequency value will be biweekly.

Change-Id: I5762a92c1e700134cfe4283d1f631904adbfe31d
BUG: 1220068
Signed-off-by: Gaurav Kumar Garg 
Reviewed-on: http://review.gluster.org/10739
Tested-by: NetBSD Build System
Tested-by: Gluster Build System 
Reviewed-by: Niels de Vos

features/bitrot: Scrubber pause/resume

2015-05-10T12:29:45+00:00

With logical scan/scrub split, pausing filesystem scrubber is an
override to the thread throttling mechanism, which effectively
throttles "down" number of scrubber threads to zero. This causes
scanner to wait until threads are spawned again (when resumed)
thereby continuing where it left off (since the file tree walk
stack is effectively preserved when the main scanner thread
is waiting for scrubbers to consume scanned entries).

The only catch is when scrubber daemon restarts: file tree walk
stack is lost and scrubbing initiates from root. This is probably
OK for now (can be changed later to persist parent directory
information before entering pause state).

> Change-Id: I5109a749b7fccd0f5367765078f46e6522dd32a1
> BUG: 1208131
> Signed-off-by: Venky Shankar 
> Reviewed-on: http://review.gluster.org/10521
> Reviewed-by: Vijay Bellur 
> Tested-by: Vijay Bellur 

Change-Id: I9b60f2ce24ca3787423a45ec7d502f89215fe45f
Signed-off-by: Venky Shankar 
BUG: 1220041
Reviewed-on: http://review.gluster.org/10721
Tested-by: Gluster Build System 
Reviewed-by: Gaurav Kumar Garg