glusterfs.git/xlators/features/bit-rot/src/bitd, branch v4.0dev

libglusterfs: Name threads on creation

2017-07-19T14:16:19+00:00

Set names to threads on creation for easier
debugging.

Output of top -H -p 
Before:
19773 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glusterfsd
19774 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glusterfsd
19775 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glusterfsd
19776 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glusterfsd
19777 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glusterfsd
19778 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glusterfsd
19779 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glusterfsd
19780 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glusterfsd
19781 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glusterfsd
19782 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glusterfsd
19783 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glusterfsd
19784 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glusterfsd
19785 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.01 glusterfsd
19786 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.01 glusterfsd
19787 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.01 glusterfsd
19789 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glusterfsd
19790 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glusterfsd
25178 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glusterfsd
 5398 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glusterfsd
 7881 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glusterfsd

After:
19773 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glusterfsd
19774 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glustertimer
19775 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glusterfsd
19776 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glustermemsweep
19777 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glustersproc0
19778 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glustersproc1
19779 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glusterepoll0
19780 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glusteridxwrker
19781 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glusteriotwr0
19782 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glusterbrssign
19783 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glusterbrswrker
19784 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glusterclogecon
19785 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.01 glusterclogd0
19786 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.01 glusterclogd1
19787 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.01 glusterclogd2
19789 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glusterposixjan
19790 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glusterposixfsy
25178 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glusterepoll1
 5398 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glusterepoll2
 7881 root      20   0 1301.3m  12.6m   8.4m S  0.0  0.1   0:00.00 glusterposixhc

Change-Id: Id5f333755c1ba168a2ffaa4fce6e71c375e10703
BUG: 1254002
Updates: #271
Signed-off-by: Raghavendra Talur 
Reviewed-on: https://review.gluster.org/11926
Reviewed-by: Prashanth Pai 
Smoke: Gluster Build System 
Reviewed-by: Niels de Vos 
CentOS-regression: Gluster Build System

feature/bitrot: Fix ondemand scrub

2017-06-16T06:01:49+00:00

The flag which keeps tracks of whether the scrub
frequency is changed from previous value should
not be considered for on-demand scrubbing. It
should be considered only for 'scrub-frequency'
where it should not be re-scheduled if it is
set to same value again. But in case ondemand
scrub, it should start the scrub immediately
no matter what the scrub-frequency.

Reproducer:
1. Enable bitrot
2. Set scrub-throttle
3. Set ondemand scrub
Make sure glusterd is not restarted while doing
below steps

Change-Id: Ice5feaece7fff1579fb009d1a59d2b8292e23e0b
BUG: 1461845
Signed-off-by: Kotresh HR 
Reviewed-on: https://review.gluster.org/17552
Smoke: Gluster Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Raghavendra Bhat 
NetBSD-regression: NetBSD Build System

core: make the per glusterfs_ctx_t timer-wheel refcounted

2017-05-01T09:30:14+00:00

xlators can use a 'global' timer-wheel for scheduling events. This
timer-wheel is managed per glusterfs_ctx_t, but does not need to be
allocated for every graph. When an xlator wants to use the timer-wheel,
it will be instanciated on demand, and provided to xlators that request
it later on.

By adding a reference counter to the glusterfs_ctx_t for the
timer-wheel, the threads and structures can be cleaned up when the last
xlator does not have a need for it anymore. In general, the xlators
request the timer-wheel in init(), and they should return it in fini().

Because the timer-wheel is managed per glusterfs_ctx_t, the functions
can be added to ctx.c and do not need to live in their very minimal
tw.[ch] files.

Change-Id: I19d225b39aaa272d9005ba7adc3104c3764f1572
BUG: 1442788
Reported-by: Poornima G 
Signed-off-by: Niels de Vos 
Reviewed-on: https://review.gluster.org/17068
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Smoke: Gluster Build System 
Reviewed-by: Amar Tumballi 
Reviewed-by: Zhou Zhengping 
Reviewed-by: Kaleb KEITHLEY

trivial: correct some spelling mistakes in comments and logs

2016-10-18T10:27:12+00:00

BUG: 1385593
Change-Id: Icfae9e557a284182c6c22e9606fdd641528906f0
Reported-by: Patrick Matthäi 
Signed-off-by: Niels de Vos 
Reviewed-on: http://review.gluster.org/15656
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Atin Mukherjee 
Reviewed-by: jiffin tony Thottan 
Reviewed-by: Kotresh HR 
Smoke: Gluster Build System

build: out-of-tree builds generates files in the wrong directory

2016-09-18T16:34:37+00:00

And minor cleanup of a few of the Makefile.am files while we're
at it.

Rewrite the make rules to do what xdrgen does. Now we can get rid
of xdrgen.

Note 1. netbsd6's sed doesn't do -i. Why are we still running
smoke tests on netbsd6 and not netbsd7? We barely support netbsd7
as it is.

Note 2. Why is/was libgfxdr.so (.../rpc/xdr/src/...) linked with
libglusterfs? A cut-and-paste mistake? It has no references to
symbols in libglusterfs.

Note3. "/#ifndef\|#define\|#endif/" (note the '\'s) is a _basic_
regex that matches the same lines as the _extended_ regex
"/#(ifndef|define|endif)/". To match the extended regex sed needs to
be run with -r on Linux; with -E on *BSD. However NetBSD's and
FreeBSD's sed helpfully also provide -r for compatibility. Using a
basic regex avoids having to use a kludge in order to run sed with
the correct option on OS X.

Note 4. Not copying the bit of xdrgen that inserts copyright/license
boilerplate. AFAIK it's silly to pretend that machine generated
files like these can be copyrighted or need license boilerplate.
The XDR source files have their own copyright and license; and
their copyrights are bound to be more up to date than old
boilerplate inserted by a script. From what I've seen of other
Open Source projects -- e.g. gcc and its C parser files generated
by yacc and lex -- IIRC they don't bother to add copyright/license
boilerplate to their generated files.

It appears that it's a long-standing feature of make (SysV, BSD,
gnu) for out-of-tree builds to helpfully pretend that the source
files it can find in the VPATH "exist" as if they are in the $cwd.
rpcgen doesn't work well in this situation and generates files
with "bad" #include directives.

E.g. if you `rpcgen ../../../../$srcdir/rpc/xdr/src/glusterfs3-xdr.x`,
you get an #include directive in the generated .c file like this:

  ...
  #include "../../../../$srcdir/rpc/xdr/src/glusterfs3-xdr.h"
  ...

which (obviously) results in compile errors on out-of-tree build
because the (generated) header file doesn't exist at that location.
Compared to `rpcgen ./glusterfs3-xdr.x` where you get:

  ...
  #include "glusterfs3-xdr.h"
  ...

Which is what we need. We have to resort to some Stupid Make Tricks
like the addition of various .PHONY targets to work around the VPATH
"help".

Warning: When doing an in-tree build, -I$(top_builddir)/rpc/xdr/...
looks exactly like -I$(top_srcdir)/rpc/xdr/...  Don't be fooled though.
And don't delete the -I$(top_builddir)/rpc/xdr/... bits

Change-Id: Iba6ab96b2d0a17c5a7e9f92233993b318858b62e
BUG: 1330604
Signed-off-by: Kaleb S KEITHLEY 
Reviewed-on: http://review.gluster.org/14085
Tested-by: Niels de Vos 
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Niels de Vos

feature/bitrot: Ondemand scrub option for bitrot

2016-08-25T21:39:38+00:00

The bitrot scrubber takes 'hourly/daily/biweekly/monthly'
as the values for 'scrub-frequency'. There is no way
to schedule the scrubbing when the admin wants it.

Ondemand scrubbing brings in the new option 'ondemand'
with which the admin can start scrubbing ondemand.
It starts the scrubbing immediately.

Ondemand scrubbing is successful only if the scrubber
is in 'Active (Idle)' (waiting for it's next frequency
cycle to start scrubbing). It is not entertained when
the scrubber is in 'Paused' or already running.

Here is the command line syntax.

gluster volume bitrot  scrub ondemand

Change-Id: I84c28904367eed827a7dae8d6a535c14b28e9f4d
BUG: 1366195
Signed-off-by: Kotresh HR 
Reviewed-on: http://review.gluster.org/15111
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Venky Shankar

eventsapi: Fix disable-events issue

2016-08-24T08:23:06+00:00

Events related sources are not loaded in libglusterfs when
configure is run with --disable-events option. Due to this
every call of gf_event should be guarded with USE_EVENTS macro.

To prevent this, USE_EVENTS macro was included in events.c
itself(Patch #15054)

Instead of disabling building entire directory "events", selectively
disabled the code. So that constants and empty function gf_event is
exposed. Code will not fail even if gf_event is called when events is
disabled.

BUG: 1368042
Change-Id: Ia6abfe9c1e46a7640c4d8ff5ccf0e9c30c87f928
Signed-off-by: Aravinda VK 
Reviewed-on: http://review.gluster.org/15198
Smoke: Gluster Build System 
Reviewed-by: Niels de Vos 
CentOS-regression: Gluster Build System 
NetBSD-regression: NetBSD Build System

eventsapi: Bitrot events

2016-08-19T11:14:43+00:00

Following Bitrot Events are added

BITROT_ENABLE/BITROT_DISABLE
{
    "nodeid": NODEID,
    "ts": TIMESTAMP,
    "event": EVENT_TYPE,
    "message": {
       "name": VOLUME_NAME,
    }
}

BITROT_SCRUB_THROTTLE/BITROT_SCRUB_FREQ/BITROT_SCRUB
{
    "nodeid": NODEID,
    "ts": TIMESTAMP,
    "event": EVENT_TYPE,
    "message": {
       "name": VOLUME_NAME,
       "value": OPTION_VALUE
    }
}

EVENT_BITROT_BAD_FILE
{
    "nodeid": NODEID,
    "ts": TIMESTAMP,
    "event": EVENT_TYPE,
    "message": {
       "gfid": GFID_OF_FILE,
       "brick": BRICK_ROOT,
       "path":  FILE_PATH_FROM_BRICK_ROOT (if available)
    }
}

Change-Id: I8c37b0e9db9f4f0f3d05d8f78b5521c7db0e2237
BUG: 1367815
Signed-off-by: Kotresh HR 
Reviewed-on: http://review.gluster.org/15190
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
Reviewed-by: Atin Mukherjee 
CentOS-regression: Gluster Build System 
Reviewed-by: Aravinda VK

feature/bitrot: Ignore files with sticky bit set

2016-07-21T13:59:49+00:00

Scrubber scrubs entries in backend. It is scrubbing files with sticky
bit as well. This might include linkfiles which should be skipped.  This
patch adds the check to ignore linkfiles during scrub.

Change-Id: Ic21367b37770d391326c55c659491a1e5a82335b
BUG: 1355706
Signed-off-by: Kotresh HR 
Reviewed-on: http://review.gluster.org/14903
Smoke: Gluster Build System 
CentOS-regression: Gluster Build System 
CentOS-regression: NetBSD Build System 
NetBSD-regression: NetBSD Build System 
Reviewed-by: Jeff Darcy

feature/bitrot: Fix scrub status with sharded volume

2016-07-19T14:41:09+00:00

Bitrot scrubs each shard entries separately. Scrub
statistics was counting each shard entry which is
incorrect. This patch skips the statistics count
for sharded entries.

Change-Id: I184c315a4bc7f2cccabc506eef083ee926ec26d3
BUG: 1356851
Signed-off-by: Kotresh HR 
Reviewed-on: http://review.gluster.org/14927
Smoke: Gluster Build System 
CentOS-regression: Gluster Build System 
NetBSD-regression: NetBSD Build System 
Reviewed-by: Jeff Darcy 
Reviewed-by: Vijay Bellur