glusterfs.git/xlators/features/changelog/src/changelog.c, branch v3.7.9

all: reduce "inline" usage

2016-01-18T09:02:34+00:00

There are three kinds of inline functions: plain inline, extern inline,
and static inline.  All three have been removed from .c files, except
those in "contrib" which aren't our problem.  Inlines in .h files, which
are overwhelmingly "static inline" already, have generally been left
alone.  Over time we should be able to "lower" these into .c files, but
that has to be done in a case-by-case fashion requiring more manual
effort.  This part was easy to do automatically without (as far as I can
tell) any ill effect.

In the process, several pieces of dead code were flagged by the
compiler, and were removed.

backport of Change-Id: I56a5e614735c9e0a6ee420dab949eac22e25c155,
http://review.gluster.org/11769, BUG: 1245331

Change-Id: Iba1efb0bc578ea4a5e9bf76b7bd93dc1be9eba44
BUG: 1283302
Signed-off-by: Kaleb S KEITHLEY 
Reviewed-on: http://review.gluster.org/12646
Smoke: Gluster Build System 
NetBSD-regression: NetBSD Build System 
CentOS-regression: Gluster Build System 
Reviewed-by: Niels de Vos

features/changelog: Capture FXATTROP and XATTROP in changelog

2015-11-25T07:22:49+00:00

GEO-REP INTEROP WITH SHARD FEATURE

shard xlator updates size of the file using FXATTROP
or XATTROP. Hence record the same in changelog.

BUG: 1284453
Change-Id: I2f14b6075f863c0bed3ee2a159085b9b536a756d
Signed-off-by: Kotresh HR 
Reviewed-on: http://review.gluster.org/12225
Tested-by: NetBSD Build System 
Tested-by: Gluster Build System 
Reviewed-by: Venky Shankar 
Reviewed-on: http://review.gluster.org/12732
Reviewed-by: Aravinda VK

features/changelog: record mknod if tier-dht linkto is set

2015-10-29T07:17:47+00:00

This is a series of patches which aims to fix geo-replication
in a Tiering Volume.

Problem:
Consider, a file is placed in volume initially and then hot tier is
attached. During any operation on the file, due to lookup a linkto
file is created in hot tier.

Now, any namespace operation carried out on the file is recorded in
both cold and hot tier.
There is a room for races when both changelogs are replayed.

Solution:
So, We are going to replay (namespace related)operations
only in the hot tier.

Why?
a. If the file is directly placed in Hot tier, all fops will be
recorded in HOT tier.

b. If  the file is already present in Cold tier, and if any fop is
carried out, it creates linkto file in Hot tier.
Now, operations like UNLINK, RENAME are captured in Hot tier(by means of linkto file).

This way, we can get both tier's operation in HOT tier itself.

But, We may miss initial Data sync immediately after creating the
file as it is only recording MKNOD. So, if MKNOD encountered
with sticky bit set, queue DATA operation for the corresponding gfid.
( This geo-rep related changes are addressed in this patch: http://review.gluster.org/12326/ )

So, If tier-dht linkto is set, we need to record the corresponding
MKNOD. Earlier this was avoided as it was set as INTERNAL fop.
(This is addressed here in this patch)

Change-Id: I25514fe3e25f68592a8d6361507f8c8a4fcb70b1
BUG: 1275173
Reviewed-on: http://review.gluster.org/12417
Signed-off-by: Saravanakumar Arumugam 
Reviewed-on: http://review.gluster.org/12428
Tested-by: Gluster Build System 
Tested-by: NetBSD Build System 
Reviewed-by: Venky Shankar

features/changelog: Always log directory rename operations

2015-06-28T14:45:36+00:00

Directory renames are being ignored as special renames. Special
renames can happen only on files. Hence always log directory
rename operations in changelog.

Change-Id: I4fbdb3e02e634a39a8846fb2f7a4c6cc2ba74400
BUG: 1235242
Reviewed-On: http://review.gluster.org/11356
Signed-off-by: Vijay Bellur 
Reviewed-on: http://review.gluster.org/11378
Tested-by: NetBSD Build System 
Tested-by: Gluster Build System 
Reviewed-by: Kotresh HR

features/changelog: Do htime setxattr without XATTR_REPLACE flag

2015-06-12T18:37:43+00:00

HTIME_KEY marks the last changelog rolled over. The xattr is
maintained on .glusterfs/changelog/htime/HTIME.TSTAMP file.
On every rollover of the changelog file, the xattr is updated.
It is being updated with XATTR_REPLACE flag as xattr gets
created during changelog enable. But it is once found that
the xattrs on the file is cleared and is not reproduced later
on. This patch protects that case, if it happens by setting
xattr without XATTR_REPLACE flag in failure case.

The reason behind doing this in failure case is not to mask
the actual cause of xattrs getting cleared. This provides
the log message if the original issue still exists but the
consequential effects are fixed.

Also changed the log messages to depict the events happened
during changelog enable.

Change-Id: I699ed09a03667fd823d01d65c9c360fa7bc0e455
BUG: 1230694
Reviewed-On: http://review.gluster.org/#/c/11150/
Signed-off-by: Kotresh HR 
Reviewed-on: http://review.gluster.org/11181
Tested-by: Gluster Build System 
Reviewed-by: Vijay Bellur

features/changelog: Avoid setattr fop logging during rename

2015-06-12T05:26:16+00:00

Problem:
        When a file is renamed and the (renamed)file's Hashing
falls into a different brick, DHT creates a special file(linkto file)
in the brick(Hashed subvolume) and carries out setattr operation
on that file.

Currently, Changelog records this(setattr) operation in Hashed
subvolume. glusterfind in turn records this operation
as MODIFY operation.

So, there is a NEW entry in Cached subvolume and MODIFY entry
in Hashed subvolume for the same file.

Solution:
        Avoid logging setattr operation carried out, by
marking the operation as internal fop using xdata.

In changelog translator, check whether setattr is set
as internal fop and skip accordingly.

Change-Id: I21b09afb5a638b88a4ccb822442216680b7b74fd
BUG: 1230687
Signed-off-by: Saravanakumar Arumugam 
Reviewed-on: http://review.gluster.org/11183
Tested-by: Gluster Build System 
Tested-by: NetBSD Build System 
Reviewed-by: Raghavendra G 
Tested-by: Raghavendra G

featuress/changelog: On snapshot, notify irrespective of failures

2015-06-01T02:39:24+00:00

During snapshot, changelog barrier is enabled and a
explicit rollover of changelog is initiated. During
rollover of changelog, if any error or changelog is
empty, the notification was not sent to reconfigure
and hence snapshot was failing because of timeout.
This patch addresses it by sending notification
irrespective of failures and sends error if any
back to barrier.

BUG: 1225543
Change-Id: I971ef3bdc63bb50bda0b655e55cd814e44254ba9
Reviewed-On: http://review.gluster.org/10951
Signed-off-by: Kotresh HR 
Reviewed-on: http://review.gluster.org/10988
Tested-by: NetBSD Build System 
Tested-by: Gluster Build System 
Reviewed-by: Venky Shankar

geo-rep: rename handling in dht volume(changelog changes)

2015-05-09T09:08:10+00:00

Background:

Glusterfs changelogs are stored in each brick, which records the changes
happened in that brick. Georep will run in all the nodes of master and processes
changelogs "independently".  Processing changelogs is in brick level, but
all the fops will be replayed on "slave mount" point.

Problem:

With a DHT volume, in changelog "internal fops" are NOT recorded.
For Rename case, Rename is recorded in "hashed" brick changelog.
(DHT's internal fops like creating linkto file, unlink is NOT recorded).
This lead us to inconsistent rename operations.

For example,
Distribute volume created with Two bricks B1, B2.

//Consider master volume mounted @ /mnt/master and following operations
executed:
cd /mnt/master
touch f1 // f1 falls on B1 Hash
mv f1 f2 // f2 falls on B2 Hash

// Here, Changelogs are recorded as below:
@B1
CREATE f1

@B2
RENAME f1 f2

Here, race exists between Brick B1 and B2, say B2 will get executed first.
Source file f1 itself is "NOT PRESENT", so it will go ahead and create
f2 (Current implementation).

We have this problem When rename falls in another brick and
file is unlinked in Master.

Similar kind of issue exists in following case too(multiple rename):
CREATE f1
RENAME f1 f2
RENAME f2 f1

Solution:

Instead of carrying out "changelogging" at "HASHED volume",
carry out  at the "CACHED volume".
This way we have rename operations carried out where actual files are present.

So,Changelog recorded as :
@B1
CREATE f1
RENAME f1 f2

Note:
This patch is dependent on dht changes from this patch.
http://review.gluster.org/10410/
changelog related changes are separated out for review.

In changelog, xdata passed from DHT is considered as:

1. In case of unlink (internal operation as part of rename), xdata value
is set , it is considered as RENAME and recorded accordingly.
2. In case of rename (Hash and Cache different), xdata value is NOT
set, recording rename operation is SKIPPED.

BUG: 1219412
Change-Id: I7691166c84991482b2cfe073df64e2317c935b13
Reviewed-On: http://review.gluster.org/#/c/10220/
Signed-off-by: Saravanakumar Arumugam 
Reviewed-on: http://review.gluster.org/10633
Reviewed-by: Aravinda VK 
Tested-by: Gluster Build System 
Reviewed-by: Vijay Bellur

feature/changelog: Capture path for deletes

2015-05-05T13:02:22+00:00

PROBLEM:
There is no way to get the path of deleted file if we
have gfid from changelog since the file is already deleted.

SOLUTION:
Do a recursive readlink on parent gfid in backend .glusterfs
path to get the complete path in I/O callpath in changelog
translator and capture it in callback.

The path captured is relative from the brick root. The field
separator used is '\0'.
e.g.,
......\0/bname\0\0

ADDITIONAL REQUIRED CHANGES:

1. The changelog translator option called "changelog.capture-del-path"
   is introduced to enable or disable the capturing of deleted entry
   path.
   e.g.,
   gluster vol set  changelog.capture-del-path on/off

   If capture-del-path is disabled, '\0' is captured instead of
   relative path.
   e.g.,
   ......\0/bname\0\0\0

2. The minor number in the version of changelog is bumped up from v1.1
   to v1.2.

3. If recursive readlink is failed for some reason, it will capture
   \0 in place of .
   e.g.,
   ......\0/bname\0\0\0
   (same as when caputre-del-path option is disabled)

4. If bname argument passed to "resolve_pargfid_to_path" function
   is NULL and pargfid is ROOT, "." is returned. This is not the
   case with changelog, where bname is always passed. This is
   applicable to other consumers of "resolve_pargfid_to_path"
   routine.

NOTE:
   Changelog parser should consider the above new changes
   and should parse accordingly.

BUG: 1218383
Change-Id: I5d89cf4157befd207771f6c0248d2493fbf85832
Signed-off-by: Kotresh HR 
Reviewed-on: http://review.gluster.org/10288
Reviewed-on: http://review.gluster.org/10535
Tested-by: Gluster Build System 
Tested-by: NetBSD Build System
Reviewed-by: Aravinda VK 
Reviewed-by: Vijay Bellur

features/changelog: Consider only changelog on/off as changelog breakage

2015-05-05T07:05:29+00:00

Earlier, both chagelog on/off and brick restart were considered
to be changelog breakage and treated as changelog not being
continuous. As a result, new HTIME.TSTAMP file was created on
both the above cases. Now the change is made such that only
on changelog enable/disable, the changelog is considered to be
discontinuous. New HTIME.TSTAMP file is not created on brick
restart, the changelogs files are appended to last HTIME.TSTAMP
file.

Treating changelog as continuous in above scenario is important
as changelog history API will fail otherwise. It can successfully
get changes between start and end timestamps only when changelog
is continuous (Changelogs in single HTIME.TSTAMP file are treated
as continuous). Without this change, changelog history API would
fail, and it would become necessary to fallback to other mechanisms
like xsync FSCrawl in case geo-rep to detect changes in this time
window. But Xsync FSCrawl would not be applicable to other
consumers like glusterfind.

Rationale:
1. In plain distributed volume, if brick goes down, no I/O can
   happen onto the brick. Hence changelog is intact with data
   on disk.
2. In distributed replicate volume, if brick goes down, since
   self-heal traffic is captured in changelog. Eventually,
   I/O happened whend brick down is captured in changelog.

BUG: 1217944
Change-Id: Ifa6d932818fe1a3a914e87ac84f1d2ded01c1288
Signed-off-by: Kotresh HR 
Reviewed-on: http://review.gluster.org/10222
Reviewed-on: http://review.gluster.org/10507
Tested-by: Gluster Build System 
Reviewed-by: Aravinda VK 
Reviewed-by: Vijay Bellur