glusterfs.git/xlators/features/changelog, branch v3.7.0

geo-rep: rename handling in dht volume(changelog changes)

2015-05-09T09:08:10+00:00

Background:

Glusterfs changelogs are stored in each brick, which records the changes
happened in that brick. Georep will run in all the nodes of master and processes
changelogs "independently".  Processing changelogs is in brick level, but
all the fops will be replayed on "slave mount" point.

Problem:

With a DHT volume, in changelog "internal fops" are NOT recorded.
For Rename case, Rename is recorded in "hashed" brick changelog.
(DHT's internal fops like creating linkto file, unlink is NOT recorded).
This lead us to inconsistent rename operations.

For example,
Distribute volume created with Two bricks B1, B2.

//Consider master volume mounted @ /mnt/master and following operations
executed:
cd /mnt/master
touch f1 // f1 falls on B1 Hash
mv f1 f2 // f2 falls on B2 Hash

// Here, Changelogs are recorded as below:
@B1
CREATE f1

@B2
RENAME f1 f2

Here, race exists between Brick B1 and B2, say B2 will get executed first.
Source file f1 itself is "NOT PRESENT", so it will go ahead and create
f2 (Current implementation).

We have this problem When rename falls in another brick and
file is unlinked in Master.

Similar kind of issue exists in following case too(multiple rename):
CREATE f1
RENAME f1 f2
RENAME f2 f1

Solution:

Instead of carrying out "changelogging" at "HASHED volume",
carry out  at the "CACHED volume".
This way we have rename operations carried out where actual files are present.

So,Changelog recorded as :
@B1
CREATE f1
RENAME f1 f2

Note:
This patch is dependent on dht changes from this patch.
http://review.gluster.org/10410/
changelog related changes are separated out for review.

In changelog, xdata passed from DHT is considered as:

1. In case of unlink (internal operation as part of rename), xdata value
is set , it is considered as RENAME and recorded accordingly.
2. In case of rename (Hash and Cache different), xdata value is NOT
set, recording rename operation is SKIPPED.

BUG: 1219412
Change-Id: I7691166c84991482b2cfe073df64e2317c935b13
Reviewed-On: http://review.gluster.org/#/c/10220/
Signed-off-by: Saravanakumar Arumugam 
Reviewed-on: http://review.gluster.org/10633
Reviewed-by: Aravinda VK 
Tested-by: Gluster Build System 
Reviewed-by: Vijay Bellur

features/changelog: Fix buffer overflow in snprintf

2015-05-09T03:54:38+00:00

BUG: 1219823
Change-Id: I3a1c7b7742671847ed3fec13e06d861c3d09f7a9
Reviewed-On: http://review.gluster.org/#/c/10687/
Signed-off-by: Kotresh HR 
Reviewed-on: http://review.gluster.org/10688
Tested-by: Gluster Build System 
Reviewed-by: Aravinda VK

features/changelog: Fixing compilation issue

2015-05-08T15:37:46+00:00

This issue introduced due to manual rebase.

Change-Id: I0589f4a0a1270190340f419b8022d6483bcf853d
Signed-off-by: Aravinda VK 
BUG: 1219479
Reviewed-on: http://review.gluster.org/10685
Reviewed-by: Niels de Vos 
Reviewed-by: Kaleb KEITHLEY 
Tested-by: Niels de Vos 
Reviewed-on: http://review.gluster.org/10686

features/changelog: Avoid creation of empty changelogs

2015-05-08T13:14:12+00:00

An empty changelog when rolled over gets unlinked and indexed with
a modified path-name in htime file. The modification is "changelog"
not "CHANGELOG" in basename of the empty changelog file.

BUG: 1219479
Change-Id: Ib5b825ab563fa34d8dcf4368cf6cbf4b25d78a6d
Original-Author: Ajeet Jha 
Original-Author: Saravanakumar Arumugam 
Reviewed-On: http://review.gluster.org/#/c/9572/
Signed-off-by: Kotresh HR 
Reviewed-on: http://review.gluster.org/10642
Tested-by: Gluster Build System 
Reviewed-by: Aravinda VK 
Reviewed-by: Vijay Bellur

features/changelog: Version support for Changelog Parser

2015-05-08T10:22:02+00:00

As optional feature, during unlink, full path will be recorded.
Changelog Version number to be bumped up to 1.2.

With this patch, parser checks the version number before parsing
and handles accordingly.

Change-Id: Ic1ad98259c39e417029a08e26a1d4b467817e65a
BUG: 1218383
Signed-off-by: Kotresh HR 
Reviewed-on: http://review.gluster.org/10166
Reviewed-on: http://review.gluster.org/10620
Tested-by: Gluster Build System 
Reviewed-by: Saravanakumar Arumugam 
Tested-by: NetBSD Build System
Reviewed-by: Vijay Bellur

changelog: Fixing buffer overrun coverity issues.

2015-05-07T12:11:24+00:00

Coverity IDs:
1214630
1214631
1214633
1234643

Backport of: http://review.gluster.org/9557

Change-Id: I172c4f49bf651b2324522f9e661023f73ca05339
BUG: 789278
Signed-off-by: Nandaja Varma 
Reviewed-on: http://review.gluster.org/9557
Tested-by: Gluster Build System 
Reviewed-by: Sakshi Bansal
Reviewed-by: Venky Shankar 
(cherry picked from commit 7d7b80efe8c745f3ff7de76fc31c4977098cae01)
Reviewed-on: http://review.gluster.org/10595
Tested-by: NetBSD Build System
Tested-by: Venky Shankar

rpc: Maintain separate xlator pointer in 'rpcsvc_state'

2015-05-05T13:29:20+00:00

The structure 'rpcsvc_state', which maintains rpc server
state had no separate pointer to track the translator.
It was using the mydata pointer itself. So callers were
forced to send xlator pointer as mydata which is opaque
(void pointer) by function prototype.

'rpcsvc_register_init' is setting svc->mydata with xlator
pointer. 'rpcsvc_register_notify' is overwriting svc->mydata
with mydata pointer. And rpc interprets svc->mydata as
xlator pointer internally. If someone passes non xlator
structure pointer to rpcsvc_register_notify as libgfchangelog
currently does, it might corrupt mydata. So interpreting opaque
mydata as xlator pointer is incorrect as it is caller's choice
to send mydata as any type of data to 'rpcsvc_register_notify'.

Maintaining two different pointers in 'rpcsvc_state' for xlator
and mydata solves the issue.

BUG: 1218381
Change-Id: I4c28937a30845e3f41b6fc7a09036149c816659b
Signed-off-by: Kotresh HR 
Reviewed-on: http://review.gluster.org/10366
Reviewed-on: http://review.gluster.org/10534
Tested-by: Gluster Build System 
Tested-by: NetBSD Build System
Reviewed-by: Aravinda VK 
Reviewed-by: Vijay Bellur

feature/changelog: Capture path for deletes

2015-05-05T13:02:22+00:00

PROBLEM:
There is no way to get the path of deleted file if we
have gfid from changelog since the file is already deleted.

SOLUTION:
Do a recursive readlink on parent gfid in backend .glusterfs
path to get the complete path in I/O callpath in changelog
translator and capture it in callback.

The path captured is relative from the brick root. The field
separator used is '\0'.
e.g.,
......\0/bname\0\0

ADDITIONAL REQUIRED CHANGES:

1. The changelog translator option called "changelog.capture-del-path"
   is introduced to enable or disable the capturing of deleted entry
   path.
   e.g.,
   gluster vol set  changelog.capture-del-path on/off

   If capture-del-path is disabled, '\0' is captured instead of
   relative path.
   e.g.,
   ......\0/bname\0\0\0

2. The minor number in the version of changelog is bumped up from v1.1
   to v1.2.

3. If recursive readlink is failed for some reason, it will capture
   \0 in place of .
   e.g.,
   ......\0/bname\0\0\0
   (same as when caputre-del-path option is disabled)

4. If bname argument passed to "resolve_pargfid_to_path" function
   is NULL and pargfid is ROOT, "." is returned. This is not the
   case with changelog, where bname is always passed. This is
   applicable to other consumers of "resolve_pargfid_to_path"
   routine.

NOTE:
   Changelog parser should consider the above new changes
   and should parse accordingly.

BUG: 1218383
Change-Id: I5d89cf4157befd207771f6c0248d2493fbf85832
Signed-off-by: Kotresh HR 
Reviewed-on: http://review.gluster.org/10288
Reviewed-on: http://review.gluster.org/10535
Tested-by: Gluster Build System 
Tested-by: NetBSD Build System
Reviewed-by: Aravinda VK 
Reviewed-by: Vijay Bellur

features/changelog: Consider only changelog on/off as changelog breakage

2015-05-05T07:05:29+00:00

Earlier, both chagelog on/off and brick restart were considered
to be changelog breakage and treated as changelog not being
continuous. As a result, new HTIME.TSTAMP file was created on
both the above cases. Now the change is made such that only
on changelog enable/disable, the changelog is considered to be
discontinuous. New HTIME.TSTAMP file is not created on brick
restart, the changelogs files are appended to last HTIME.TSTAMP
file.

Treating changelog as continuous in above scenario is important
as changelog history API will fail otherwise. It can successfully
get changes between start and end timestamps only when changelog
is continuous (Changelogs in single HTIME.TSTAMP file are treated
as continuous). Without this change, changelog history API would
fail, and it would become necessary to fallback to other mechanisms
like xsync FSCrawl in case geo-rep to detect changes in this time
window. But Xsync FSCrawl would not be applicable to other
consumers like glusterfind.

Rationale:
1. In plain distributed volume, if brick goes down, no I/O can
   happen onto the brick. Hence changelog is intact with data
   on disk.
2. In distributed replicate volume, if brick goes down, since
   self-heal traffic is captured in changelog. Eventually,
   I/O happened whend brick down is captured in changelog.

BUG: 1217944
Change-Id: Ifa6d932818fe1a3a914e87ac84f1d2ded01c1288
Signed-off-by: Kotresh HR 
Reviewed-on: http://review.gluster.org/10222
Reviewed-on: http://review.gluster.org/10507
Tested-by: Gluster Build System 
Reviewed-by: Aravinda VK 
Reviewed-by: Vijay Bellur

features/bit-rot: Mark versioning fsetxattr as internal fop

2015-04-11T05:07:19+00:00

Changelog xlator was capturing bitrot-stub's fsetxattr sent
for versioning. Since it was using the same frame as of the
create fop, there was inconsistency in fop number and gfid
of capturing metadata. So fix is to mark fsetxattr used for
versioning as internal and add internal fop filter in
changelog_fsetxattr.

Change-Id: I51ff468995139838b22bf293a59a0713a92ee7a5
BUG: 1170075
Signed-off-by: Kotresh HR 
Reviewed-on: http://review.gluster.org/10148
Tested-by: Gluster Build System 
Reviewed-by: Venky Shankar 
Tested-by: Venky Shankar