summaryrefslogtreecommitdiffstats
path: root/xlators/features/changelog/src/changelog-misc.h
Commit message (Collapse)AuthorAgeFilesLines
* libglusterfs: Move devel headers under glusterfs directoryShyamsundarR2018-12-051-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | libglusterfs devel package headers are referenced in code using include semantics for a program, this while it works can be better especially when dealing with out of tree xlator builds or in general out of tree devel package usage. Towards this, the following changes are done, - moved all devel headers under a glusterfs directory - Included these headers using system header notation <> in all code outside of libglusterfs - Included these headers using own program notation "" within libglusterfs This change although big, is just moving around the headers and making it correct when including these headers from other sources. This helps us correctly include libglusterfs includes without namespace conflicts. Change-Id: Id2a98854e671a7ee5d73be44da5ba1a74252423b Updates: bz#1193929 Signed-off-by: ShyamsundarR <srangana@redhat.com>
* Land clang-format changesGluster Ant2018-09-121-79/+78
| | | | Change-Id: I6f5d8140a06f3c1b2d196849299f8d483028d33b
* changelog xlator: strncpy()->sprintf(), reduce strlen()'sYaniv Kaul2018-08-311-4/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | xlators/features/changelog/lib/src/gf-changelog-journal-handler.c xlators/features/changelog/lib/src/gf-changelog.c xlators/features/changelog/src/changelog-helpers.c xlators/features/changelog/src/changelog-misc.h strncpy may not be very efficient for short strings copied into a large buffer: If the length of src is less than n, strncpy() writes additional null bytes to dest to ensure that a total of n bytes are written. Instead, use snprintf(). Where possible, ensure there's no truncation of the output. Also: - save the result of strlen() and re-use it when possible. - move from strlen to SLEN (sizeof() ) for const strings. - switch a strncpy to a memcpy. Compile-tested only! Change-Id: Ia7a52bce0b243613ad910192ec163c93d944e077 updates: bz#1193929 Signed-off-by: Yaniv Kaul <ykaul@redhat.com>
* fips: Replace md5sum usage to enable fips supportKotresh HR2017-12-191-10/+10
| | | | | | | | | | | | md5sum is not fips compliant. Using xxhash64 instead of md5sum for socket file generation in glusterd and changelog to enable fips support. NOTE: md5sum is 128 bit hash. xxhash used is 64 bit. Updates: #230 Change-Id: I1bf2ea05905b9151cd29fa951f903685ab0dc84c Signed-off-by: Kotresh HR <khiremat@redhat.com>
* features/changelog: Add xattr as special type in changelog entryMohammed Rafi KC2017-05-101-3/+4
| | | | | | | | | | | | | | | | | When both SETATTR and SETXATTR fops are happening on gfid within the rollover time then, SETXATTR were not logged. In Which case we will miss the xattr fop in slave. This patch will be fix the same Change-Id: Ia75538ad1fd2797dbcf90d20dfa89f756009243d BUG: 1448914 Signed-off-by: Mohammed Rafi KC <rkavunga@redhat.com> Reviewed-on: https://review.gluster.org/17205 NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org> CentOS-regression: Gluster Build System <jenkins@build.gluster.org> Reviewed-by: Kotresh HR <khiremat@redhat.com> Smoke: Gluster Build System <jenkins@build.gluster.org>
* core: use syscall wrappers instead of direct syscallsKaleb S. KEITHLEY2015-10-281-1/+1
| | | | | | | | | | | | | | | various xlators and other components are invoking system calls directly instead of using the libglusterfs/syscall.[ch] wrappers. If not using the system call wrappers there should be a comment in the source explaining why the wrapper isn't used. Change-Id: I8ef94c48728666465abf126c778b70c9e5c00e47 BUG: 1267967 Signed-off-by: Kaleb S. KEITHLEY <kkeithle@redhat.com> Reviewed-on: http://review.gluster.org/12273 Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Jeff Darcy <jdarcy@redhat.com>
* features/changelog: Fixing compilation issueAravinda VK2015-05-081-1/+1
| | | | | | | | | | | | This issue introduced due to manual rebase. Change-Id: I0589f4a0a1270190340f419b8022d6483bcf853d Signed-off-by: Aravinda VK <avishwan@redhat.com> BUG: 1219479 Reviewed-on: http://review.gluster.org/10685 Reviewed-by: Niels de Vos <ndevos@redhat.com> Reviewed-by: Kaleb KEITHLEY <kkeithle@redhat.com> Tested-by: Niels de Vos <ndevos@redhat.com>
* features/changelog: Avoid creation of empty changelogsSaravanakumar Arumugam2015-05-081-1/+1
| | | | | | | | | | | | | | | | | | An empty changelog when rolled over gets unlinked and indexed with a modified path-name in htime file. The modification is "changelog" not "CHANGELOG" in basename of the empty changelog file. Change-Id: I77fd0b48b5c33c245418f5ac7a9756f08ece24d9 BUG: 1208470 Signed-off-by: Ajeet Jha <ajha@redhat.com> Signed-off-by: Saravanakumar Arumugam <sarumuga@redhat.com> Reviewed-on: http://review.gluster.org/9572 Tested-by: NetBSD Build System Reviewed-by: Aravinda VK <avishwan@redhat.com> Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Kotresh HR <khiremat@redhat.com> Reviewed-by: Venky Shankar <vshankar@redhat.com> Reviewed-by: Vijay Bellur <vbellur@redhat.com>
* features/changelog: Version support for Changelog ParserAravinda VK2015-05-061-2/+4
| | | | | | | | | | | | | | | | | | As optional feature, during unlink, full path will be recorded. Changelog Version number to be bumped up to 1.2. With this patch, parser checks the version number before parsing and handles accordingly. Change-Id: Ic1ad98259c39e417029a08e26a1d4b467817e65a BUG: 1214561 Signed-off-by: Aravinda VK <avishwan@redhat.com> Signed-off-by: Kotresh HR <khiremat@redhat.com> Reviewed-on: http://review.gluster.org/10166 Reviewed-by: Saravanakumar Arumugam <sarumuga@redhat.com> Tested-by: NetBSD Build System Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Venky Shankar <vshankar@redhat.com>
* feature/changelog: Capture path for deletesKotresh HR2015-05-041-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | PROBLEM: There is no way to get the path of deleted file if we have gfid from changelog since the file is already deleted. SOLUTION: Do a recursive readlink on parent gfid in backend .glusterfs path to get the complete path in I/O callpath in changelog translator and capture it in callback. The path captured is relative from the brick root. The field separator used is '\0'. e.g., ......\0<pgfid>/bname\0<relative-path>\0<next-record> ADDITIONAL REQUIRED CHANGES: 1. The changelog translator option called "changelog.capture-del-path" is introduced to enable or disable the capturing of deleted entry path. e.g., gluster vol set <vol-name> changelog.capture-del-path on/off If capture-del-path is disabled, '\0' is captured instead of relative path. e.g., ......\0<pgfid>/bname\0\0\0<next-record> 2. The minor number in the version of changelog is bumped up from v1.1 to v1.2. 3. If recursive readlink is failed for some reason, it will capture \0 in place of <relative path>. e.g., ......\0<pgfid>/bname\0\0\0<next-record> (same as when caputre-del-path option is disabled) 4. If bname argument passed to "resolve_pargfid_to_path" function is NULL and pargfid is ROOT, "." is returned. This is not the case with changelog, where bname is always passed. This is applicable to other consumers of "resolve_pargfid_to_path" routine. NOTE: Changelog parser should consider the above new changes and should parse accordingly. Change-Id: I040ed429b5aa7d391033fc6a540edbf07fc37827 BUG: 1214561 Signed-off-by: Kotresh HR <khiremat@redhat.com> Reviewed-on: http://review.gluster.org/10288 Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Venky Shankar <vshankar@redhat.com> Tested-by: NetBSD Build System
* features/changelog: Consider only changelog on/off as changelog breakageKotresh HR2015-05-021-0/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Earlier, both chagelog on/off and brick restart were considered to be changelog breakage and treated as changelog not being continuous. As a result, new HTIME.TSTAMP file was created on both the above cases. Now the change is made such that only on changelog enable/disable, the changelog is considered to be discontinuous. New HTIME.TSTAMP file is not created on brick restart, the changelogs files are appended to last HTIME.TSTAMP file. Treating changelog as continuous in above scenario is important as changelog history API will fail otherwise. It can successfully get changes between start and end timestamps only when changelog is continuous (Changelogs in single HTIME.TSTAMP file are treated as continuous). Without this change, changelog history API would fail, and it would become necessary to fallback to other mechanisms like xsync FSCrawl in case geo-rep to detect changes in this time window. But Xsync FSCrawl would not be applicable to other consumers like glusterfind. Rationale: 1. In plain distributed volume, if brick goes down, no I/O can happen onto the brick. Hence changelog is intact with data on disk. 2. In distributed replicate volume, if brick goes down, since self-heal traffic is captured in changelog. Eventually, I/O happened whend brick down is captured in changelog. Change-Id: I2eb66efe6ee9a9228fb1fcb38d6e7696b9559d5b BUG: 1211327 Signed-off-by: Kotresh HR <khiremat@redhat.com> Reviewed-on: http://review.gluster.org/10222 Reviewed-by: Venky Shankar <vshankar@redhat.com> Tested-by: Venky Shankar <vshankar@redhat.com> Tested-by: Gluster Build System <jenkins@build.gluster.com> Tested-by: NetBSD Build System
* changelog: Fixing buffer overrun coverity issues.Nandaja Varma2015-04-301-6/+6
| | | | | | | | | | | | | | | | Coverity IDs: 1214630 1214631 1214633 1234643 Change-Id: I172c4f49bf651b2324522f9e661023f73ca05339 BUG: 789278 Signed-off-by: Nandaja Varma <nvarma@redhat.com> Reviewed-on: http://review.gluster.org/9557 Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Sakshi Bansal Reviewed-by: Venky Shankar <vshankar@redhat.com>
* features/changelog: RPC'fy {libgf}changelogVenky Shankar2015-03-181-0/+14
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This patch introduces RPC based communication between the changelog translator and libgfchangelog. It replaces the old pathetic stream based interaction that existed earlier (due to time constraints :-/). Changelog, upon initialization starts a RPC server (rpcsvc) allowing clients to invoke a probe API as a bootup mechanism to request for event notifications. During probe, clients can choose an event filter specifying the type(s) of events they are interested in. As of now there is no way to change the event notification set once the probe RPC call is made, but that is easier to implement. The actual event notifications is done on a separate RPC session. The client (libgfchangelog) itself starts and RPC server which the changelog translator "connects back" during probe. Notifications are dispatched by a bunch of threads from the server (translator) and the client optionally orders them if ordered notifications are requried. FOPs fill in their respective event details in a buffer (rot-buffs to be particular) and a bunch of threads (consumers) swap the buffers out of roatation and dispatch them via RPC. To avoid writer starvation, then number of dispatcher threads is one less than the number of buffer list in rot-buffs.x libgfchangelog becomes purely callback based -- upon event notification from the server (and re-ordering them if required) invoke a callback routine specified by consumer(s). A major part of the patch is also aimed at providing backward compatibility for geo-replication, which was one of the main consumer of the stream based API. Also, this patch does not\ "turn on" event notifications for all fops, just a bunch which is currently in requirement. Another pain point is that the server does not filter events before dispatching it to the clients. That load is taken up by the client itself (although it's done at the library layer rather than making it hard on the callback implementor). This needs improvement and care needs to be taken to not load the server up with expensive filtering mechanisms. Change-Id: Ibf60a432b68f2dfa60c6f9add2bcfd37a9c41395 BUG: 1170075 Signed-off-by: Venky Shankar <vshankar@redhat.com> Reviewed-on: http://review.gluster.org/9708 Reviewed-by: Jeff Darcy <jdarcy@redhat.com> Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Vijay Bellur <vbellur@redhat.com>
* features/changelog: Crash consistency of changelog wrt snapshotAjeet Jha2014-08-261-0/+6
| | | | | | | | | | | | | This patch introduces call-path fop details logging for data operations in CHANGELOG.SNAP. This feature is enabled with barrier-enable notification and disabled with barrier-disable notification. Change-Id: Ib548d34203eb99cea478a6baff402e82251c73a4 BUG: 1128093 Signed-off-by: Ajeet Jha <ajha@redhat.com> Reviewed-on: http://review.gluster.org/8533 Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Vijay Bellur <vbellur@redhat.com>
* features/changelog : historical journal consumption.Venky Shankar2014-05-141-0/+7
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Facilitates Glusterfs with the ability to detect file-operations happened in past by scanning the back-end(brick-level) glusterfs journal (changelog). Design: * List of changelogs produces in one perfectly running session are stored in htime file which also holds necessary information about the session start and end time. * Involves fixed sized seeks to identify N'th changelog in the list. * Requires O(log n), (where n is number of changelogs in the list), time to identify the end changelog for the given start-end time interval. Currently the background processing of changelogs is sub optimal. BZ 1097041 tracks the development effort. For complete design, refer the below link: http://lists.nongnu.org/archive/html/gluster-devel/2014-02/msg00206.html Change-Id: I27e49f75e492e843084d0ecaf9130224d08462a0 BUG: 1091961 Signed-off-by: Ajeet Jha <ajha@redhat.com> Signed-off-by: Venky Shankar <vshankar@redhat.com> Signed-off-by: Ajeet Jha <ajha@redhat.com> Reviewed-on: http://review.gluster.org/6930 Reviewed-by: Kotresh HR <khiremat@redhat.com> Tested-by: Gluster Build System <jenkins@build.gluster.com>
* features/changelog: more changelog fixes.Ajeet Jha2013-12-121-1/+1
| | | | | | | | | | | | | | -> log additional records. -> include FOP number for metadata. -> prevent crash if inode is not found in a fop. Change-Id: I9edd4b71819ebd68c6a2b4150ae279c471d129da BUG: 1036536 Signed-off-by: Ajeet Jha <ajha@redhat.com> Reviewed-on: http://review.gluster.org/6403 Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Amar Tumballi <amarts@gmail.com> Reviewed-by: Venky Shankar <vshankar@redhat.com>
* features/changelog: changelog translatorAvra Sengupta2013-07-221-0/+101
This is the initial version of the Changelog Translator. What is it ----------- Goal is to capture changes performed on a GlusterFS volume. The translator needs to be loaded on the server (bricks) and captures changes in a plain text file inside a configured directory path (controlled by "changelog-dir", should be somewhere in <export>/.glusterfs/changelog by default). Changes are classified into 3 types: - Data: : TYPE-I - Metadata : TYPE-II - Entry : TYPE-III Changelog file is rolled over after a certain time interval (defauls to 60 seconds) after which a changelog is started. The thing to be noted here is that for a time interval (time slice) multiple changes for an inode are recorded only once (ie. say for 100+ writes on an inode that happens within the time slice has only a single corresponding entry in the changelog file). That way we do not bloat up the changelog and also save lots of writes. Changelog Format ----------------- TYPE-I and TYPE-II changes have the gfid on the entity on which the operation happened. TYPE-III being a entry op requires the parent gfid and the basename. Changelog format has been kept to a minimal and it's upto the consumers to do the heavy loading of figuring out deletes, renames etc.. A single changelog file records all three types of changes, with each change starting with an identifier ("D": DATA, "M": METADATA and "E": ENTRY). Option is provided for the encoding type (See TUNABLES). Consumers ---------- The only consumer as of today would be geo-replication, although backup utilities, self-heal, bit-rot detection could be possible consumers in the future. CLI ---- By default, change-logging is disabled (the translator is present in the server graph but does nothing). When enabled (via cli) each brick starts to log the changes. There are a set of tunable that can be used to change the translators behaviour: - enable/disable changelog (disabled by default) gluster volume set <volume> changelog {on|off} - set the logging directory (<brick>/.glusterfs/changelogs is the default) gluster volume set <volume> changelog-dir /path/to/dir - select encoding type (binary (default) or ascii) gluster volume set <volume> encoding {binary|ascii} - change the rollover time for the logs (60 secs by default) gluster volume set <volume> rollover-time <secs> - when secs > 0, changelog file is not open()'d with O_SYNC flag - and fsync is trigerred periodically every <secs> seconds. gluster volume set <volume> fsync-interval <secs> features/changelog: changelog consumer library (libgfchangelog) A shared library is provided for the consumer of the changelogs for easy acess via APIs. Application can link against this library and request for changelog updates. Conversion of binary logs to human-readable ascii format is also taken care by the library which keeps a copy of the changelog in application provided working directory. Change-Id: I75575fb7f1c53d2bec3dba1a329ea7bb3c628497 BUG: 847839 Original Author: Venky Shankar <vshankar@redhat.com> Signed-off-by: Avra Sengupta <asengupt@redhat.com> Reviewed-on: http://review.gluster.org/5127 Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Vijay Bellur <vbellur@redhat.com>
generated by cgit (git 2.34.1) at 2025-06-30 12:55:11 +0000