summaryrefslogtreecommitdiffstats
path: root/xlators/features/changelog/src/changelog-helpers.c
Commit message (Collapse)AuthorAgeFilesLines
* features/changelog: simplify and cleanup internal time managementDmitry Antipov2020-08-221-34/+16
| | | | | | | | | | Drop extra point of failure in changelog_fill_rollover_data(), changelog_init() and reconfigure(), consistently use time_t and gf_time() where appropriate, adjust all related users. Change-Id: Id8e4236e96789cd74da5fdc3da05f0c1df98a62b Signed-off-by: Dmitry Antipov <dmantipov@yandex.ru> Updates: #1002
* libglusterfs: add library wrapper for time()Dmitry Antipov2020-08-171-1/+1
| | | | | | | | | Add thin convenient library wrapper gf_time(), adjust related users and comments as well. Change-Id: If8969af2f45ee69c30c3406bce5baa8305fb7f80 Signed-off-by: Dmitry Antipov <dmantipov@yandex.ru> Updates: #1002
* xlator/changelog: structure loggingyatipadia2020-01-141-66/+56
| | | | | | | | | convert all gf_msg() to gf_smsg() Change-Id: I2303aeaa0775b663e2ea5741166a457cd747d2ac Updates: #657 Signed-off-by: yatipadia <ypadia@redhat.com> Signed-off-by: yatip <ypadia@redhat.com>
* features/changelog: Optimization in changelogShwetha K Acharya2020-01-011-2/+29
| | | | | | | | | | | | Problem: Currently changelog is written in one directory, which over time, results in very large changelog files. Solution: Seperate directory under the changelogs directory is created on daily basis following the format year/month/day. Updates: #154 Change-Id: I1cdabe33728a0ba1f298c8908bd8c323b1871bda Signed-off-by: Shwetha K Acharya <sacharya@redhat.com>
* logging: Structured logging reference PRAravinda VK2019-08-201-56/+38
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | To convert the existing `gf_msg` to `gf_smsg`: - Define `_STR` of respective Message ID as below(In `*-messages.h`) #define PC_MSG_REMOTE_OP_FAILED_STR "remote operation failed." - Change `gf_msg` to use `gf_smsg`. Convert values into fields and add any missing fields. Note: `errno` and `error` fields will be added automatically to log message in case errnum is specified. Example: gf_smsg( this->name, // Name or log domain GF_LOG_WARNING, // Log Level rsp.op_errno, // Error number PC_MSG_REMOTE_OP_FAILED, // Message ID "path=%s", local->loc.path, // Key Value 1 "gfid=%s", loc_gfid_utoa(&local->loc), // Key Value 2 NULL // Log End ); Key value pairs formatting Help: gf_slog( this->name, // Name or log domain GF_LOG_WARNING, // Log Level rsp.op_errno, // Error number PC_MSG_REMOTE_OP_FAILED, // Message ID "op=CREATE", // Static Key and Value "path=%s", local->loc.path, // Format for Value "brick-%d-status=%s", brkidx, brkstatus, // Use format for key and val NULL // Log End ); Before: [2019-07-03 08:16:18.226819] W [MSGID: 114031] [client-rpc-fops_v2.c \ :2633:client4_0_lookup_cbk] 0-gv3-client-0: remote operation failed. \ Path: / (00000000-0000-0000-0000-000000000001) [Transport endpoint \ is not connected] After: [2019-07-29 07:50:15.773765] W [MSGID: 114031] \ [client-rpc-fops_v2.c:2633:client4_0_lookup_cbk] 0-gv1-client-0: \ remote operation failed. [{path=/f1}, \ {gfid=00000000-0000-0000-0000-000000000000}, \ {errno=107}, {error=Transport endpoint is not connected}] To add new `gf_smsg`, Add a Message ID in respective `*-messages.h` file and the follow the steps mentioned above. Change-Id: I4e7d37f27f106ab398e991d931ba2ac7841a44b1 Updates: #657 Signed-off-by: Aravinda VK <avishwan@redhat.com>
* changelog: remove unused code.Yaniv Kaul2019-04-031-21/+0
| | | | | | | | Seems to be unused. Change-Id: I75eed9641dd030a1fbb1b942a9d818f10a7e1437 updates: bz#1193929 Signed-off-by: Yaniv Kaul <ykaul@redhat.com>
* libglusterfs: Move devel headers under glusterfs directoryShyamsundarR2018-12-051-5/+5
| | | | | | | | | | | | | | | | | | | | | | | | libglusterfs devel package headers are referenced in code using include semantics for a program, this while it works can be better especially when dealing with out of tree xlator builds or in general out of tree devel package usage. Towards this, the following changes are done, - moved all devel headers under a glusterfs directory - Included these headers using system header notation <> in all code outside of libglusterfs - Included these headers using own program notation "" within libglusterfs This change although big, is just moving around the headers and making it correct when including these headers from other sources. This helps us correctly include libglusterfs includes without namespace conflicts. Change-Id: Id2a98854e671a7ee5d73be44da5ba1a74252423b Updates: bz#1193929 Signed-off-by: ShyamsundarR <srangana@redhat.com>
* all: fix warnings on non 64-bits architecturesXavi Hernandez2018-10-101-1/+1
| | | | | | | | | | When compiling in other architectures there appear many warnings. Some of them are actual problems that prevent gluster to work correctly on those architectures. Change-Id: Icdc7107a2bc2da662903c51910beddb84bdf03c0 fixes: bz#1632717 Signed-off-by: Xavi Hernandez <xhernandez@redhat.com>
* Land part 2 of clang-format changesGluster Ant2018-09-121-1526/+1461
| | | | | Change-Id: Ia84cc24c8924e6d22d02ac15f611c10e26db99b4 Signed-off-by: Nigel Babu <nigelb@redhat.com>
* changelog xlator: strncpy()->sprintf(), reduce strlen()'sYaniv Kaul2018-08-311-16/+27
| | | | | | | | | | | | | | | | | | | | | | | | | | xlators/features/changelog/lib/src/gf-changelog-journal-handler.c xlators/features/changelog/lib/src/gf-changelog.c xlators/features/changelog/src/changelog-helpers.c xlators/features/changelog/src/changelog-misc.h strncpy may not be very efficient for short strings copied into a large buffer: If the length of src is less than n, strncpy() writes additional null bytes to dest to ensure that a total of n bytes are written. Instead, use snprintf(). Where possible, ensure there's no truncation of the output. Also: - save the result of strlen() and re-use it when possible. - move from strlen to SLEN (sizeof() ) for const strings. - switch a strncpy to a memcpy. Compile-tested only! Change-Id: Ia7a52bce0b243613ad910192ec163c93d944e077 updates: bz#1193929 Signed-off-by: Yaniv Kaul <ykaul@redhat.com>
* features/changelog: Fix a file handle leak in htime_create()Vijay Bellur2018-08-231-2/+7
| | | | | | | | | In a few error scenarios, ht_file_hd was not being cleaned up. Addresses CID: 1325549 Change-Id: If9b4388aa700303c1eebbf1410dc35d18c4637df updates: bz#789278 Signed-off-by: Vijay Bellur <vbellur@redhat.com>
* features/changelog: Fix missing unlocksVijay Bellur2018-08-171-4/+6
| | | | | | | | Addresses CID 1210981 Change-Id: Icd325588ae0639e09d924fdde171931dedd06ca6 updates: bz#789278 Signed-off-by: Vijay Bellur <vbellur@redhat.com>
* features/changelog: Fix a resource leakVijay Bellur2018-08-151-0/+1
| | | | | | | | Fixes CID 1382359 Change-Id: Iaafbdb9a45496091327e3dc9092e09148fa9a5c5 updates: bz#789278 Signed-off-by: Vijay Bellur <vbellur@redhat.com>
* All: run codespell on the code and fix issues.Yaniv Kaul2018-07-221-4/+4
| | | | | | | | | | | | Please review, it's not always just the comments that were fixed. I've had to revert of course all calls to creat() that were changed to create() ... Only compile-tested! Change-Id: I7d02e82d9766e272a7fd9cc68e51901d69e5aab5 updates: bz#1193929 Signed-off-by: Yaniv Kaul <ykaul@redhat.com>
* Fix compile warningsXavi Hernandez2018-07-101-8/+31
| | | | | | | | | | | This patch fixes compile warnings that appear with newer compilers. The solution applied is only to remove the warnings, but it doesn't always solve the problem in the best way. It assumes that the problem will never happen, as the previous code assumed. Change-Id: I6e8470d6c2e2dbd3bd7d324b5fd2f92ffdc3d6ec updates: bz#1193929 Signed-off-by: Xavi Hernandez <xhernandez@redhat.com>
* features/changelog: Structured logging supportAravinda VK2017-11-151-93/+127
| | | | | | | | | | | Structured logging infra is available in logging library with issue #240. Log messages with dynamic content are identified and converted to structured logging format(`gf_msg` to `gf_smsg` and `gf_log` to `gf_slog`) BUG: 1501054 Change-Id: I5fccc354730c07cb9ae444d0b959d1d72bd9be49 Signed-off-by: Aravinda VK <avishwan@redhat.com>
* features/changelog: Fix htime xattr during brick crashv3.11devKotresh HR2017-01-191-1/+30
| | | | | | | | | | | | | | | | | | | | | | The htime file contains the path of all the changelogs that is rolloved over till now. It also maintains xattr which tracks the latest changelog file rolloved over and the number of changelogs. The path and and xattr update happens in two different system calls. If the brick is crashed between them, the xattr value becomes stale and can lead to the failure of gf_history_changelog. To identify this, the total number of changelogs is being calculated based on htime file size and the record length. The above value is used in case of mismatch. Change-Id: Ia1c3efcfda7b74227805bb2eb933c9bd4305000b BUG: 1413967 Signed-off-by: Kotresh HR <khiremat@redhat.com> Reviewed-on: http://review.gluster.org/16420 NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org> Smoke: Gluster Build System <jenkins@build.gluster.org> CentOS-regression: Gluster Build System <jenkins@build.gluster.org> Reviewed-by: Aravinda VK <avishwan@redhat.com>
* core: fix unused variable warnings/errorsKaleb S. KEITHLEY2016-08-291-2/+0
| | | | | | | | | | | | | | | | | http://review.gluster.org/14085 fixes a/the "leak" - via the generated rpc/xdr headers - of pragmas that mask these warnings. However 14085 won't pass the smoke test until all the warnings are fixed. Change-Id: I3d8a7a3de35058aa97eab59d3f59208396298b03 BUG: 1369124 Signed-off-by: Kaleb S. KEITHLEY <kkeithle@redhat.com> Reviewed-on: http://review.gluster.org/15246 Smoke: Gluster Build System <jenkins@build.gluster.org> NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org> CentOS-regression: Gluster Build System <jenkins@build.gluster.org> Reviewed-by: Kotresh HR <khiremat@redhat.com>
* features/changelog: Change barrier notification mechanismKotresh HR2016-05-311-46/+38
| | | | | | | | | | | | | | | | | | | | The barrier notification mechanism was fd based and 'select' was being used. 'select' breaks when number of fds opened by brick process exceeds 1024. To avoid this and also the maintainance of pipe between notify and 'changelog_rollover', the pipe has been replaced with pthread condition signal and timed wait mechanism. Change-Id: I530ea90d9a06953f8b23b4e12d122872ee1925de BUG: 1334314 Signed-off-by: Kotresh HR <khiremat@redhat.com> Reviewed-on: http://review.gluster.org/14272 Smoke: Gluster Build System <jenkins@build.gluster.com> NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org> Reviewed-by: Aravinda VK <avishwan@redhat.com> CentOS-regression: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Raghavendra Bhat <raghavendra@redhat.com> Reviewed-by: Jeff Darcy <jdarcy@redhat.com>
* features/changelog: Don't modify 'pargfid' in 'resolve_pargfid_to_path'Kotresh HR2016-03-301-1/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | If 'changelog' is enabled and 'changelog.capture-del-path' option is on it calls 'resolve_pargfid_to_path' which modifies 'pargfid' sent by caller. 'changelog_unlink' calls this routine directly with 'loc->pargfid' resulting it being modified and point to root instead of actual pargfid. This is a nasty bug and could cause the deletion of entry on root directory instead on actual parent when 'loc->path' is not present. Hence this fix to make 'pargfid' a const pointer and 'resolve_pargfid' to work on copy of pargfid. Glusterfind session creation enables these options by default to capture deleted entry path in changelog. Thanks Pranith for root causing this. Change-Id: I1d275a86f91c981b6889bedef93401c039d01d71 BUG: 1321955 Signed-off-by: Kotresh HR <khiremat@redhat.com> Reviewed-on: http://review.gluster.org/13845 Smoke: Gluster Build System <jenkins@build.gluster.com> NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org> Reviewed-by: Pranith Kumar Karampuri <pkarampu@redhat.com> Reviewed-by: Venky Shankar <vshankar@redhat.com> Reviewed-by: Ravishankar N <ravishankar@redhat.com> CentOS-regression: Gluster Build System <jenkins@build.gluster.com>
* features/changelog: Remove incorrect error messageKotresh HR2015-12-141-4/+0
| | | | | | | | | | | | | | | | | | During changelog_rollover, the rename error is logged even though changelog rollover is successful. This got mistakenly added as part of commit 64e6836ac8fb96f38a859eea0d8525cd9bd2693f (http://review.gluster.org/10532). Removing the same. Change-Id: I7ac1a16b41de5a6a4d7e714946585dff3ce0e83b BUG: 1290421 Signed-off-by: Kotresh HR <khiremat@redhat.com> Reviewed-on: http://review.gluster.org/12940 Reviewed-by: Aravinda VK <avishwan@redhat.com> Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Venky Shankar <vshankar@redhat.com> Tested-by: NetBSD Build System <jenkins@build.gluster.org>
* core: use syscall wrappers instead of direct syscallsKaleb S. KEITHLEY2015-10-281-19/+20
| | | | | | | | | | | | | | | various xlators and other components are invoking system calls directly instead of using the libglusterfs/syscall.[ch] wrappers. If not using the system call wrappers there should be a comment in the source explaining why the wrapper isn't used. Change-Id: I8ef94c48728666465abf126c778b70c9e5c00e47 BUG: 1267967 Signed-off-by: Kaleb S. KEITHLEY <kkeithle@redhat.com> Reviewed-on: http://review.gluster.org/12273 Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Jeff Darcy <jdarcy@redhat.com>
* all: reduce "inline" usageJeff Darcy2015-09-011-13/+13
| | | | | | | | | | | | | | | | | | | | | | | | | There are three kinds of inline functions: plain inline, extern inline, and static inline. All three have been removed from .c files, except those in "contrib" which aren't our problem. Inlines in .h files, which are overwhelmingly "static inline" already, have generally been left alone. Over time we should be able to "lower" these into .c files, but that has to be done in a case-by-case fashion requiring more manual effort. This part was easy to do automatically without (as far as I can tell) any ill effect. In the process, several pieces of dead code were flagged by the compiler, and were removed. Change-Id: I56a5e614735c9e0a6ee420dab949eac22e25c155 BUG: 1245331 Signed-off-by: Jeff Darcy <jdarcy@redhat.com> Reviewed-on: http://review.gluster.org/11769 Tested-by: NetBSD Build System <jenkins@build.gluster.org> Reviewed-by: Dan Lambright <dlambrig@redhat.com> Reviewed-by: Kaleb KEITHLEY <kkeithle@redhat.com> Reviewed-by: Niels de Vos <ndevos@redhat.com> Reviewed-by: Raghavendra Bhat <raghavendra@redhat.com> Reviewed-by: Venky Shankar <vshankar@redhat.com>
* features/changelog: Porting log messages to new logging frameworkAnusha Rao2015-08-111-139/+211
| | | | | | | | | | | Change-Id: Ic7f842acca52908fd88e0796dc90b82650405b25 BUG: 1194640 Signed-off-by: Anusha Rao <anusha91rao@gmail.com> Reviewed-on: http://review.gluster.org/10532 Tested-by: NetBSD Build System <jenkins@build.gluster.org> Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Venky Shankar <vshankar@redhat.com> Reviewed-by: Kotresh HR <khiremat@redhat.com>
* features/changelog: fix buffer overrun in changelog-helpersPrasanna Kumar Kalever2015-07-141-5/+3
| | | | | | | | | | | | | | | | | changelog-helpers.c:1911:17: warning: Size argument is greater than the free space in the destination buffer strncat (result, pre_dir_name, PATH_MAX); ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ changelog-helpers.c:1919:17: warning: Size argument is greater than the free space in the destination buffer strncat (result, bname, PATH_MAX); ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Change-Id: I60ca7fe762f07cb72fe7b69f0253835becaff7b9 BUG: 1222238 Signed-off-by: Prasanna Kumar Kalever <prasanna.kalever@redhat.com> Reviewed-on: http://review.gluster.org/10802 Tested-by: NetBSD Build System <jenkins@build.gluster.org> Reviewed-by: Venky Shankar <vshankar@redhat.com>
* features/bitrot: cleanup, v2Venky Shankar2015-06-251-26/+4
| | | | | | | | | | | | | | | This patch uses "cleanup, v1" infrastrcuture to cleanup scrubber (data structures, threads, timers, etc..) on brick disconnection. Signer is not cleaned up yet: probably would be done as part of another patch. Change-Id: I78a92b8a7f02b2f39078aa9a5a6b101fc499fd70 BUG: 1231619 Signed-off-by: Venky Shankar <vshankar@redhat.com> Reviewed-on: http://review.gluster.org/11148 Tested-by: NetBSD Build System <jenkins@build.gluster.org> Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Raghavendra Bhat <raghavendra@redhat.com>
* features/changelog: Do htime setxattr without XATTR_REPLACE flagKotresh HR2015-06-121-8/+25
| | | | | | | | | | | | | | | | | | | | | | | | | | HTIME_KEY marks the last changelog rolled over. The xattr is maintained on .glusterfs/changelog/htime/HTIME.TSTAMP file. On every rollover of the changelog file, the xattr is updated. It is being updated with XATTR_REPLACE flag as xattr gets created during changelog enable. But it is once found that the xattrs on the file is cleared and is not reproduced later on. This patch protects that case, if it happens by setting xattr without XATTR_REPLACE flag in failure case. The reason behind doing this in failure case is not to mask the actual cause of xattrs getting cleared. This provides the log message if the original issue still exists but the consequential effects are fixed. Also changed the log messages to depict the events happened during changelog enable. Change-Id: I699ed09a03667fd823d01d65c9c360fa7bc0e455 BUG: 1230015 Signed-off-by: Kotresh HR <khiremat@redhat.com> Reviewed-on: http://review.gluster.org/11150 Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Venky Shankar <vshankar@redhat.com>
* features/changelog: Remove inline keyword to avoid warnings (gcc v5.1.1)Anoop C S2015-06-011-6/+6
| | | | | | | | | | | | | | | | | | | | | | | When compiled with gcc5, following warnings were displayed and volume start failed: changelog-helpers.h:499:1: warning: inline function 'changelog_dispatch_event' declared but never defined changelog_dispatch_event (xlator_t *, changelog_priv_t *, changelog_event_t *); gf-changelog-journal-handler.c:692:17: warning: 'list_add_tail' is static but used in inline function 'gf_changelog_queue_journal' which is not static list_add_tail (&entry->list, &jnl_proc->entries); Fix is to remove the keyword from function prototype and definitions. Change-Id: I188b35b7ca087a94d7a48a052b05a6d845e3b74b BUG: 1226307 Signed-off-by: Anoop C S <achiraya@redhat.com> Reviewed-on: http://review.gluster.org/11004 Tested-by: NetBSD Build System <jenkins@build.gluster.org> Reviewed-by: Venky Shankar <vshankar@redhat.com>
* featuress/changelog: On snapshot, notify irrespective of failuresKotresh HR2015-05-311-20/+23
| | | | | | | | | | | | | | | | | | | | During snapshot, changelog barrier is enabled and a explicit rollover of changelog is initiated. During rollover of changelog, if any error or changelog is empty, the notification was not sent to reconfigure and hence snapshot was failing because of timeout. This patch addresses it by sending notification irrespective of failures and sends error if any back to barrier. Change-Id: I898af624b44555281a9e43c69066077e0e121c17 BUG: 1225542 Signed-off-by: Kotresh HR <khiremat@redhat.com> Reviewed-on: http://review.gluster.org/10951 Tested-by: NetBSD Build System <jenkins@build.gluster.org> Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Aravinda VK <avishwan@redhat.com> Reviewed-by: Venky Shankar <vshankar@redhat.com>
* build: do not #include "config.h" in each fileNiels de Vos2015-05-291-5/+0
| | | | | | | | | | | | | | | | | | Instead of including config.h in each file, and have the additional config.h included from the compiler commandline (-include option). When a .c file tests for a certain #define, and config.h was not included, incorrect assumtions were made. With this change, it can not happen again. BUG: 1222319 Change-Id: I4f9097b8740b81ecfe8b218d52ca50361f74cb64 Signed-off-by: Niels de Vos <ndevos@redhat.com> Reviewed-on: http://review.gluster.org/10808 Tested-by: Gluster Build System <jenkins@build.gluster.com> Tested-by: NetBSD Build System Reviewed-by: Kaleb KEITHLEY <kkeithle@redhat.com> Reviewed-by: Pranith Kumar Karampuri <pkarampu@redhat.com>
* features/changelog: Fix buffer overflow in snprintfKotresh HR2015-05-081-8/+8
| | | | | | | | | | Change-Id: Ie7e7c6028c7bffe47e60a2e93827e0e8767a3d66 BUG: 1219894 Signed-off-by: Kotresh HR <khiremat@redhat.com> Reviewed-on: http://review.gluster.org/10687 Reviewed-by: Aravinda VK <avishwan@redhat.com> Tested-by: Gluster Build System <jenkins@build.gluster.com> Tested-by: NetBSD Build System
* features/changelog: Fixing compilation issueAravinda VK2015-05-081-1/+4
| | | | | | | | | | | | This issue introduced due to manual rebase. Change-Id: I0589f4a0a1270190340f419b8022d6483bcf853d Signed-off-by: Aravinda VK <avishwan@redhat.com> BUG: 1219479 Reviewed-on: http://review.gluster.org/10685 Reviewed-by: Niels de Vos <ndevos@redhat.com> Reviewed-by: Kaleb KEITHLEY <kkeithle@redhat.com> Tested-by: Niels de Vos <ndevos@redhat.com>
* features/changelog: Avoid creation of empty changelogsSaravanakumar Arumugam2015-05-081-15/+115
| | | | | | | | | | | | | | | | | | An empty changelog when rolled over gets unlinked and indexed with a modified path-name in htime file. The modification is "changelog" not "CHANGELOG" in basename of the empty changelog file. Change-Id: I77fd0b48b5c33c245418f5ac7a9756f08ece24d9 BUG: 1208470 Signed-off-by: Ajeet Jha <ajha@redhat.com> Signed-off-by: Saravanakumar Arumugam <sarumuga@redhat.com> Reviewed-on: http://review.gluster.org/9572 Tested-by: NetBSD Build System Reviewed-by: Aravinda VK <avishwan@redhat.com> Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Kotresh HR <khiremat@redhat.com> Reviewed-by: Venky Shankar <vshankar@redhat.com> Reviewed-by: Vijay Bellur <vbellur@redhat.com>
* feature/changelog: Capture path for deletesKotresh HR2015-05-041-0/+85
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | PROBLEM: There is no way to get the path of deleted file if we have gfid from changelog since the file is already deleted. SOLUTION: Do a recursive readlink on parent gfid in backend .glusterfs path to get the complete path in I/O callpath in changelog translator and capture it in callback. The path captured is relative from the brick root. The field separator used is '\0'. e.g., ......\0<pgfid>/bname\0<relative-path>\0<next-record> ADDITIONAL REQUIRED CHANGES: 1. The changelog translator option called "changelog.capture-del-path" is introduced to enable or disable the capturing of deleted entry path. e.g., gluster vol set <vol-name> changelog.capture-del-path on/off If capture-del-path is disabled, '\0' is captured instead of relative path. e.g., ......\0<pgfid>/bname\0\0\0<next-record> 2. The minor number in the version of changelog is bumped up from v1.1 to v1.2. 3. If recursive readlink is failed for some reason, it will capture \0 in place of <relative path>. e.g., ......\0<pgfid>/bname\0\0\0<next-record> (same as when caputre-del-path option is disabled) 4. If bname argument passed to "resolve_pargfid_to_path" function is NULL and pargfid is ROOT, "." is returned. This is not the case with changelog, where bname is always passed. This is applicable to other consumers of "resolve_pargfid_to_path" routine. NOTE: Changelog parser should consider the above new changes and should parse accordingly. Change-Id: I040ed429b5aa7d391033fc6a540edbf07fc37827 BUG: 1214561 Signed-off-by: Kotresh HR <khiremat@redhat.com> Reviewed-on: http://review.gluster.org/10288 Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Venky Shankar <vshankar@redhat.com> Tested-by: NetBSD Build System
* features/changelog: Consider only changelog on/off as changelog breakageKotresh HR2015-05-021-9/+203
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Earlier, both chagelog on/off and brick restart were considered to be changelog breakage and treated as changelog not being continuous. As a result, new HTIME.TSTAMP file was created on both the above cases. Now the change is made such that only on changelog enable/disable, the changelog is considered to be discontinuous. New HTIME.TSTAMP file is not created on brick restart, the changelogs files are appended to last HTIME.TSTAMP file. Treating changelog as continuous in above scenario is important as changelog history API will fail otherwise. It can successfully get changes between start and end timestamps only when changelog is continuous (Changelogs in single HTIME.TSTAMP file are treated as continuous). Without this change, changelog history API would fail, and it would become necessary to fallback to other mechanisms like xsync FSCrawl in case geo-rep to detect changes in this time window. But Xsync FSCrawl would not be applicable to other consumers like glusterfind. Rationale: 1. In plain distributed volume, if brick goes down, no I/O can happen onto the brick. Hence changelog is intact with data on disk. 2. In distributed replicate volume, if brick goes down, since self-heal traffic is captured in changelog. Eventually, I/O happened whend brick down is captured in changelog. Change-Id: I2eb66efe6ee9a9228fb1fcb38d6e7696b9559d5b BUG: 1211327 Signed-off-by: Kotresh HR <khiremat@redhat.com> Reviewed-on: http://review.gluster.org/10222 Reviewed-by: Venky Shankar <vshankar@redhat.com> Tested-by: Venky Shankar <vshankar@redhat.com> Tested-by: Gluster Build System <jenkins@build.gluster.com> Tested-by: NetBSD Build System
* changelog: Fixing buffer overrun coverity issues.Nandaja Varma2015-04-301-1/+1
| | | | | | | | | | | | | | | | Coverity IDs: 1214630 1214631 1214633 1234643 Change-Id: I172c4f49bf651b2324522f9e661023f73ca05339 BUG: 789278 Signed-off-by: Nandaja Varma <nvarma@redhat.com> Reviewed-on: http://review.gluster.org/9557 Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Sakshi Bansal Reviewed-by: Venky Shankar <vshankar@redhat.com>
* Avoid conflict between contrib/uuid and system uuidEmmanuel Dreyfus2015-04-041-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | glusterfs relies on Linux uuid implementation, which API is incompatible with most other systems's uuid. As a result, libglusterfs has to embed contrib/uuid, which is the Linux implementation, on non Linux systems. This implementation is incompatible with systtem's built in, but the symbols have the same names. Usually this is not a problem because when we link with -lglusterfs, libc's symbols are trumped. However there is a problem when a program not linked with -lglusterfs will dlopen() glusterfs component. In such a case, libc's uuid implementation is already loaded in the calling program, and it will be used instead of libglusterfs's implementation, causing crashes. A possible workaround is to use pre-load libglusterfs in the calling program (using LD_PRELOAD on NetBSD for instance), but such a mechanism is not portable, nor is it flexible. A much better approach is to rename libglusterfs's uuid_* functions to gf_uuid_* to avoid any possible conflict. This is what this change attempts. BUG: 1206587 Change-Id: I9ccd3e13afed1c7fc18508e92c7beb0f5d49f31a Signed-off-by: Emmanuel Dreyfus <manu@netbsd.org> Reviewed-on: http://review.gluster.org/10017 Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Niels de Vos <ndevos@redhat.com>
* features/changelog: RPC'fy {libgf}changelogVenky Shankar2015-03-181-39/+171
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This patch introduces RPC based communication between the changelog translator and libgfchangelog. It replaces the old pathetic stream based interaction that existed earlier (due to time constraints :-/). Changelog, upon initialization starts a RPC server (rpcsvc) allowing clients to invoke a probe API as a bootup mechanism to request for event notifications. During probe, clients can choose an event filter specifying the type(s) of events they are interested in. As of now there is no way to change the event notification set once the probe RPC call is made, but that is easier to implement. The actual event notifications is done on a separate RPC session. The client (libgfchangelog) itself starts and RPC server which the changelog translator "connects back" during probe. Notifications are dispatched by a bunch of threads from the server (translator) and the client optionally orders them if ordered notifications are requried. FOPs fill in their respective event details in a buffer (rot-buffs to be particular) and a bunch of threads (consumers) swap the buffers out of roatation and dispatch them via RPC. To avoid writer starvation, then number of dispatcher threads is one less than the number of buffer list in rot-buffs.x libgfchangelog becomes purely callback based -- upon event notification from the server (and re-ordering them if required) invoke a callback routine specified by consumer(s). A major part of the patch is also aimed at providing backward compatibility for geo-replication, which was one of the main consumer of the stream based API. Also, this patch does not\ "turn on" event notifications for all fops, just a bunch which is currently in requirement. Another pain point is that the server does not filter events before dispatching it to the clients. That load is taken up by the client itself (although it's done at the library layer rather than making it hard on the callback implementor). This needs improvement and care needs to be taken to not load the server up with expensive filtering mechanisms. Change-Id: Ibf60a432b68f2dfa60c6f9add2bcfd37a9c41395 BUG: 1170075 Signed-off-by: Venky Shankar <vshankar@redhat.com> Reviewed-on: http://review.gluster.org/9708 Reviewed-by: Jeff Darcy <jdarcy@redhat.com> Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Vijay Bellur <vbellur@redhat.com>
* feature/changelog: Logging entry as well for explicit sync virtual xattr.Kotresh HR2015-01-091-0/+72
| | | | | | | | | | | | | | | | | | | | | | | | | | | | This is an improvement over the patch 'http://review.gluster.org/9337' to trigger explicit geo-rep sync on regular files even if entry is not present on the slave. An attempt is made to find the pargfid and if available captures CREATE along with DATA in changelog. CREATE is captured with default file permissions. Setting this virtual setxattr on directories captures MKDIR in changelog. The value of setxattr can be as follows. If value = "1" : Both CREATE and DATA is captured in changelog if pargfid is available, else on DATA is captured. value = "any other: ENOTSUP is returned. Usage: setfattr -n glusterfs.geo-rep.trigger-sync -v "1" <file-path> NOTE: This patch supports explicit record of entries only for directories and regular files. Change-Id: Iedde8b2c8bc3b78db524050d8c866ff664811d01 BUG: 1176934 Signed-off-by: Kotresh HR <khiremat@redhat.com> Reviewed-on: http://review.gluster.org/9370 Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Venky Shankar <vshankar@redhat.com> Tested-by: Venky Shankar <vshankar@redhat.com>
* features/changelog: Crash consistency of changelog wrt snapshotAjeet Jha2014-08-261-0/+156
| | | | | | | | | | | | | This patch introduces call-path fop details logging for data operations in CHANGELOG.SNAP. This feature is enabled with barrier-enable notification and disabled with barrier-disable notification. Change-Id: Ib548d34203eb99cea478a6baff402e82251c73a4 BUG: 1128093 Signed-off-by: Ajeet Jha <ajha@redhat.com> Reviewed-on: http://review.gluster.org/8533 Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Vijay Bellur <vbellur@redhat.com>
* build: make GLUSTERD_WORKDIR rely on localstatedirHarshavardhana2014-08-071-2/+2
| | | | | | | | | | | | | | | | | | | | | | - Break-way from '/var/lib/glusterd' hard-coded previously, instead rely on 'configure' value from 'localstatedir' - Provide 's/lib/db' as default working directory for gluster management daemon for BSD and Darwin based installations - loff_t is really off_t on Darwin - fix-off the warnings generated by clang on FreeBSD/Darwin - Now 'tests/*' use GLUSTERD_WORKDIR a common variable for all platforms. - Define proper environment for running tests, define correct PATH and LD_LIBRARY_PATH when running tests, so that the desired version of glusterfs is used, regardless where it is installed. (Thanks to manu@netbsd.org for this additional work) Change-Id: I2339a0d9275de5939ccad3e52b535598064a35e7 BUG: 1111774 Signed-off-by: Harshavardhana <harsha@harshavardhana.net> Signed-off-by: Emmanuel Dreyfus <manu@netbsd.org> Reviewed-on: http://review.gluster.org/8246 Tested-by: Gluster Build System <jenkins@build.gluster.com>
* changelog: variable 'writen' is changedHumble Chirammal2014-07-141-5/+5
| | | | | | | | | | | | | This patch change the variable name to 'written' in changelog, along with few other typo fixing. Change-Id: I1d3cbea5c89fc0a15233bb6e61b4fbd4bfdd3a67 BUG: 1075417 Signed-off-by: Humble Chirammal <hchiramm@redhat.com> Reviewed-on: http://review.gluster.org/8267 Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Venky Shankar <vshankar@redhat.com> Tested-by: Venky Shankar <vshankar@redhat.com>
* features/changelog: prevent deadlock on thread cancellationVenky Shankar2014-07-111-0/+34
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | helper threads (fsync, rollover) wake up periodically and perform their respective operation under a lock (crt->lock). These threads are also subjected to cancellation under some circumstance such as disabling changelog. This is inherently dangerous when funtions which are cancellation points for pthread_cancel(3) are used in the locked region. Consider this pthread_mutex_lock(&mutex); { /* ... */ ret = fsync (fd); <-- cancellation point /* ... */ } pthread_mutex_unlock(&mutex); A pthread_cancel(3) by another thread just before fsync(3) but after pthread_mutex_lock(3) would result in the thread getting cancelled when fsync(3) is invoked, thereby never unlocking the mutex. Moreover, in case of changelog translator, the locked region (under crt->lock in changelog-rt.c) is also the code path for fop changelog updation. Therefore, unlocking the mutex in thread cleanup handler (pthread_cleanup_pop(3)) might prematurely release the mutex during fop updation path. This patch fixes such problems existing in fsync and rollover threads. Fix is to enter the locked region with cancellation disabled and enable it after mutex unlock. Also, test for a cancellation request early on in case none of the functions are cancellation points. Change-Id: I1795627a12827609c1da659d07fc1457ffa033de BUG: 1110917 Signed-off-by: Venky Shankar <vshankar@redhat.com> Reviewed-on: http://review.gluster.org/8106 Reviewed-by: Kotresh HR <khiremat@redhat.com> Reviewed-by: Vijay Bellur <vbellur@redhat.com> Tested-by: Gluster Build System <jenkins@build.gluster.com>
* feature/changelog: Fix for missing changelogs at backend.Kotresh H R2014-06-301-2/+26
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Problem: A few changelog files are missing at the backend during snapshot with changelog enabled. Cause: Race between actual rollover and explicit rollover. Changelog rollover can happen either due to actual or the explict rollover due to snapshot. Actual rollover is controlled by tuneable called rollover-time. The minimum granularity for rollover-time is 1 second Explicit rollover is asynchronous in nature and happens during snapshot. Basically, rollover renames the current CHANGELOG file to CHANGELOG.TIMESTAMP after rollover-time. Let's assume, at time 't1', actual and explicit rollover raced against each other and actual rollover won the race renaming the CHANGELOG file to CHANGELOG.t1 and opens a new CHANGELOG file. An immediate explicit rollover at time 't1' happened with in the same second to rename CHANGELOG file to CHANGELOG.t1 resulting in purging the earlier CHANGELOG.t1 file created by actual rollover. Solution: Adding a delay of 1 sec guarantees unique CHANGELOG.TIMESTAMP during explicit rollover. Thanks Venky, for the all the help in root causing the issue. Change-Id: I8958824e107e16f61be9f09a11d95f8645ecf34d BUG: 1109692 Signed-off-by: Kotresh H R <khiremat@redhat.com> Reviewed-on: http://review.gluster.org/8196 Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Venky Shankar <vshankar@redhat.com> Tested-by: Venky Shankar <vshankar@redhat.com>
* features/changelog: skipping empty changelogs in historyAjeet Jha2014-06-101-1/+3
| | | | | | | | | | | | | Empty changelgs are unlinked and htime-update is bypassed while publishing partially written changelog. Change-Id: I1533ad431e4fbf592d2f38837fce782932d01e9c BUG: 1105466 Signed-off-by: Ajeet Jha <ajha@redhat.com> Reviewed-on: http://review.gluster.org/7949 Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Venky Shankar <vshankar@redhat.com> Tested-by: Venky Shankar <vshankar@redhat.com>
* features/changelog : historical journal consumption.Venky Shankar2014-05-141-1/+102
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Facilitates Glusterfs with the ability to detect file-operations happened in past by scanning the back-end(brick-level) glusterfs journal (changelog). Design: * List of changelogs produces in one perfectly running session are stored in htime file which also holds necessary information about the session start and end time. * Involves fixed sized seeks to identify N'th changelog in the list. * Requires O(log n), (where n is number of changelogs in the list), time to identify the end changelog for the given start-end time interval. Currently the background processing of changelogs is sub optimal. BZ 1097041 tracks the development effort. For complete design, refer the below link: http://lists.nongnu.org/archive/html/gluster-devel/2014-02/msg00206.html Change-Id: I27e49f75e492e843084d0ecaf9130224d08462a0 BUG: 1091961 Signed-off-by: Ajeet Jha <ajha@redhat.com> Signed-off-by: Venky Shankar <vshankar@redhat.com> Signed-off-by: Ajeet Jha <ajha@redhat.com> Reviewed-on: http://review.gluster.org/6930 Reviewed-by: Kotresh HR <khiremat@redhat.com> Tested-by: Gluster Build System <jenkins@build.gluster.com>
* features/changelog: Fix to remove Warning message.Kotresh H R2014-05-021-2/+0
| | | | | | | | | | | | | | | Fix warning message introduced in changelog draining patch. It is not caught in gcc 4.4, caught in gcc 4.8!! Signed-off-by: Kotresh H R <khiremat@redhat.com> Change-Id: I01a8fda2102a409dd6e935805d4d48b41b615fe8 BUG: 1091817 Signed-off-by: Kotresh H R <khiremat@redhat.com> Reviewed-on: http://review.gluster.org/7636 Reviewed-by: Venky Shankar <vshankar@redhat.com> Tested-by: Venky Shankar <vshankar@redhat.com>
* features/changelog: Barrier in changelog during snapshot.Kotresh H R2014-05-011-2/+15
| | | | | | | | | | | | | | | | | | Changelog barriers unlink, rename, rmdir fops on barrier 'on' notification from glusterfsd mgmt layer and unbarriers the same on barrier 'off' notification during snapshot. Please see the following link for more details. http://www.gluster.org/community/documentation/index.php/Changelog_Design_changes_for_snapshot Signed-off-by: Kotresh H R <khiremat@redhat.com> Change-Id: Iea9c62fafc86242f9404e03679b1941aa9c88c9a Signed-off-by: Kotresh H R <khiremat@redhat.com> Reviewed-on: http://review.gluster.org/7415 Reviewed-by: Venky Shankar <vshankar@redhat.com> Reviewed-by: Varun Shastry <vshastry@redhat.com> Tested-by: Gluster Build System <jenkins@build.gluster.com>
* feature/changelog: Draining of in-transit fops in changelog.Kotresh H R2014-05-011-9/+324
| | | | | | | | | | | | | | | | | | | | This is required for Geo-rep to work with snapshots. Following things are done in this patch. 1. Draining of in-transit fops during changelog rollover. 2. Explicit rollover of changelog when snapshot barrier notification comes. During this, intransit fops are drained and changelog is rolled over. For more details on the purpose of the patch. Please visit following link. http://www.gluster.org/community/documentation/index.php/Changelog_Design_changes_for_snapshot Change-Id: I22690131e19d3027f6d8957178bdc3431b9062f6 Signed-off-by: Kotresh H R <khiremat@redhat.com> Reviewed-on: http://review.gluster.org/7216 Reviewed-by: Venky Shankar <vshankar@redhat.com> Reviewed-by: Varun Shastry <vshastry@redhat.com> Tested-by: Gluster Build System <jenkins@build.gluster.com>
* features/changelog: more changelog fixes.Ajeet Jha2013-12-121-0/+3
| | | | | | | | | | | | | | -> log additional records. -> include FOP number for metadata. -> prevent crash if inode is not found in a fop. Change-Id: I9edd4b71819ebd68c6a2b4150ae279c471d129da BUG: 1036536 Signed-off-by: Ajeet Jha <ajha@redhat.com> Reviewed-on: http://review.gluster.org/6403 Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Amar Tumballi <amarts@gmail.com> Reviewed-by: Venky Shankar <vshankar@redhat.com>
generated by cgit (git 2.34.1) at 2024-04-24 02:53:57 +0000