summaryrefslogtreecommitdiffstats
path: root/tools/glusterfind/src/nodeagent.py
Commit message (Collapse)AuthorAgeFilesLines
* glusterfind: revert shebangs to #!/usr/bin/python3Kaleb S. KEITHLEY2019-02-261-1/+1
| | | | | | | | | | | | | | | Using #!/usr/bin/env python is illegal in Fedora and RHEL packaging. (Debian and SUSE packaging will complain about it too but don't enforce it like Fedora packaging does. Yet.) We went through a giant exercise fixing these once already. The python needs to be python2/python3 clean. There is a build scriptlet that converts the shebangs back to /usr/bin/python on rhel7. Change-Id: If208a2557f7e5d727cda98c27905e249cf485d5b updates: bz#1193929 Signed-off-by: Kaleb S. KEITHLEY <kkeithle@redhat.com>
* glusterfind: python2 to python3 compatShwetha K Acharya2019-02-021-2/+2
| | | | | | | | Made necessary modifications to ensure python3 compatibilty. fixes: bz#1658116 Change-Id: I5cf1d0447eaf3c44eb444245d1f67aadd60705c3 Signed-off-by: Shwetha K Acharya <sacharya@redhat.com>
* core: python3Kaleb S. KEITHLEY2018-09-031-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | see https://review.gluster.org/#/c/19788/, https://review.gluster.org/#/c/19871/, https://review.gluster.org/#/c/19952/, https://review.gluster.org/#/c/20104/, https://review.gluster.org/#/c/20162/, https://review.gluster.org/#/c/20185/, https://review.gluster.org/#/c/20207/, https://review.gluster.org/#/c/20227/, https://review.gluster.org/#/c/20307/, https://review.gluster.org/#/c/20320/, https://review.gluster.org/#/c/20332/, https://review.gluster.org/#/c/20364/, https://review.gluster.org/#/c/20441/, and https://review.gluster.org/#/c/20484 shebangs changed from /usr/bin/python2 to /usr/bin/python3. (Reminder, various distribution packaging guidelines require use of explicit python version and don't allow '#!/usr/bin/env python', regardless of how handy that idiom may be.) glusterfs.spec(.in) package python{2,3}-gluster and python2 or python3 dependencies as appropriate. configure(.ac): + test for and use python2 or python3 as appropriate. If build machine has python2 and python3, use python3. Override by setting PYTHON=/usr/bin/python2 when running configure. + PYTHONDEV_CPPFLAGS from python[23]-config --includes is a better match to the original python sysconfig.get_python_inc(). All those other extraneous flags breaks the build. + Only change the shebangs once. Changing them over and over again, e.g., during a `make glusterrpms` in extras/LinuxRPM just sends make (is it really make that's looping?) into an infinite loop. If you figure out why, let me know. + Oldest python2 is python2.6 on CentOS 6 and Debian 8 (Jessie). Everything else has 2.7 or 3.x + logic from https://review.gluster.org/c/glusterfs/+/21050, which needs to be removed/merged after that patch is merged. Builds on CentOS 6, CentOS 7, Fedora 28, Fedora rawhide, and the mysterious RHEL > 7. Change-Id: Idae21d3b6f58b32372e1daa0d234e491e563198f updates: #411 Signed-off-by: Kaleb S. KEITHLEY <kkeithle@redhat.com>
* glusterfind: Fix problems in python2 -> python3 compatKotresh HR2018-08-231-2/+2
| | | | | | | | | 1. Import configparser module correctly 2. Fix issues with urllib usage Updates: #411 Change-Id: I060af29f1ebac3a5e2aeeee2720b7372d4c31697 Signed-off-by: Kotresh HR <khiremat@redhat.com>
* core/various: python3 compat, prepare for python2 -> python3Kaleb S. KEITHLEY2018-06-041-1/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | see https://review.gluster.org/#/c/19788/, https://review.gluster.org/#/c/19871/, and https://review.gluster.org/#/c/19952/ This patch adds version agnostic imports for urllib, cpickle, socketserver, _thread, queue, etc., suggested by Aravinda in https://review.gluster.org/#/c/19767/1 Note: Fedora packaging guidelines require explicit shebangs, so popular practices like #!/usr/bin/env python and #!/usr/bin/python are not allowed; they must be #!/usr/bin/python2 or #!/usr/bin/python3 Note: Selected small fixes from 2to3 utility. Specifically apply, basestring, funcattrs, idioms, numliterals, set_literal, types, urllib, and zip have already been applied. Note: these 2to3 fixes report no changes are necessary: exec, execfile, exitfunc, filter, getcwdu, intern, itertools, metaclass, methodattrs, ne, next, nonzero, operator, paren, raw_input, reduce, reload, renames, repr, standarderror, sys_exc, throw, tuple_params, xreadlines. Change-Id: I8d393064a1837874d8b4bc87c8ce05c679664642 updates: #411 Signed-off-by: Kaleb S. KEITHLEY <kkeithle@redhat.com>
* core/various: python3 compat, prepare for python2 -> python3Kaleb S. KEITHLEY2018-05-301-3/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | see https://review.gluster.org/#/c/19788/ and https://review.gluster.org/#/c/19871/ Selected small fixes from 2to3 utility. Specifically apply, basestring, funcattrs, idioms, numliterals, set_literal, types, urllib, zip Note: these 2to3 fixes report no changes are necessary: exec, execfile, exitfunc, filter, getcwdu, intern, itertools, metaclass, methodattrs, ne, next, nonzero, operator, paren, raw_input, reduce, reload, renames, repr, standarderror, sys_exc, throw, tuple_params, xreadlines. Any 2to3 fixes not in the above two lists have more extensive changes which will follow in separate patches. most unicode changes suggested by 2to3 will need to be applied at the same time as changing the shebangs from python2 to python3. Prashanth notes that unicode strings in py2 need 'u' prefix; unicode strings in py3 3.0, 3.1, and 3.2 a 'u' prefix will throw an error, but in py3 3.3+ it is legal (or just ignored). All Linux dists we care about have 3.3 or later so we can leave 'u' prefixes on unicode strings. Change-Id: I49bba2f328b0ee24b9a8115a7183be979981563e updates: #411 Signed-off-by: Kaleb S. KEITHLEY <kkeithle@redhat.com>
* core/build/various: python3 compat, prepare for python2 -> python3Kaleb S. KEITHLEY2018-04-121-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | Note 1) we're not supposed to be using #!/usr/bin/env python, see https://fedoraproject.org/wiki/Packaging:Guidelines?rd=Packaging/Guidelines#Shebang_lines Note 2) we're also not supposed to be using "!/usr/bin/python, see https://fedoraproject.org/wiki/Changes/Avoid_usr_bin_python_in_RPM_Build#Quick_Opt-Out The previous patch (https://review.gluster.org/19767) tried to do too much in one patch, so it was abandoned. This patch does two things: 1) minor cleanup of configure(.ac) to explicitly use python2 2) change all the shebang lines to #!/usr/bin/python2 and add them where they were missing based on warnings emitted during rpmbuild. In a follow-up patch python2 will eventually be changed to python3. Before that python2-isms (e.g. print, string.join(), etc.) need to be converted to python3. Some of those can be rewritten in version agnostic python. E.g. print statements become print() with "from __future_ import print_function". The python 2to3 utility will be used for some of those. Also Aravinda has given guidance in the comments to the first patch for changes. updates: #411 Change-Id: I471730962b2526022115a1fc33629fb078b74338 Signed-off-by: Kaleb S. KEITHLEY <kkeithle@redhat.com>
* tools/glusterfind: kill remote processes and separate run-time directoriesMilind Changire2016-10-251-1/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Problem #1: Hitting CTRL+C leaves stale processes on remote nodes if glusterfind pre has been initiated. Solution #1: Adding "-t -t" to ssh command-line forces pseudo-terminal to be assigned to remote process. When local process receives Keyboard Interrupt, SIGHUP is immediately conveyed to the remote terminal causing remote changelog.py process to terminate immediately. Problem #2: Concurrent glusterfind pre runs are not possible on the same glusterfind session in case of a runaway process. Solution #2: glusterfind pre runs now add random directory name to the working directory to store and manage temporary database and changelog processing. If KeyboardInterrupt is received, the function call run_cmd_nodes("cleanup", args, tmpfilename=gtmpfilename) cleans up the remote run specific directory. Patch: 7571380 cli/xml: Fix wrong XML format in volume get command broke "gluster volume get <vol> changelog.rollover-time --xml" Now fixed function utils.py::get_changelog_rollover_time() Fixed spurious trailing space getting written if second path is empty in main.py::write_output() Fixed repetitive changelog processing in changelog.py::get_changes() Change-Id: Ia8d96e2cd47bf2a64416bece312e67631a1dbf29 BUG: 1382236 Signed-off-by: Milind Changire <mchangir@redhat.com> Reviewed-on: http://review.gluster.org/15609 Smoke: Gluster Build System <jenkins@build.gluster.org> CentOS-regression: Gluster Build System <jenkins@build.gluster.org> NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org> Reviewed-by: Aravinda VK <avishwan@redhat.com>
* tools/glusterfind: New option --no-encodeAravinda VK2016-02-261-0/+1
| | | | | | | | | | | | | | | | | | | | | | | | | New option added to skip encoding path in output file. Also handled Unicode strings. File paths can have newline characters, to differentiate between each path patch is encoded according to RFC3986(https://www.ietf.org/rfc/rfc3986.txt). Due to this consumer applications have to decode the path before consuming it. With this option Paths are not encoded, can be directly consumed by applications. Unicode encoding is handled automatically BUG: 1310080 Change-Id: I83d59831997dbd1264b48e9b1aa732c7dfc700b5 Signed-off-by: Aravinda VK <avishwan@redhat.com> Reviewed-on: http://review.gluster.org/13477 Smoke: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Milind Changire <mchangir@redhat.com> NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org> Reviewed-by: Kotresh HR <khiremat@redhat.com> CentOS-regression: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Venky Shankar <vshankar@redhat.com>
* tools/glusterfind: Cleanup session dir after deleteAravinda VK2015-06-121-0/+10
| | | | | | | | | | | | | | | | | Sessions directories are created in /var/lib/glusterd/glusterfind/<SESSION>/<VOLUME>, only <VOLUME> dir is created since same session name can exists for two volumes. Now cleanup is added, <SESSION> dir will be removed if <SESSION> contains only one Volume. Change-Id: I236393659e63eb98342570e0255739f4adea4491 Signed-off-by: Aravinda VK <avishwan@redhat.com> BUG: 1230111 Reviewed-on: http://review.gluster.org/11157 Reviewed-by: Kotresh HR <khiremat@redhat.com> Reviewed-by: Vijay Bellur <vbellur@redhat.com> Tested-by: Vijay Bellur <vbellur@redhat.com>
* tools/glusterfind: Partial FindAravinda VK2015-05-081-0/+123
Enabled by default, if one node fails Glusterfind will not fail to return list of files from other nodes. This behavior can be changed using --disable-partial Now session is maintained in each nodes as well as in initiator node. Every pre command will pick the status file from respective node and start collecting list of changes happened after the status time. --reset-session-time, new option to force reset the session time. Next incremental run will start from this time. Change-detector argument is removed since Changelog mode is required to detect deletes and Renames. Change-Id: I0b83bc7c0e1b30b13de772b2d21fe968db4ff964 Signed-off-by: Aravinda VK <avishwan@redhat.com> BUG: 1201289 Reviewed-on: http://review.gluster.org/10320 Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Kotresh HR <khiremat@redhat.com> Reviewed-by: Vijay Bellur <vbellur@redhat.com>