summaryrefslogtreecommitdiffstats
path: root/glustolibs-io/glustolibs
Commit message (Collapse)AuthorAgeFilesLines
* [Test] Add 2 memory leak tests and fix library issueskshithijiyer2020-10-211-27/+52
| | | | | | | | | | | | | | | | | | | | | | | | | | | | Scenarios added: ---------------- Test case: 1. Create a volume, start it and mount it. 2. Start I/O from mount point. 3. Check if there are any memory leaks and OOM killers. Test case: 1. Create a volume, start it and mount it. 2. Set features.cache-invalidation to ON. 3. Start I/O from mount point. 4. Run gluster volume heal command in a loop 5. Check if there are any memory leaks and OOM killers on servers. Design change: -------------- - self.id() is moved into test class as it was hitting bound errors in the original logic. - Logic changed for checking leaks fuse. - Fixed breakage in methods where ever needed. Change-Id: Icb600d833d0c08636b6002abb489342ea1f946d7 Signed-off-by: kshithijiyer <kshithij.ki@gmail.com>
* [Lib] Add memory and cpu leak testing frameworkkshithijiyer2020-09-111-0/+899
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Summary: ======== Currently we don't have a memory and cpu leak testing framework which blocks automation development of all testcases around memory and CPU leaks. To solve this problem we need the libraries for: 1. Logging memory and CPU utilization 2. Checking for OOM killer on gluster processes 3. Checking for memory leaks 4. Checking for cpu usage spikes 5. Wrapper functions in base class to make development easy 6. Compute statistics of usage Detailed description: ===================== We have already added script to log CPU and memory usage through patch [1]. In this patch we would be using patch [1] and building logic to process the CSV files generated by the script which would help us to achieve the following: 1. Checking if there are memory leaks or CPU spikes 2. Computing statistics of memory and CPU usage Function sets added: ~~~~~~~~~~~~~~~~~~~~ Set 1 - Functions to perfrom logging using script ------------------------------------------------- Public functions: 1. check_upload_memory_and_cpu_logger_script() 2. log_memory_and_cpu_usage_on_servers() 3. log_memory_and_cpu_usage_on_clients() 4. log_memory_and_cpu_usage_on_cluster() 5. wait_for_logging_processes_to_stop() 6. kill_all_logging_processes() Private functions to support public functions: 1. _start_logging_processes() 2. _process_wait_flag_append() Set 2 - Functions to check for OOM killers ------------------------------------------ Public functions: 1. check_for_oom_killers_on_servers() 2. check_for_oom_killers_on_clients() Private functions to support public functions: 1. _check_for_oom_killers() Set 3 - Functions to check for memory leaks ------------------------------------------- Public functions: 1. check_for_memory_leaks_in_glusterd() 2. check_for_memory_leaks_in_glusterfs() 3. check_for_memory_leaks_in_glusterfsd() 4. check_for_memory_leaks_in_glusterfs_fuse() Private functions to support public functions: 1. _perform_three_point_check_for_memory_leak() Set 4 - Functions to check for cpu usage spikes ----------------------------------------------- Public functions: 1. check_for_cpu_usage_spikes_on_glusterd() 2. check_for_cpu_usage_spikes_on_glusterfs() 3. check_for_cpu_usage_spikes_on_glusterfsd() 4. check_for_cpu_usage_spikes_on_glusterfs_fuse() Private functions to support public functions: 1. _check_for_cpu_usage_spikes() Set 7 - Functions to calculate stats ------------------------------------ Public functions: 1. compute_data_usage_stats_on_servers() 2. compute_data_usage_stats_on_clients() Private functions to support public functions: 1. _get_min_max_mean_median() 2. _compute_min_max_mean_median() Set 6 - Wrapper functions added to base class --------------------------------------------- 1. start_memory_and_cpu_usage_logging() 2. compute_and_print_usage_stats() 3. check_for_memory_leaks_and_oom_kills_on_servers() 4. check_for_memory_leaks_and_oom_kills_on_clients() 5. check_for_cpu_usage_spikes_on_servers() 6. check_for_cpu_spikes_on_clients() Set 7 - Other generic functions ------------------------------- Public functions: 1. create_dataframe_from_csv() Third party libraries added to glusto-tests through patch: 1. Numpy(It is being used in file_dir_ops.py but it's installed on clients and not on management node.) 2. Pandas 3. Statistics How do I use it in my testcase? ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ For example if we take the testcase to Check the server side memory leak with fuse mount, we would need to perfrom the below steps: 1. Create Disperse volumes 2. Fuse mount on the client 3. Start creating files in 1000's in parallel with 3. Create directories in 1000's in parallel 4. Do linux untar and wait for completion 5. Watch the memory usage on the server side and check for the OOM killers. Here steps 1-4 would be as usual, for step five what we need to do would be after step 2 we would need to start logging memory usage with the below code: ``` proc_dict = cls.start_memory_and_cpu_usage_logging() assertIsNotNone(proc_dict, <Error message>) ``` Once step 4 is complete we would need to wait for the logging process to stop with the below code: ``` ret = wait_for_logging_processes_to_stop(proc_dict, cluster=True) assertTrue(ret, <Error message>) ``` And lastly to check for memory leaks and OOM killers we would need use the below code: ``` ret = cls.check_for_memory_leaks_and_oom_kills_on_servers() assertTrue(ret, 'Memory leaks or OOM killer found') ``` NOTE: Interval and count of function start_memory_and_cpu_usage_logging() and gain of check_for_memory_leaks_and_oom_kills_on_servers() would need tweaking on a case by case scenario. Links: ====== [1] https://review.gluster.org/#/c/glusto-tests/+/24659/ Change-Id: Ib617fae102b8280723e54d0a38f77791558f5658 Signed-off-by: kshithijiyer <kshithij.ki@gmail.com>
* [Lib] Add run_linux_untar() to io/utils.pykshithijiyer2020-07-301-0/+46
| | | | | | | | | | Add method run_linux_untar() to io/utils.py which downloads linux-5.4.54.tar.xz and untar it on specific dirs of mount point or by default on mount point. Change-Id: Id00ea50b4d7fb7c360150aeaac65baec5612e589 Signed-off-by: kshithijiyer <kshithij.ki@gmail.com>
* [LibFix] Add kwargs start_range and end_rangesayaleeraut2020-06-241-3/+21
| | | | | | | | | Adding the kwargs start_range and end_range to the method open_file_fd() so that FD can be opened for multiple files if required. Change-Id: Ia6d78941935c7fb26045d000c428aba9b9f2425b Signed-off-by: sayaleeraut <saraut@redhat.com>
* [LibFix] Add arequal support on given pathubansal2020-06-151-3/+7
| | | | | | | | Calculates arequal for a particular path in the mountpoint Change-Id: I018302e6dbb11a9c11d42fc0381ec4183b3725a0 Signed-off-by: ubansal <ubansal@redhat.com>
* [Lib] Add Lib to open FDubansal2020-06-101-1/+19
| | | | | | | | Opens a FD to a file , waits and then writes to the file Change-Id: Ib993b646ba45d2b05a5765e02b6b1b7b2869ecd3 Signed-off-by: ubansal <ubansal@redhat.com>
* [lib] Adding function upload_file_dir_ops() to io/utils.pykshithijiyer2020-01-071-0/+28
| | | | | | | | | | | | | | | | | | | | | Adding a function upload_file_dir_ops() to upload file_dir_ops.py to all clients. This is to remove the redundant code given below: ``` g.log.info("Upload io scripts to clients %s for running IO on " "mounts", cls.clients) cls.script_upload_path = ("/usr/share/glustolibs/io/scripts/" "file_dir_ops.py") ret = upload_scripts(cls.clients, cls.script_upload_path) if not ret: raise ExecutionError("Failed to upload IO scripts to clients %s" % cls.clients) g.log.info("Successfully uploaded IO scripts to clients %s", cls.clients) ``` Change-Id: I28ca528bfbaea20139808b7af812559715a27022 Signed-off-by: kshithijiyer <kshithij.ki@gmail.com>
* [py2to3] Fix files located in the 'glustolibs-io' dirValerii Ponomarov2019-12-021-62/+45
| | | | | | | | | | | | | Do following things in the files from the mentioned directory: - Make it be python 2/3 compatible. - Fix pep8 issues. - Place imports in alphabetical order. - Improve some parts for the better readability. - Add comma separators in the setup.py for it's classifiers. Before it was mistakenly made as one long line. Change-Id: I3f0a5921a5386275a8a35bf2f22ee3952700e490 Signed-off-by: Valerii Ponomarov <kiparis.kh@gmail.com>
* Cthon test case for NFS GaneshaArjun Sharma2019-09-121-0/+44
| | | | | Change-Id: I3fb826bd0ecbe46bee4b9f8594b23f16921adbec Signed-off-by: Arjun Sharma <arjsharm@redhat.com>
* Adding library to run_crefi() to glustolibs-IO.kshithijiyer2019-08-271-0/+128
| | | | | | | | | | | | | | | | | | | | | | | What is crefi? Crefi is a Python command-line tool to create multi threaded workload on a filesystem and do file operations on created data. It provides basic operations to create files of different sizes, different types, in different directory structures, along with creating symlinks, hardlinks to files, and also rename, truncate, chmod, chown, chgrp and setxattr on the created files. Typical use involves creating large number of files of different sizes, over different layout, and do different operations on the created files. Which components need this tool? - Geo-rep - AFR Changes needed in infrastructure: https://github.com/gluster/centosci/commit/1fe330d34a62b56438f6eb86538286962b1abd90 Change-Id: I1e579f6f1f2c6ef0d018c055234abbf0f147e621 Signed-off-by: kshithijiyer <kshithij.ki@gmail.com>
* Library: check arequal bricks checksum for replicated volumekshithijiyer2019-05-241-0/+60
| | | | | | | | | calculates arequal checksum for first brick in the subvol and compares it with all other remaining bricks in the subvol. Change-Id: Ifc34b29d7971673a8a19b3ba603f63e985be4150 Signed-off-by: srivickynesh <sselvan@redhat.com> Signed-off-by: kshithijiyer <kshithij.ki@gmail.com>
* Copy-pasta typo: bonnie -> fioYaniv Kaul2019-04-131-1/+1
| | | | | | | | Fix small copy-pasta error. Change-Id: I69166d429f87fa9e94b58d165275414ec49bf3f5 updates: bz#1193929 Signed-off-by: Yaniv Kaul <ykaul@redhat.com>
* Added ret for io_resultsVitalii Koriakov2019-02-051-0/+2
| | | | | Change-Id: I34af7c73bae3a6ad7b32a2282f2e26da21597949 Signed-off-by: Vitalii Koriakov <vkoriako@redhat.com>
* Delete quorum method from old file. Added fixes for flake8Vitalii Koriakov2019-01-301-1/+1
| | | | | Change-Id: I2acf835a4cf7301c64c4c8a9423f78672cdf9aa4 Signed-off-by: Vitalii Koriakov <vkoriako@redhat.com>
* New method for checking not connected mount pointVitalii Koriakov2019-01-241-0/+64
| | | | | Change-Id: Idd492c3e806881ddc030023ecab30e4a8333467c Signed-off-by: Vitalii Koriakov <vkoriako@redhat.com>
* Using a raw strings for fixing flake warningsVitalii Koriakov2018-10-291-2/+2
| | | | | Change-Id: Ie134790ea05919a4b396657c11cbc3fc7a7fc529 Signed-off-by: Vitalii Koriakov <vkoriako@redhat.com>
* Shorten all the logs around verify_io_procsYaniv Kaul2018-07-171-1/+4
| | | | | | | | No functional change, just make the tests a bit more readable. It could be moved to a decorator later on, wrapping tests. Change-Id: I484bb8b46907ee8f33dfcf4c960737a21819cd6a Signed-off-by: Yaniv Kaul <ykaul@redhat.com>
* Adding test case for change owner, group, permission for directoryMohit Agrawal2018-06-291-0/+42
| | | | | | | | | | | | | | | | | 1) Create Dir with some file inside dir 2) Verify dir exists on all bricks as well as mount point 3) Compare dir stat with mount-point and brick location path 4) Change the ownership of directory 5) Compare dir stats with mount-point and brick path 6) Try to change pemission with different user for directory 7) Compare dir stat with mount-point and brick path 8) Try to change permission with different user for directory 9) change permission of directory 10) Compare dir stat with mount-point and brick path 11) Try to change permission with different user for same directory Change-Id: I284842be8c7562d4618d4e69e202c4d80945f1c5 Signed-off-by: Mohit Agrawal <moagrawa@redhat.com>
* Snapshot:: Fix: view_snaps_from_mounts functionsrivickynesh2018-06-151-1/+1
| | | | | | | | | | fix for function view_snaps_from_mounts 1. Iteration through snap_list was incorrect as we were comparing snaps taken from snap_list against snap_list itself. 2. Now it is changed to snaps which is superset of all snaps. Change-Id: Ib14e7819f6fd49e563fd9e8a8f7699581a8900b4 Signed-off-by: srivickynesh <sselvan@redhat.com>
* Adding is_io_procs_fail_with_rofs functionVijay Avuthu2018-02-061-0/+50
| | | | | Change-Id: If20aac0247dc42194a23c2b64952aac83234292e Signed-off-by: Vijay Avuthu <vavuthu@redhat.com>
* Added nfs ganesha tests to run IO test suites from multiple clients and ↵Arthy Loganathan2017-06-201-0/+204
| | | | | | | checks nfs ganesha behaviour Change-Id: I2dc7f0fb016982b7b7fa4a87c0310e4c96376f94 Signed-off-by: Arthy Loganathan <aloganat@redhat.com>
* Adding Quota, Snapshot Components Sanity Tests.Shwetha Panduranga2017-03-081-7/+63
| | | | | | | | | 1) Quota: Enabling, Setting Limit, Disabling, Listing of Quota is tested. 2) Snapshot: Creating, Listing, Activating, Viewing the snap from mount, De-Activating of snapshots is tested. Change-Id: Ia91e86e121d5d3fcc038704031617594d3d601d4 Signed-off-by: Shwetha Panduranga <spandura@redhat.com>
* Adding a Gluster Basic Features Sanity test:Shwetha Panduranga2017-03-021-11/+46
| | | | | | | | | | Test volume set option while IO is in progress. This basically tests IO to be successful after the client graph changes. (Note: This case will be run as part of Build Verification Test Suite) Change-Id: I111cf0214596fe32c872fdc73c5ccb8ab4a308be Signed-off-by: Shwetha Panduranga <spandura@redhat.com>
* Adding io utils:Shwetha Panduranga2017-02-141-0/+286
| | | | | | | | | | | | | | | | | 1) collect_mounts_arequal: Required for most of the testcase validation. 2) log_mounts_info: Get some data of mounts before the test and after the test for debugging purposes in case of failure 3) get_mounts_stat: Recursively get stat from mounts. 4) validate_io_procs: Validates the io's which were started asynchronously on all the mounts. 5) cleanup_mounts: remove all the data from mounts. Change-Id: Idce5304a7c7656088ea73675f953afe976eb1721 Signed-off-by: Shwetha Panduranga <spandura@redhat.com>
* glustolibs: add initial treeJonathan Holloway2016-09-292-0/+1
Change-Id: I08900d87fcd9a11b4f157d235facb766905d5ae2 Signed-off-by: Jonathan Holloway <jholloway@redhat.com>
generated by cgit (git 2.34.1) at 2024-04-30 21:36:43 +0000