summaryrefslogtreecommitdiffstats
path: root/tests/basic/tier
Commit message (Collapse)AuthorAgeFilesLines
* tests: Add mechanism for disabled testsRaghavendra Talur2016-03-0918-2/+24
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Requirements: Should be able to skip tests from run-tests.sh run. Should be granular enough to disable on subset of OSes. Solution: Tests can have special comment lines with some comma separated values within them. Key names used to determine test status are G_TESTDEF_TEST_STATUS_CENTOS6 G_TESTDEF_TEST_STATUS_NETBSD7 Some examples: G_TESTDEF_TEST_STATUS_CENTOS6=BAD_TEST,BUG=123456 G_TESTDEF_TEST_STATUS_NETBSD7=KNOWN_ISSUE,BUG=4444444 G_TESTDEF_TEST_STATUS_CENTOS6=BAD_TEST,BUG=123456;555555 You can change status of test to enabled or delete the line only if all the bugs are closed or modified or if the patch fixes it. Change-Id: Idee21fecaa5837fd4bd06e613f5c07a024f7b0c2 BUG: 1295704 Signed-off-by: Raghavendra Talur <rtalur@redhat.com> Reviewed-on: http://review.gluster.org/13393 Smoke: Gluster Build System <jenkins@build.gluster.com> NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org> CentOS-regression: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Rajesh Joseph <rjoseph@redhat.com> Reviewed-by: Kaushal M <kaushal@redhat.com>
* Tier: making detach start fail when brick on hot tier is downhari2016-03-031-0/+6
| | | | | | | | | | | | | | | | | | | Currently detach tier start happens even when a hot brick is down this might lead to data loss. This patch prevents the detach tier start from being executed successfully if a brick in hot tier is down Change-Id: I3b6047a44bd01b8a6887d41f799f64de6bf075ef BUG: 1309999 Signed-off-by: hari <hgowtham@redhat.com> Reviewed-on: http://review.gluster.org/13474 Smoke: Gluster Build System <jenkins@build.gluster.com> Tested-by: hari gowtham <hari.gowtham005@gmail.com> Reviewed-by: Gaurav Kumar Garg <ggarg@redhat.com> NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org> CentOS-regression: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Jeff Darcy <jdarcy@redhat.com>
* tests/glusterd: Parse the new time formatMohammed Rafi KC2016-02-251-9/+4
| | | | | | | | | | | | | | | | | | A recent change in cli changed elapsed time format that broke a test. This patch will fix the issue with parsing. Change-Id: I9a4a4b28f654cf2ac223e25abfc9df6570607d74 BUG: 1312036 Signed-off-by: Mohammed Rafi KC <rkavunga@redhat.com> Reviewed-on: http://review.gluster.org/13524 Reviewed-by: Kaleb KEITHLEY <kkeithle@redhat.com> CentOS-regression: Gluster Build System <jenkins@build.gluster.com> Smoke: Gluster Build System <jenkins@build.gluster.com> NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org> Reviewed-by: Ravishankar N <ravishankar@redhat.com> Reviewed-by: Raghavendra Talur <rtalur@redhat.com>
* glusterd/rebalance: initialize defrag variable after glusterd restartMohammed Rafi KC2016-02-221-2/+5
| | | | | | | | | | | | | | | | | During reblance restart after glusterd restarted, we are not connecting to rebalance process from glusterd, because the defrag variable in volinfo will be null. Initializing the variable will connect the rpc Change-Id: Id820cad6a3634a9fc976427fbe1c45844d3d4b9b BUG: 1303028 Signed-off-by: Mohammed Rafi KC <rkavunga@redhat.com> Reviewed-on: http://review.gluster.org/13319 Smoke: Gluster Build System <jenkins@build.gluster.com> NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org> Reviewed-by: Dan Lambright <dlambrig@redhat.com> CentOS-regression: Gluster Build System <jenkins@build.gluster.com>
* Tier: "tier start force" command implementationhari gowtham2015-12-221-0/+61
| | | | | | | | | | | | | | | | The start command doesnt restart the tier deamon if the deamon is running at one node. hence to bring up the tierd on the nodes where the deamon is down, the force command is implemented. It skips the check for tierd running. Change-Id: I0037d3e5ecfe56637d0da201a97903c435d26436 BUG: 1292112 Signed-off-by: hari gowtham <hgowtham@redhat.com> Reviewed-on: http://review.gluster.org/12983 Reviewed-by: Atin Mukherjee <amukherj@redhat.com> Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Dan Lambright <dlambrig@redhat.com> Tested-by: Dan Lambright <dlambrig@redhat.com>
* tier:unlink during migrationMohammed Rafi KC2015-12-161-0/+91
| | | | | | | | | | | | | | | | | | | | | | files deleted during promotion were not deleting as the files are moving from hashed to non-hashed. On deleting a file that is undergoing promotion, the unlink call is not sent to the dst file as the hashed subvol == cached subvol. This causes the file to reappear once the migration is complete. This patch also fixes a problem with stale linkfile deleting. Change-Id: I4b02a498218c9d8eeaa4556fa4219e91e7fa71e5 BUG: 1282390 Signed-off-by: Mohammed Rafi KC <rkavunga@redhat.com> Reviewed-on: http://review.gluster.org/12829 Tested-by: NetBSD Build System <jenkins@build.gluster.org> Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Dan Lambright <dlambrig@redhat.com> Tested-by: Dan Lambright <dlambrig@redhat.com>
* tier/glusterd: Only positive values for freq-thresholdsJoseph Fernandes2015-12-161-0/+4
| | | | | | | | | | | | Fixed error handling for validation for freq-thresholds Change-Id: Ibe3a9752ac0b525b0c8c0d6c4b4e4d694bd91b88 BUG: 1291603 Signed-off-by: Joseph Fernandes <josferna@redhat.com> Reviewed-on: http://review.gluster.org/12970 Reviewed-by: Dan Lambright <dlambrig@redhat.com> Tested-by: Gluster Build System <jenkins@build.gluster.com> Tested-by: Dan Lambright <dlambrig@redhat.com>
* tier/glusterd : making new tier detach command throw warninghari gowtham2015-12-151-0/+65
| | | | | | | | | | | | | | | | | | | | | | | | | | For detach tier, the validation was done using the string "detach-tier" but the new commands used has the string "tier". Making the string use "tier" to compare, creates problem as the tier status and tier detach have the keyword "tier". So tier detach and tier status were separated. and strtok was used to prevent the condition from passing when the volume name has a substring of "tier". (only the second word from the string is got and checked if the feature is tier) Problem: new detach tier command doesnt throw warnings like "not a tier volume" or " detach tier not started" respectively instead it prints empty output. Fix: while validate the volume is checked if its a tiered volume if yes it is checked if the detach tier is started, else a warning is thrown respectively. Change-Id: I94246d53b18ab0e9406beaf459eaddb7c5b766c2 BUG: 1288517 Signed-off-by: hari gowtham <hgowtham@redhat.com> Reviewed-on: http://review.gluster.org/12883 Tested-by: NetBSD Build System <jenkins@build.gluster.org> Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Atin Mukherjee <amukherj@redhat.com>
* cluster/tier: fix loading tier.so into glusterdN Balachandran2015-12-031-2/+2
| | | | | | | | | | | | | | | | | glusterd occasionally loads shared libraries of translators. This failed for tiering due to a reference to dht_methods which is defined as a global variable which is not necessary. The global variable has been removed and this is now a member of dht_conf and is now initialised in the *_init calls. Change-Id: Ifa0a21e3962b5cd8d9b927ef1d087d3b25312953 BUG: 1287842 Signed-off-by: N Balachandran <nbalacha@redhat.com> Reviewed-on: http://review.gluster.org/12863 Tested-by: NetBSD Build System <jenkins@build.gluster.org> Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Dan Lambright <dlambrig@redhat.com> Tested-by: Dan Lambright <dlambrig@redhat.com>
* tier/libgfdb/sql: Correcting logic in sql query for readJoseph Fernandes2015-12-021-6/+9
| | | | | | | | | | | | | | | | | | | | | | Correcting query in the where clause from "a & b | c" to "a & (b | c)" where "a" is the condition to join the gf_file_tb and gf_flink_tb through gfids "b" is the condition for the write heat and "c" is the condition for read heat Change-Id: I99226d82b0efb68fbef3a40f02b215bb2b4370d6 BUG: 1286656 Signed-off-by: Joseph Fernandes <josferna@redhat.com> Signed-off-by: Dan Lambright <dlambrig@redhat.com> Reviewed-on: http://review.gluster.org/12823 Tested-by: NetBSD Build System <jenkins@build.gluster.org> Tested-by: Gluster Build System <jenkins@build.gluster.com>
* tier/glusterd : Validation for frequency thresholds and record-countersJoseph Fernandes2015-12-011-0/+78
| | | | | | | | | | | | | | | | | | | | | | | | | 1) if record-counters is set to off check if both the frequency thresholds are non-zero, then pop an error message, with volume set failed. 2) if record-counters is set to on check if both the frequency thresholds are zero, then pop an note, but volume set is not failed. 3) If any of the frequency thresholds are set to a non-zero value, switch record-counters on, if not already on 4) If both the frequency thresholds are set to zero, switch record-counters off, if not already off NOTE: In this fix we have 1) removed unnecessary ctr vol set options. 2) changed ctr_hardlink_heal_expire_period to ctr_lookupheal_link_timeout Change-Id: Ie7ccfd3f6e021056905a79de5a3d8f199312f315 BUG: 1286346 Signed-off-by: Joseph Fernandes <josferna@redhat.com> Signed-off-by: Dan Lambright <dlambrig@redhat.com> Reviewed-on: http://review.gluster.org/12780 Tested-by: Gluster Build System <jenkins@build.gluster.com> Tested-by: NetBSD Build System <jenkins@build.gluster.org>
* cluster/ec: Create copy of dict for setting internal xattrsPranith Kumar K2015-12-011-0/+59
| | | | | | | | | | | | | | | | | | | | | | Problem: Ec takes a ref of the request xdata and sets trusted.ec.version/algo etc xattrs as part of it. But this request xdata could be using same dictionary to do the operation on multiple subvolumes, due to which other subvolumes will have internal xattrs of ec in it and will be created on subvols where they are not supposed to appear. Fix: Take a copy of the request xdata/dict to prevent this from happening. Most of the debugging work and test script is contributed by Nitya. BUG: 1286910 Change-Id: If146435dfb89656158dbed3862a3e9a0cda60581 Signed-off-by: Pranith Kumar K <pkarampu@redhat.com> Reviewed-on: http://review.gluster.org/12831 Tested-by: NetBSD Build System <jenkins@build.gluster.org> Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Xavier Hernandez <xhernandez@datalab.es>
* glusterd: copy snapshot object during duplication of volfileMohammed Rafi KC2015-11-251-0/+48
| | | | | | | | | | | | | | | | When creating duplicate volfile for hot/cold tier, we need to copy the snapshot object in to volfile as it requires to generate snapshot brick volfile. Change-Id: I39ccfa20cd1c16ef2801901e3cd3a31c76f8995d BUG: 1284789 Signed-off-by: Mohammed Rafi KC <rkavunga@redhat.com> Reviewed-on: http://review.gluster.org/12734 Reviewed-by: Pranith Kumar Karampuri <pkarampu@redhat.com> Tested-by: Pranith Kumar Karampuri <pkarampu@redhat.com> Tested-by: NetBSD Build System <jenkins@build.gluster.org> Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Avra Sengupta <asengupt@redhat.com>
* tier/test: Fixing tests/basic/tier/legacy-many.tJoseph Fernandes2015-11-251-1/+1
| | | | | | | | | | | Change-Id: Icbd83afdeac053aec5b3b8fa19665a2908e87d8e BUG: 1285483 Signed-off-by: Joseph Fernandes <josferna@redhat.com> Reviewed-on: http://review.gluster.org/12751 Tested-by: NetBSD Build System <jenkins@build.gluster.org> Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Dan Lambright <dlambrig@redhat.com> Tested-by: Dan Lambright <dlambrig@redhat.com>
* tier/ctr: Correcting rename logicJoseph Fernandes2015-11-251-0/+49
| | | | | | | | | | | | | | | | | | | | | | | | | | | Problem: When a file with old_file_name and GFID_1 is renamed with a new_file_name which already exists and with GFID_2, this is what happens in linux internaly. a. "new_file_name" is unlinked for GFID_2 b. a hardlink "new_file_name" is created to GFID_1 c. "old_file_name" hardlink is unlinked for GFID_2. Well this is all internal to linux, and gluster just issues a rename system call at POSIX layer. But CTR Xlator doesn't delete the entries corresponding to the "new_file_name" and GFID_2. Thus leaving the stale entry in the DB. The following are the implications. a. Promotion are tried on these stale entries which will fail and show false results in the status of migration, b. GFID_2 Files with 2 hardlinks, which will have only one hardlink after the rename will not be promoted or demoted as the DB shows 2 entries. Solution: Delete the older database entry for the replaced hardlink Change-Id: I4eafa0872253e29ff1f0bec4283bcfc579ecf0e2 BUG: 1284090 Signed-off-by: Joseph Fernandes <josferna@redhat.com> Reviewed-on: http://review.gluster.org/12711 Tested-by: NetBSD Build System <jenkins@build.gluster.org> Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Dan Lambright <dlambrig@redhat.com> Tested-by: Dan Lambright <dlambrig@redhat.com>
* cluster/tier: readdirp to cold tier onlyDan Lambright2015-11-232-10/+67
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | It is possible a file would get migrated in the middle of a readdir operation. If there are four subvolumes A,B,C,D, and if readdir reads them in order and reaches subvol B, then, if a file is moved from D to A, it will not be included in the readdir output. This phenonema has pre-existed in DHT migration but is more apparent in tiering. When a file is moved off the hashed subvolume a T file is created. For tiering, we will make the cold subvolume the hashed subvolume. This will ensure the creation of a T file. Readdir will not skip T files in the tier translator. Making the cold subvolume the hashed subvolume ensures the T files created on promotions or creates will be less likely to fill the volume. Creates still put the data on the hot subvolume. Change-Id: Ifde557d3d0e94a4570ca9f115adee3db2ee75407 BUG: 1281598 Signed-off-by: Dan Lambright <dlambrig@redhat.com> Reviewed-on: http://review.gluster.org/12530 Tested-by: Gluster Build System <jenkins@build.gluster.com> Tested-by: NetBSD Build System <jenkins@build.gluster.org> Reviewed-by: N Balachandran <nbalacha@redhat.com> Reviewed-by: Raghavendra G <rgowdapp@redhat.com>
* cluster/tier make cache mode default for tiered volumesDan Lambright2015-11-176-5/+11
| | | | | | | | | | | | | The default mode for tiered volumes must be cache. The current test mode was for engineering and should ordinarily not be used by customers. Change-Id: I20583f54a9269ce75daade645be18ab8575b0b9b BUG: 1282076 Signed-off-by: Dan Lambright <dlambrig@redhat.com> Reviewed-on: http://review.gluster.org/12581 Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: mohammed rafi kc <rkavunga@redhat.com>
* tier/ctr: Providing option to record or ignore metadata heatJoseph Fernandes2015-11-151-0/+101
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | Currently we heat up a file for both data and metadata write. Here we provide a ctr xlator option called "ctr-record-metadata-heat" were the admin can decide on recording metadata heat i.e heatup a file on metadata writes or not. Metadata data operation are a. setattr: explicit changing of atime/mtime using utimes, changing of posix permissions of the file b. rename: Renaming a file, c. unlink, link: adding or deleting hardlinks d. xattrs: setting or removal of xattrs. NOTE: atime, mtime and ctime change through writev, readv, truncate, mknod and create will not be considered here as these fops are data and primary metadata fops. Defaultly "ctr-record-metadata-heat" is off. Admin can switch it on using gluster volume set command. Change-Id: I91157509255dd5cb429cda2b6d4f64582e155e7b BUG: 1279166 Signed-off-by: Joseph Fernandes <josferna@redhat.com> Reviewed-on: http://review.gluster.org/12540 Tested-by: NetBSD Build System <jenkins@build.gluster.org> Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Dan Lambright <dlambrig@redhat.com> Tested-by: Dan Lambright <dlambrig@redhat.com>
* tests: fix spurious error in fops-during-migration-pause.tDan Lambright2015-11-111-2/+2
| | | | | | | | | | | | | | The test did not spend long enough time moving the file for the pause to occur simultaneously, leading to failure. Solution is to elongate that time by increasing the file size. Change-Id: I1727fa9e3f7a987dfa07dd5da44c68d3f17218d9 BUG: 1280428 Signed-off-by: Dan Lambright <dlambrig@redhat.com> Reviewed-on: http://review.gluster.org/12570 Tested-by: NetBSD Build System <jenkins@build.gluster.org> Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Joseph Fernandes
* cluster/tier correct promotion cycle calculationDan Lambright2015-11-072-19/+17
| | | | | | | | | | | | | | | The tier translator should only choose candidate files for promotion from the most recent cycle, not a multiple of the most recent cycles. Otherwise user observed behavior can be inconsistent. Remove related test in tier.t that is subject to race condition. Change-Id: I9ad1523cac00f904097ce468efa6ddd515857024 BUG: 1275524 Signed-off-by: root <root@rhs-cli-15.gdev.lab.eng.bos.redhat.com> Signed-off-by: Dan Lambright <dlambrig@redhat.com> Reviewed-on: http://review.gluster.org/12480 Reviewed-by: Joseph Fernandes Tested-by: Gluster Build System <jenkins@build.gluster.com>
* tier/libgfdb: Replacing ASCII query file with binaryJoseph Fernandes2015-11-061-7/+10
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Earlier, when the database was queried we used to save all the queried records in an ASCII format in the query file. This caused issues like filename having ASCII delimiter and used to take a lot of space. The tier.c file also had a lot of parsing code. Here we changed the format of the query file to binary. All the logic of serialization and formating of query record is done by libgfdb. Libgfdb provides API, gfdb_write_query_record() and gfdb_read_query_record(), which the user i.e tier migrator and CTR xlator can use to write to and read from query file. With this binary format we save on disk space i.e reduce to 50% atleast as we are saving GFID's in binary format 16 bytes and not the string format which takes 36 bytes + We are not saving path of the file + we are also saving on ASCII delimiters. The on disk format of query record is as follows, +---------------------------------------------------------------------------+ | Length of serialized query record | Serialized Query Record | +---------------------------------------------------------------------------+ 4 bytes Length of serialized query record | | -------------------------------------------------| | | V Serialized Query Record Format: +---------------------------------------------------------------------------+ | GFID | Link count | <LINK INFO> |..... | FOOTER | +---------------------------------------------------------------------------+ 16 B 4 B Link Length 4 B | | | | -----------------------------| | | | | | V | Each <Link Info> will be serialized as | +-----------------------------------------------+ | | PGID | BASE_NAME_LENGTH | BASE_NAME | | +-----------------------------------------------+ | 16 B 4 B BASE_NAME_LENGTH | | | ------------------------------------------------------------------------| | | V FOOTER is a magic number 0xBAADF00D indicating the end of the record. This also serves as a serialized schema validator. Change-Id: I9db7416fd421e118dd44eafab8b535caafe50d5a BUG: 1272207 Signed-off-by: Joseph Fernandes <josferna@redhat.com> Reviewed-on: http://review.gluster.org/12354 Reviewed-by: N Balachandran <nbalacha@redhat.com> Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Dan Lambright <dlambrig@redhat.com> Tested-by: Dan Lambright <dlambrig@redhat.com>
* cluster/tier : Files skipped during tier query parsingN Balachandran2015-11-031-0/+69
| | | | | | | | | | | | | | | | | The tier query parsing code was using fscanf to read each record. As space is a delimiter for fscanf, filenames containing spaces caused the parsing to return unexpected values causing various issues in the tier process, including crashes due to buffer overflows. Change-Id: Ife602cb7ecb158fccbc2c89e4d2959bd97098a87 BUG: 1276562 Signed-off-by: N Balachandran <nbalacha@redhat.com> Reviewed-on: http://review.gluster.org/12469 Tested-by: NetBSD Build System <jenkins@build.gluster.org> Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Dan Lambright <dlambrig@redhat.com> Tested-by: Dan Lambright <dlambrig@redhat.com>
* cluster/tier enable CTR on attach tierDan Lambright2015-10-301-2/+0
| | | | | | | | | | | | | CTR is currently disabled by default, and must be manually enabled for tiering to start. This is an overhead on the administrator and easy to overlook. Enable it automatically when a tier is attached. Change-Id: I0c29de8762faec1bfe6d1376a57eeef3357ad15a BUG: 1274847 Signed-off-by: Dan Lambright <dlambrig@redhat.com> Reviewed-on: http://review.gluster.org/12420 Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: mohammed rafi kc <rkavunga@redhat.com>
* tests/tier: Move common functions to tier.rcN Balachandran2015-10-215-170/+10
| | | | | | | | | | | | Move common functions in tier .t files to tier.rc Change-Id: Ibc312d987be9d93e7cc7fc47d0bf598bb1c944c2 BUG: 1272319 Signed-off-by: N Balachandran <nbalacha@redhat.com> Reviewed-on: http://review.gluster.org/12404 Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Dan Lambright <dlambrig@redhat.com> Tested-by: Dan Lambright <dlambrig@redhat.com>
* cluster/tier: add pause tier for snapshotsDan Lambright2015-10-211-0/+87
| | | | | | | | | | | | | | | | | | | | Snaps of tiered volumes cannot handle files undergoing migration. We implement a helper mechanism to "pause" migration. Any files undergoing migration are aborted. Clean up is done to remove sticky bits and data at the destination. Migration is restarted after snap completes. For testing an internal switch is added. It is not exposed externally. gluster volume set vol1 tier-pause [true|false] Change-Id: Ia85bbf89ac142e9b7e73fcbef98bb9da86097799 BUG: 1267950 Signed-off-by: Dan Lambright <dlambrig@redhat.com> Reviewed-on: http://review.gluster.org/12304 Reviewed-by: N Balachandran <nbalacha@redhat.com> Tested-by: NetBSD Build System <jenkins@build.gluster.org> Tested-by: Gluster Build System <jenkins@build.gluster.com>
* cluster/dht : Do not migrate files with POSIX locks heldN Balachandran2015-10-152-0/+187
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | dht_migrate_file does not migrate file locks to the dst file. Any locks held on the source file are lost once the migration is complete. This issue is magnified in the case of a tier volume as file migrations occur more frequently and repeatedly as compared to a DHT rebalance. The fix makes 2 changes: 1. Before starting the actual migration process, check if there are any locks held on the file. If yes, do not migrate the file. 2. The rebalance process tries to lock on the entire file just before moving into the Phase 2 of the file migration. If the lock acquisition fails, the file migration does not proceed. If the lock is granted, the file migration proceeds. This still leaves a small window where conflicting locks can be granted to different clients. If client1 requests a lock on the src file just after it is converted to a linkto file and client2 requests a lock on the dst data file, they will both be granted, but all FOPs will be redirected to the dst data file. This issue will be taken up in a subsequent patch. Change-Id: I8c895fc3cced50dd2894259d40a827c7b43d58ac BUG: 1271148 Signed-off-by: N Balachandran <nbalacha@redhat.com> Reviewed-on: http://review.gluster.org/12347 Tested-by: NetBSD Build System <jenkins@build.gluster.org> Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Dan Lambright <dlambrig@redhat.com> Tested-by: Dan Lambright <dlambrig@redhat.com>
* tier/shd: make shd commands compatible with tieringMohammed Rafi KC2015-10-121-0/+97
| | | | | | | | | | | | | | | | tiering volfiles may contain afr and disperse together or multiple time based on configuration. And the informations for those configurations are stored in tier_info. So most of the volgen code generation need to be changed to make compatible with it. Change-Id: I563d1ca6f281f59090ebd470b7fda1cc4b1b7e1d BUG: 1261276 Signed-off-by: Mohammed Rafi KC <rkavunga@redhat.com> Reviewed-on: http://review.gluster.org/12135 Tested-by: NetBSD Build System <jenkins@build.gluster.org> Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Pranith Kumar Karampuri <pkarampu@redhat.com>
* tier/ctr: CTR DB named lookup heal of cold tier during attach tierJoseph Fernandes2015-10-101-0/+122
| | | | | | | | | | | | | | | | | | | | | Heal hardlink in the db for already existing data in the cold tier during attach tier. i.e during fix layout do lookup to files in the cold tier. CTR xlator on the brick/server side does db update/insert of the hardlink on a namelookup. Currently the namedlookup is done synchronous to the fixlayout that is triggered by attach tier. This is not performant, adding more time to fixlayout. The performant approach is record the hardlinks on a compressed datastore and then do the namelookup asynchronously later, giving the ctr db eventual consistency Change-Id: I4ffc337fffe7d447804786851a9183a51b5044a9 BUG: 1252586 Signed-off-by: Joseph Fernandes <josferna@redhat.com> Reviewed-on: http://review.gluster.org/11828 Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Dan Lambright <dlambrig@redhat.com> Tested-by: Dan Lambright <dlambrig@redhat.com>
* cluster/tier: add watermarks and policy driverDan Lambright2015-10-101-0/+11
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This fix introduces infrastructure to support different policies for promotion and demotion. Currently the tier feature automatically promotes and demotes files periodically based on access. This is good for testing but too stringent for most real workloads. It makes it difficult to fully utilize a hot tier- data will be demoted before it is touched- its unlikely a 100GB hot SSD will have all its data touched in a window of time. A new parameter "mode" allows the user to pick promotion/demotion polcies. The "test mode" will be used for *.t and other general testing. This is the current mechanism. The "cache mode" introduces watermarks. The watermarks represent levels of data residing on the hot tier. "cache mode" policy: The % the hot tier is full is called P. Do not promote or demote more than D MB or F files. A random number [0-100] is called R. Rules for migration: if (P < watermark_low) don't demote, always promote. if (P >= watermark_low) && (P < watermark_hi) demote if R < P; promote if R > P. if (P > watermark_hi) always demote, don't promote. gluster volume set {vol} cluster.watermark-hi % gluster volume set {vol} cluster.watermark-low % gluster volume set {vol} cluster.tier-max-mb {D} gluster volume set {vol} cluster.tier-max-files {F} gluster volume set {vol} cluster.tier-mode {test|cache} Change-Id: I157f19667ec95aa1d53406041c1e3b073be127c2 BUG: 1257911 Signed-off-by: Dan Lambright <dlambrig@redhat.com> Reviewed-on: http://review.gluster.org/12039 Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Atin Mukherjee <amukherj@redhat.com>
* cluster/tier: fix transpoint endpoint not connected in tier.t (rare)Dan Lambright2015-10-091-0/+4
| | | | | | | | | | | | | | The script did not cleanly unmount/mount gluster and change the current working directory when stopping and starting the volume. Most of the time this problem would self-resolve before subsequent tests, but very occasionally races would lead to the errors/failures. Change-Id: I128b913a71e2745512ee81c3d71852311e3b4a1b BUG: 1270328 Signed-off-by: Dan Lambright <dlambrig@redhat.com> Reviewed-on: http://review.gluster.org/12327 Reviewed-by: Joseph Fernandes Tested-by: Gluster Build System <jenkins@build.gluster.com>
* cluster/tier re-enable tier.t in automatic testsDan Lambright2015-09-291-15/+29
| | | | | | | | | | | | | Re-enable tier.t in automatic tests. Disable check for BSD until recurring problem with SQLlite on it is understood. Change-Id: Ib13b269ab841a59a0a41d8478c8627b180b16c61 BUG: 1231268 Signed-off-by: Dan Lambright <dlambrig@redhat.com> Reviewed-on: http://review.gluster.org/12208 Tested-by: NetBSD Build System <jenkins@build.gluster.org> Reviewed-by: mohammed rafi kc <rkavunga@redhat.com> Tested-by: Gluster Build System <jenkins@build.gluster.com>
* cluster/tier: Handle FOPs on files being migratedN Balachandran2015-09-221-0/+121
| | | | | | | | | | | | | | | | Determine which DHT level is responsible for handling fops on a file undergoing migration based on the name of the the linkto xattr set on the file being migrated and process accordingly. Change-Id: I82772e39314d4fe7f2ba0dcf22de0c6a374ee139 BUG: 1254428 Signed-off-by: N Balachandran <nbalacha@redhat.com> Signed-off-by: Nithya Balachandran <nbalacha@redhat.com> Reviewed-on: http://review.gluster.org/12090 Tested-by: NetBSD Build System <jenkins@build.gluster.org> Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Raghavendra G <rgowdapp@redhat.com>
* tier/dht: unlink fails after lookup in a directoryMohammed Rafi KC2015-09-171-4/+7
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | unlink fails with invalid argument for files that are being present on cold tier, before attaching. All of the fops will be hashed to hot_tier after attach-tier (unless explicitly set the "rule" option). Lookups sent to directory, will eventually search the directory using readdirp, and will populate inode_ctx for the inodes based on the output, in respective dht_xlators. So the readdirp will populate inodes_ctx for the files (that is already present in volume before attaching) in cold-dht only because it got the entries from the cold-tier. So when an unlink comes on such an inode, the lookup associated with the unlink will be send as a re validate request to cold-tier only, since already a lookup was performed on the inode, and the new lookup will succeed. So from the unlink of dht, it will hash to cold-tier but the cached_subvol will be cold, since there is a mismatch in hash and cach , it chose hashed subvolume and will sent the fop to hot dht, and the fops fail with EINVAL from the hot-dht since it does not have inode_ctx stored for that inode (because, no lookup was performed from hot-dht). Change-Id: Ib7c14a9297a22d615f7a890a060be4809b5a745a BUG: 1236032 Signed-off-by: Mohammed Rafi KC <rkavunga@redhat.com> Signed-off-by: Dan Lambright <dlambrig@redhat.com> Reviewed-on: http://review.gluster.org/11675 Tested-by: NetBSD Build System <jenkins@build.gluster.org> Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Raghavendra G <rgowdapp@redhat.com>
* cluster/tier: add gluster v tier <vol>Dan Lambright2015-09-091-6/+6
| | | | | | | | | | | | | | | | | | | | | | | Currently the tier feature piggy backs off the rebalance command syntax to obtain status and this is clumsy. Introduce a new tier command that can do tier specific operations, starting with volume status to display counters. Old commands: gluster volume attach-tier <vol> [replica count] {bricklist..} gluster volume detach-tier <vol> {start|stop|commit} New commands: gluster volume tier <vol> attach [replica count] {bricklist} | detach {start|stop|commit} | status Change-Id: Ic07b3c6260588162de7d34380f8cbd3d8a7f35d3 BUG: 1255693 Signed-off-by: Dan Lambright <dlambrig@redhat.com> Reviewed-on: http://review.gluster.org/11984 Reviewed-by: Gaurav Kumar Garg <ggarg@redhat.com> Tested-by: Gluster Build System <jenkins@build.gluster.com>
* cluster/dht: Don't set posix acls on linkto filesNithya Balachandran2015-08-311-1/+1
| | | | | | | | | | | | | | | | | | | | Posix acls on a linkto file change the file's permission bits and cause DHT to treat it as a non-linkto file.This happens on the migration failure of a file on which posix acls were set. The fix prevents posix acls from being set on a linkto file and copies them across only after a file has been successfully migrated. Change-Id: Iccf7ff6fba49fe05d691d9b83bf76a240848b212 BUG: 1247563 Signed-off-by: Nithya Balachandran <nbalacha@redhat.com> Signed-off-by: N Balachandran <nbalacha@redhat.com> Reviewed-on: http://review.gluster.org/12025 Tested-by: NetBSD Build System <jenkins@build.gluster.org> Reviewed-by: Raghavendra G <rgowdapp@redhat.com> Reviewed-by: Pranith Kumar Karampuri <pkarampu@redhat.com>
* tiering/glusterd: start tier daemon during volume startMohammed Rafi KC2015-08-171-0/+5
| | | | | | | | | | | | | | | | | | | Tier daemon should always run with tier volume. If volume is stopped and started again, we manually need to start the tier-daemon, instead this patch will automatically trigger tier process along with volume start. A snapshot restored volume will not have node_state_info, so we need to create and store it dynamically Change-Id: I659387c914bec7a1b6929ee5cb61f7b406402075 BUG: 1238593 Signed-off-by: Mohammed Rafi KC <rkavunga@redhat.com> Signed-off-by: Dan Lambright <dlambrig@redhat.com> Reviewed-on: http://review.gluster.org/11525 Tested-by: NetBSD Build System <jenkins@build.gluster.org> Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Krishnan Parthasarathi <kparthas@redhat.com>
* cluster/tier: fixed pattern matching error in tier.tPamela Ousley2015-08-111-1/+1
| | | | | | | | | | | | | | | The check_counters function contained "grep -o '[0-9*]'", which was in error. This patch corrects it to "grep -o '[0-9]*'". This fix was necessary to accommodate for double-digit counters. Change-Id: Idaa09de4403bf66e741176a7377eba264819ca3b BUG: 1252121 Signed-off-by: Pamela Ousley <pousley@redhat.com> Reviewed-on: http://review.gluster.org/11877 Reviewed-by: Dan Lambright <dlambrig@redhat.com> Tested-by: NetBSD Build System <jenkins@build.gluster.org> Tested-by: Dan Lambright <dlambrig@redhat.com>
* cluster/tier : fix for logical bugs/timing errors in tier.tPamela Ousley2015-07-131-41/+96
| | | | | | | | | | | | | | | | | | The md5sum fingerprints were not correctly compared after moving files between the hot and cold tiers. This version of tier.t uses a new function, "check_counters", to ensure that the number of promotions/demotions is as expected. This is intended to avoid spurious timing-related errors that were seen with the old script. Change-Id: I4a0ae7315493bfd307a0f68f21fa3ea33c88b08f BUG: 1231268 Signed-off-by: Pamela Ousley <pousley@redhat.com> Reviewed-on: http://review.gluster.org/11285 Tested-by: NetBSD Build System <jenkins@build.gluster.org> Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Dan Lambright <dlambrig@redhat.com>
* cluster/tier: stop tier migration after graph switchDan Lambright2015-06-261-3/+18
| | | | | | | | | | | | | | | | | On a graph switch, a new xlator and private structures are created. The tier migration daemon must stop using the old xlator and private structures and begin using the new ones. Otherwise, when RPCs arrive (such as counter queries from glusterd), the new xlator will be consulted but it will not have up to date information. The fix detects a graph switch and exits the daemon in this case. Typical graph switches for the tier case would be turning off performance translators. Change-Id: Ibfbd4720dc82ea179b77c81b8f534abced21e3c8 BUG: 1226005 Signed-off-by: Dan Lambright <dlambrig@redhat.com> Reviewed-on: http://review.gluster.org/11372
* tiering/rebalance: tier daemon stopped with out updating statusMohammed Rafi KC2015-06-251-0/+3
| | | | | | | | | | | | | | | | When a subvol goes down, tier daemon stopped immediately, and the status shows as "Progressing". With this change, with respect to tier xlator, when a subvol goes offline it will update the status as failed. Change-Id: I9f722ed0d35cda8c7fc1a7e75af52222e2d0fdb7 BUG: 1227803 Signed-off-by: Mohammed Rafi KC <rkavunga@redhat.com> Reviewed-on: http://review.gluster.org/11068 Tested-by: NetBSD Build System <jenkins@build.gluster.org> Reviewed-by: Dan Lambright <dlambrig@redhat.com> Tested-by: Dan Lambright <dlambrig@redhat.com>
* glusterd/tier: configure tier daemon during volume restartMohammed Rafi KC2015-06-191-1/+2
| | | | | | | | | | | | | | rebalance daemon will be running on every tier volume for promoting/demoting the files. When volume/glusterd is restarted, then we need to configure the daemon. Change-Id: Ib565240a70edea2ec8bc1601c52b40c0783491d3 BUG: 1225330 Signed-off-by: Mohammed Rafi KC <rkavunga@redhat.com> Signed-off-by: Joseph Fernandes <josferna@redhat.com> Reviewed-on: http://review.gluster.org/10933 Reviewed-by: Dan Lambright <dlambrig@redhat.com> Tested-by: Dan Lambright <dlambrig@redhat.com>
* tier/volume set: Validate volume set option for tierMohammed Rafi KC2015-06-101-6/+15
| | | | | | | | | | | | | | | | Volume set option related to tier volume can only be set for tier volume, also currently all volume set i for tier option accepts a non-negative integer. This patch validate both condition. Change-Id: I3611af048ff4ab193544058cace8db205ea92336 BUG: 1216960 Signed-off-by: Mohammed Rafi KC <rkavunga@redhat.com> Signed-off-by: Dan Lambright <dlambrig@redhat.com> Reviewed-on: http://review.gluster.org/10751 Tested-by: Gluster Build System <jenkins@build.gluster.com> Tested-by: NetBSD Build System <jenkins@build.gluster.org> Reviewed-by: Joseph Fernandes
* Tests portability: umount(8)Emmanuel Dreyfus2015-06-091-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | 1) Avoid hangs on unmounting NFS on NetBSD NetBSD umount(8) on a NFS mount whose server is gone will wait forever because umount(8) calls realpath(3) and tries to access the mount before it calls unmount(2). The non-portable, NetBSD-specific umount -R flag prevent that behavior. We therefore introduce UMOUNT_F, defined as "umount -f" on Linux and "umount -f -R" on NetBSD to take care of forced unmounts, especially in the NFS case. 2) Enforce usage of force_umount wrapper with timeout Whenever umount is used it should be wrapped in force_umount with tiemout handling. That saves us timing issues, and it handles the NetBSD NFS case. 3) Cleanup kernel cache flush. We used (cd $M0 && umount $M0 ) as a portable kernel cache flush trick, but it does not flush everything we need on Linux. Introduce a drop_cache() shell function that reverts to previously used echo 3 > /proc/sys/vm/drop_caches on Linux, and keeps (cd $M0 && umount $M0 ) on other systems. BUG: 1129939 Change-Id: Iab1f5a023405f1f7270c42b595573702ca1eb6f3 Signed-off-by: Emmanuel Dreyfus <manu@netbsd.org> Reviewed-on: http://review.gluster.org/11114 Tested-by: NetBSD Build System <jenkins@build.gluster.org> Reviewed-by: Kaleb KEITHLEY <kkeithle@redhat.com> Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Vijay Bellur <vbellur@redhat.com>
* tiering/nfs: duplication of nodes in client graphMohammed Rafi KC2015-05-282-1/+7
| | | | | | | | | | | | | | | | | | | | When creating client volfiles, xlator tier-dht will be loaded for each volume. So for services like nfs have one or more volumes . So for each volume in the graph a tier-dht xlator will be created. So the graph parser will fail because of the redundant node in graph. By this change tier-dht will be renamed as volname-tier-dht Change-Id: I3c9b9c23ddcb853773a8a02be7fd8a5d09a7f972 BUG: 1222840 Signed-off-by: Mohammed Rafi KC <rkavunga@redhat.com> Reviewed-on: http://review.gluster.org/10820 Reviewed-by: Atin Mukherjee <amukherj@redhat.com> Tested-by: Gluster Build System <jenkins@build.gluster.com> Tested-by: NetBSD Build System Reviewed-by: Kaushal M <kaushal@redhat.com>
* cli/tiering: Enhance cli output for tieringMohammed Rafi KC2015-05-081-1/+1
| | | | | | | | | | | | | | Fix for handling cli output for attach-tier and detach-tier Change-Id: I4d17f4b09612754fe1b8cec6c2e14927029b9678 BUG: 1211562 Signed-off-by: Mohammed Rafi KC <rkavunga@redhat.com> Reviewed-on: http://review.gluster.org/10284 Reviewed-by: Dan Lambright <dlambrig@redhat.com> Tested-by: Gluster Build System <jenkins@build.gluster.com> Tested-by: NetBSD Build System Reviewed-by: Vijay Bellur <vbellur@redhat.com>
* Tests: use a portable way to flush kernel cacheEmmanuel Dreyfus2015-05-071-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | On Linux, kernel cache can be flushed using echo 3 > /proc/sys/vm/drop_caches This non-portable approach can be replaced by an on-purpose failed attempt to unmount: if the mount point is the current directory and umount is called, the kernel will flush inodes until it realize it cannot complete the operation because root of filesystem is busy: ( cd $M0 ; umount $M0 ) Unfortunately this does not flush everything. Entries may still be present in the kenrel FUSE cache. Using $GFS to mount the filesystem ensure --entry-timeout=0 and clears this problem. Some stall information may also remain in glusterfs caches, and that may have to be adressed by appropriate volume option. For instance tests/bugs/rpc/bug-954057.t needs to disable performance.stat-prefetch. Qtherwise, root's new credentials are not evaluated after root-quash is enabled. The test could also be done with performance.stat-prefetch enabled using various tricks: copying the file to read, creating a hard link on it, or just waiting long enough for metadata cache to expire. BUG: 1129939 Change-Id: I54929e899d55c04dcd9d947809133549f01fd0e1 Signed-off-by: Emmanuel Dreyfus <manu@netbsd.org> Reviewed-on: http://review.gluster.org/10411 Reviewed-by: Dan Lambright <dlambrig@redhat.com> Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Vijay Bellur <vbellur@redhat.com>
* ctr/xlator: Named lookup heal of pre-existing files, before ctr was ON.Joseph Fernandes2015-05-061-0/+72
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Problem: The CTR xlator records file meta (heat/hardlinks) into the data. This works fine for files which are created after ctr xlator is switched ON. But for files which were created before CTR xlator is ON, CTR xlator is not able to record either of the meta i.e heat or hardlinks. Thus making those files immune to promotions/demotions. Solution: The solution that is implemented in this patch is do ctr-db heal of all those pre-existent files, using named lookup. For this purpose we use the inode-xlator context variable option in gluster. The inode-xlator context variable for ctr xlator will have the following, a. A Lock for the context variable b. A hardlink list: This list represents the successful looked up hardlinks. These are the scenarios when the hardlink list is updated: 1) Named-Lookup: Whenever a named lookup happens on a file, in the wind path we copy all required hardlink and inode information to ctr_db_record structure, which resides in the frame->local variable. We dont update the database in wind. During the unwind, we read the information from the ctr_db_record and , Check if the inode context variable is created, if not we create it. Check if the hard link is there in the hardlink list. If its not there we add it to the list and send a update to the database using libgfdb. Please note: The database transaction can fail(and we ignore) as there already might be a record in the db. This update to the db is to heal if its not there. If its there in the list we ignore it. 2) Inode Forget: Whenever an inode forget hits we clear the hardlink list in the inode context variable and delete the inode context variable. Please note: An inode forget may happen for two reason, a. when the inode is delete. b. the in-memory inode is evicted from the inode table due to cache limits. 3) create: whenever a create happens we create the inode context variable and add the hardlink. The database updation is done as usual by ctr. 4) link: whenever a hardlink is created for the inode, we create the inode context variable, if not present, and add the hardlink to the list. 5) unlink: whenever a unlink happens we delete the hardlink from the list. 6) mknod: same as create. 7) rename: whenever a rename happens we update the hardlink in list. if the hardlink was not present for updation, we add the hardlink to the list. What is pending: 1) This solution will only work for named lookups. 2) We dont track afr-self-heal/dht-rebalancer traffic for healing. Change-Id: Ia4bbaf84128ad6ce8c3ddd70bcfa82894c79585f BUG: 1212037 Signed-off-by: Joseph Fernandes <josferna@redhat.com> Signed-off-by: Dan Lambright <dlambrig@redhat.com> Reviewed-on: http://review.gluster.org/10370 Reviewed-by: Jeff Darcy <jdarcy@redhat.com> Tested-by: Gluster Build System <jenkins@build.gluster.com> Tested-by: NetBSD Build System Reviewed-by: Vijay Bellur <vbellur@redhat.com>
* tiering: Send both attach-tier and tier-start togetherMohammed Rafi KC2015-05-052-8/+77
| | | | | | | | | | | | After attaching tier, we have to start tier rebalance process. This patch is to trigger tier start along with attch-tier. Change-Id: I39380f95123f0087a82213ef263f9f33adcc5adc BUG: 1214222 Signed-off-by: Mohammed Rafi KC <rkavunga@redhat.com> Reviewed-on: http://review.gluster.org/10363 Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Dan Lambright <dlambrig@redhat.com>
* tiering/cli: Check replica count and bricks are proper or notMohammed Rafi KC2015-05-041-1/+2
| | | | | | | | | | | | | | Right now, attach-tier calls parsing function for add-brick. Add-brick does not have any check for brick count and replca count compatibility. Change-Id: I44ec13eadffc003a9ebf8c4eb0193df559933a68 BUG: 1215122 Signed-off-by: Mohammed Rafi KC <rkavunga@redhat.com> Reviewed-on: http://review.gluster.org/10428 Reviewed-by: Dan Lambright <dlambrig@redhat.com> Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Vijay Bellur <vbellur@redhat.com>
* glusterd: support for tier volumes 'detach start' and 'detach commit'Dan Lambright2015-04-221-10/+12
| | | | | | | | | | | | | | | | | | | | | These commands work in a manner analagous to rebalancing when removing a brick. The existing migration daemon detects "detach start" and switches to moving data off the hot tier. While in this state all lookups are directed to the cold tier. gluster v detach-tier <vol> start gluster v detach-tier <vol> commit The status and stop cli commands shall be submitted separately. Change-Id: I24fda5cc3ba74f5fb8aa9a3234ad51f18b80a8a0 BUG: 1205540 Signed-off-by: Dan Lambright <dlambrig@redhat.com> Signed-off-by: root <root@localhost.localdomain> Signed-off-by: Dan Lambright <dlambrig@redhat.com> Reviewed-on: http://review.gluster.org/10108 Reviewed-by: Kaleb KEITHLEY <kkeithle@redhat.com> Tested-by: NetBSD Build System
generated by cgit (git 2.34.1) at 2024-05-26 18:18:28 +0000