summaryrefslogtreecommitdiffstats
path: root/xlators/cluster/dht/src/tier.c
Commit message (Collapse)AuthorAgeFilesLines
* Tier: Avoiding stale entries from causing demotion to stophari gowtham2016-03-221-2/+27
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | back-port of : http://review.gluster.org/#/c/13501/ When the parent GFID is a stale entry, the lookup on this parent fails and this in turn fails the demotion process. This patch will make the stale entry error to be skipped. Situation for pargfid to be stale: Consider a folder from a tar file. Once the tar file is untared the files in the tar-file will start to demote. when the demotion is under progress, if we tend to delete the actual folder, then the files under it which are undergoing demotion will do a lookup on the parent which was deleted and become stale entry. This stale entry fails the Lookup and this will fail the demotion of the other files(not from tar) that are supposed to be demoted. >Change-Id: I3d47c32c4077526d477a25912b0135bab98b23fc >BUG: 1311178 >Signed-off-by: hari gowtham <hgowtham@redhat.com> >Reviewed-on: http://review.gluster.org/13501 >Tested-by: hari gowtham <hari.gowtham005@gmail.com> >Smoke: Gluster Build System <jenkins@build.gluster.com> >CentOS-regression: Gluster Build System <jenkins@build.gluster.com> >Reviewed-by: Dan Lambright <dlambrig@redhat.com> >NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org> Change-Id: I80a77d149180e2a05b3d7943f370b3ef162e545b BUG: 1317366 Signed-off-by: Hari Gowtham <hgowtham@redhat.com> Reviewed-on: http://review.gluster.org/13692 Smoke: Gluster Build System <jenkins@build.gluster.com> Tested-by: hari gowtham <hari.gowtham005@gmail.com> NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org> CentOS-regression: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Dan Lambright <dlambrig@redhat.com>
* TIER: stopping the tierd when the volume goes downhari gowtham2016-03-221-0/+9
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | back-port of : http://review.gluster.org/#/c/13646/ If there are large number of files to be migrated and by this time if the volume goes down, then the tierd has to be stopped. But on a huge query file list it keeps checking for each file before stopping. If the volume comes up before the old tierd dies then due to the presence of old tierd new one won't be created. After the old one completes the task, it dies and the status ends up as failed. This patch will check if the status is still running and then let it continue its work. Else it will stop running the tierd. >Change-Id: I6522a4e2919e84bf502b99b13873795b9274f3cd >BUG: 1315659 >Signed-off-by: hari gowtham <hgowtham@redhat.com> >Reviewed-on: http://review.gluster.org/13646 >Tested-by: Dan Lambright <dlambrig@redhat.com> >Smoke: Gluster Build System <jenkins@build.gluster.com> >NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org> >CentOS-regression: Gluster Build System <jenkins@build.gluster.com> >Reviewed-by: Dan Lambright <dlambrig@redhat.com> Change-Id: I8326dbe5edaaea921e5401f39d148aac322c78d0 BUG: 1318498 Signed-off-by: hari <hgowtham@redhat.com> Reviewed-on: http://review.gluster.org/13756 Smoke: Gluster Build System <jenkins@build.gluster.com> Tested-by: hari gowtham <hari.gowtham005@gmail.com> NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org> CentOS-regression: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Dan Lambright <dlambrig@redhat.com>
* cluster/tier: break out of iterating query file once cycle time endsDan Lambright2016-03-161-0/+27
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | When iterating the query file during migration, tiering should break out of the loop once cycle time completes. Otherwise it may be possible to stay in the loop for a long time. If that happens updates to files will become stale and have not impact migration. This is a backport of 13284 > Change-Id: Ib60cf74bc84e8646e6a0da21ff04954b1b83c414 > BUG: 1301227 > Signed-off-by: Dan Lambright <dlambrig@redhat.com> > Reviewed-on: http://review.gluster.org/13284 > Smoke: Gluster Build System <jenkins@build.gluster.com> > NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org> > CentOS-regression: Gluster Build System <jenkins@build.gluster.com> Signed-off-by: Dan Lambright <dlambrig@redhat.com> Change-Id: Ia305784b0aa570b70e54f532bf5d68a91b0cf305 BUG: 1318203 Reviewed-on: http://review.gluster.org/13750 Smoke: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: hari gowtham <hari.gowtham005@gmail.com> Tested-by: Dan Lambright <dlambrig@redhat.com> NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org> CentOS-regression: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Dan Lambright <dlambrig@redhat.com>
* cluster/tier: Additional details in error messagesN Balachandran2016-03-071-37/+41
| | | | | | | | | | | | | | | | | | | | | | | | | Added file path/gfid when available to the tier log messages to make debugging easier. > Change-Id: I22dda329367df2b846dcf254594312c997b66083 > BUG: 1273043 > Signed-off-by: N Balachandran <nbalacha@redhat.com> > Reviewed-on: http://review.gluster.org/13114 > Reviewed-by: mohammed rafi kc <rkavunga@redhat.com> > Tested-by: Gluster Build System <jenkins@build.gluster.com> > Reviewed-by: Dan Lambright <dlambrig@redhat.com> > Tested-by: Dan Lambright <dlambrig@redhat.com> Change-Id: I5cbf8407fb7c493c4be1dda65c0f722ede33487f BUG: 1311865 Signed-off-by: N Balachandran <nbalacha@redhat.com> Reviewed-on: http://review.gluster.org/13518 Smoke: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: mohammed rafi kc <rkavunga@redhat.com> Reviewed-by: Dan Lambright <dlambrig@redhat.com> CentOS-regression: Gluster Build System <jenkins@build.gluster.com> NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org> Tested-by: Dan Lambright <dlambrig@redhat.com>
* cluster/tier: Create linkfiles to hardlinks correctlyN Balachandran2016-02-271-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | There is a bug in the way hardlinks are handled in tiered volumes. Ideally, the tier linkto files on the cold tier to files that are hardlinks to each other on the hot tier, should themselves be hardlinks of each other. As they are not, they end up being files with the same gfid but different names for the cold tier dht, and end up overwriting the cached-subvol information stored in the dht inode-ctx. > Change-Id: Ic658a316836e6a1729cfea848b7d212674b0edd2 > BUG: 1305277 > Signed-off-by: N Balachandran <nbalacha@redhat.com> > Reviewed-on: http://review.gluster.org/13391 > Smoke: Gluster Build System <jenkins@build.gluster.com> > NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org> > CentOS-regression: Gluster Build System <jenkins@build.gluster.com> > Reviewed-by: Dan Lambright <dlambrig@redhat.com> > Tested-by: Dan Lambright <dlambrig@redhat.com> (cherry picked from commit eb362c74db84d95aac07febf0d888bd98b3fb2b9) Change-Id: I5dd98e6d248619147974a630d7d72e1942a1cc83 BUG: 1311836 Signed-off-by: N Balachandran <nbalacha@redhat.com> Reviewed-on: http://review.gluster.org/13517 Smoke: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: mohammed rafi kc <rkavunga@redhat.com> NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org> CentOS-regression: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Dan Lambright <dlambrig@redhat.com>
* cluster/tier : Fixed wrong variable comparisonN Balachandran2016-02-271-2/+15
| | | | | | | | | | | | | | | | | | | | | | | | | The wrong variable was being checked to determine the watermark value. > Change-Id: If4c97fa70b772187f1fcbdf5193e077cb356a8b1 > BUG: 1303895 > Signed-off-by: N Balachandran <nbalacha@redhat.com> > Reviewed-on: http://review.gluster.org/13357 > Smoke: Gluster Build System <jenkins@build.gluster.com> > Reviewed-by: Dan Lambright <dlambrig@redhat.com> > NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org> > CentOS-regression: Gluster Build System <jenkins@build.gluster.com> Signed-off-by: N Balachandran <nbalacha@redhat.com> Change-Id: I0a98e0efbc093a727912107038477239e6d85765 BUG: 1306129 Reviewed-on: http://review.gluster.org/13516 Smoke: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: mohammed rafi kc <rkavunga@redhat.com> Tested-by: mohammed rafi kc <rkavunga@redhat.com> NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org> CentOS-regression: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Dan Lambright <dlambrig@redhat.com>
* tier:delete the linkfile if data file creation failsMohammed Rafi KC2016-02-221-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | If we are creating data file in a hot subvolume then we will create a linkfile in cold subvolume. Linkfile creation happens first. If linkfile creation was successful and data file creation failed, then linkfile in cold subvolume will become stale This patch will delete the linkfile as well, if data file creation fails Also this code duplicates dht_create to make tier_create backport of> >Change-Id: I377a90dad47f288e9576c7323b23cf694a91a7a3 >BUG: 1290677 >Signed-off-by: Mohammed Rafi KC <rkavunga@redhat.com> >Reviewed-on: http://review.gluster.org/12948 >Reviewed-by: N Balachandran <nbalacha@redhat.com> >Tested-by: NetBSD Build System <jenkins@build.gluster.org> >Reviewed-by: Raghavendra G <rgowdapp@redhat.com> >Reviewed-by: Dan Lambright <dlambrig@redhat.com> >Tested-by: Dan Lambright <dlambrig@redhat.com> Change-Id: I3689e8ee387fb6731b4b3a7fe8f8190143482eaa BUG: 1295359 Signed-off-by: Mohammed Rafi KC <rkavunga@redhat.com> Reviewed-on: http://review.gluster.org/13161 Smoke: Gluster Build System <jenkins@build.gluster.com> NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org> CentOS-regression: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Dan Lambright <dlambrig@redhat.com>
* tier:unlink during migrationMohammed Rafi KC2016-02-221-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | files deleted during promotion were not deleting as the files are moving from hashed to non-hashed On deleting a file that is undergoing promotion, the unlink call is not sent to the dst file as the hashed subvol == cached subvol. This causes the file to reappear once the migration is complete. This patch also fixes a problem with stale linkfile deleting. Backport of> >Change-Id: I4b02a498218c9d8eeaa4556fa4219e91e7fa71e5 >BUG: 1282390 >Signed-off-by: Mohammed Rafi KC <rkavunga@redhat.com> >Reviewed-on: http://review.gluster.org/12829 >Tested-by: NetBSD Build System <jenkins@build.gluster.org> >Tested-by: Gluster Build System <jenkins@build.gluster.com> >Reviewed-by: Dan Lambright <dlambrig@redhat.com> >Tested-by: Dan Lambright <dlambrig@redhat.com> (cherry picked from commit b5de382afa8c5777e455c7a376fc4f1f01d782d1) Change-Id: I951adb4d929926bcd646dd7574f7a2d41d57479d BUG: 1282388 Signed-off-by: Mohammed Rafi KC <rkavunga@redhat.com> Reviewed-on: http://review.gluster.org/12991 Smoke: Gluster Build System <jenkins@build.gluster.com> NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org> CentOS-regression: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Dan Lambright <dlambrig@redhat.com>
* cluster/tier : Reset watermarks in tierN Balachandran2016-02-191-9/+36
| | | | | | | | | | | | | | | | | | | | | | | | | | A node which contains only cold bricks and has detected that the high watermark value has been breached on the hot tier will never reset the watermark to the correct value. The promotion check will thus always fail and no promotions will occur from that node. > Change-Id: I0f0804744cd184c263acbea1ee50cd6010a49ec5 > BUG: 1303895 > Signed-off-by: N Balachandran <nbalacha@redhat.com> > Reviewed-on: http://review.gluster.org/13341 > Smoke: Gluster Build System <jenkins@build.gluster.com> > NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org> > CentOS-regression: Gluster Build System <jenkins@build.gluster.com> > Reviewed-by: Dan Lambright <dlambrig@redhat.com> (cherry picked from commit 545f4ed2c7195a21210e6a055c27c1b7a115e18c) Change-Id: Iba3aa9c57cf5828ab87140c2c8257146a8772836 BUG: 1306129 Signed-off-by: N Balachandran <nbalacha@redhat.com> Reviewed-on: http://review.gluster.org/13411 Smoke: Gluster Build System <jenkins@build.gluster.com> CentOS-regression: Gluster Build System <jenkins@build.gluster.com> NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org> Reviewed-by: Dan Lambright <dlambrig@redhat.com>
* cluster/tier: allow db queries to be interruptableDan Lambright2016-02-181-58/+95
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | A query to the database may take a long time if the database has many entries. The tier daemon also sends IPC calls to the bricks which can run slowly, espcially in RHEL6. While it is possible to track down each such instance, the snapshot feature should not be affected by database operations. It requires no migration be underway. Therefore it is okay to pause tiering at any time except when DHT is moving a file. This fix implements this strategy by monitoring when control passes to DHT to migrate a file using the GF_XATTR_FILE_MIGRATE_KEY trigger. If it is not, the pause operation is successful. > Change-Id: I21f168b1bd424077ad5f38cf82f794060a1fabf6 > BUG: 1287842 > Signed-off-by: Dan Lambright <dlambrig@redhat.com> > Reviewed-on: http://review.gluster.org/13104 > Reviewed-by: Joseph Fernandes > Tested-by: Gluster Build System <jenkins@build.gluster.com> Signed-off-by: Dan Lambright <dlambrig@redhat.com> Change-Id: I667e0af24eaa66afefa860c4d73b324e4f39b997 BUG: 1288352 Signed-off-by: Dan Lambright <dlambrig@redhat.com> Reviewed-on: http://review.gluster.org/13199 Smoke: Gluster Build System <jenkins@build.gluster.com> NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org> CentOS-regression: Gluster Build System <jenkins@build.gluster.com>
* tier/gfdb : Round-Robin read of query filesJoseph Fernandes2016-02-141-77/+308
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | 1. Each brick on a host will get a separate query file. 2. While reading query record from these query files we read them in a Round-Robin manner. 3. When an error occurs during migration we rename it to query file with an time stamp and .err extension for better debugging. Backport of http://review.gluster.org/13293 > Change-Id: I27c4285d24fd695d2d5cbd9fd7db3879d277ecc8 > BUG: 1302772 > Signed-off-by: Joseph Fernandes <josferna@redhat.com> > Reviewed-on: http://review.gluster.org/13293 > Smoke: Gluster Build System <jenkins@build.gluster.com> > Tested-by: N Balachandran <nbalacha@redhat.com> > NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org> > Reviewed-by: Dan Lambright <dlambrig@redhat.com> > CentOS-regression: Gluster Build System <jenkins@build.gluster.com> Change-Id: I9448f19c10a82fff9b1b3c9387c9026adc0742f1 BUG: 1306514 Signed-off-by: Joseph Fernandes <josferna@redhat.com> Reviewed-on: http://review.gluster.org/13427 Smoke: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Joseph Fernandes Tested-by: Joseph Fernandes NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org> CentOS-regression: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Dan Lambright <dlambrig@redhat.com>
* tier/dht : Default value for demote-freq, max files and mbJoseph Fernandes2016-02-141-2/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Default value for tier-demote-frequency is 3600 sec to avoid frequent demotions. Default value for tier-max-mb is 4000 mb Default value for tier-max-files is 10000 files Backport of http://review.gluster.org/13270 > Change-Id: Ie60951c478a7462c425059699ab82511aa13fa0a > BUG: 1300412 > Signed-off-by: Joseph Fernandes <josferna@redhat.com> > Reviewed-on: http://review.gluster.org/13270 > Smoke: Gluster Build System <jenkins@build.gluster.com> > Tested-by: Dan Lambright <dlambrig@redhat.com> > NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org> > CentOS-regression: Gluster Build System <jenkins@build.gluster.com> > Reviewed-by: Dan Lambright <dlambrig@redhat.com> Signed-off-by: Joseph Fernandes <josferna@redhat.com> Change-Id: Ic4458b1b78dc3c6059713f3871e9e2dc8f38b39a BUG: 1306302 Reviewed-on: http://review.gluster.org/13424 Smoke: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Joseph Fernandes Tested-by: Joseph Fernandes NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org> CentOS-regression: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Dan Lambright <dlambrig@redhat.com>
* cluster/tier: Ignore quota-deem-statfs for watermark calculationN Balachandran2016-01-311-0/+21
| | | | | | | | | | | | | | | | | | | | | | | | | | The tier process watermark calculations were incorrect when the quota-deem-statfs option was enabled. We now ignore this while calculating hot tier usage to determine watermark levels. Change-Id: I308bc24432e2fa5ad1d5703e80fc391433538bbb BUG: 1302012 > Signed-off-by: N Balachandran <nbalacha@redhat.com> > Reviewed-on: http://review.gluster.org/13288 > Smoke: Gluster Build System <jenkins@build.gluster.com> > Reviewed-by: mohammed rafi kc <rkavunga@redhat.com> > Tested-by: mohammed rafi kc <rkavunga@redhat.com> > CentOS-regression: Gluster Build System <jenkins@build.gluster.com> > Reviewed-by: Dan Lambright <dlambrig@redhat.com> > NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org> > Tested-by: Dan Lambright <dlambrig@redhat.com> (cherry picked from commit 9ce86c63e349eaafbd923b0b541d75e929ecbae7) Signed-off-by: Nithya Balachandran <nbalacha@redhat.com> Reviewed-on: http://review.gluster.org/13292 Reviewed-by: Pranith Kumar Karampuri <pkarampu@redhat.com> Tested-by: Pranith Kumar Karampuri <pkarampu@redhat.com> Smoke: Gluster Build System <jenkins@build.gluster.com> CentOS-regression: Gluster Build System <jenkins@build.gluster.com> NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org>
* cluster/dht : Ftruncate on migrating file fails with EINVALN Balachandran2016-01-291-0/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | What: If dht_open is called on a migrating file after the inode_ctx is set, subsequent FOPs on that fd do not open the fd on the dst subvol. This is seen when the open-ftruncate-close sequence is repeatedly called on a migrating file. A second call to the sequence described above causes dht_truncate_cbk to call dht_truncate2 as the dht_inode_ctx was already set by the first call. As dht_rebalance_in_progress_check is not called, the fd is not opened on the dst subvol. On a distributed-replicate volume, this causes AFR to open the fd using afr_fix_open, but with the wrong flags, causing posix_ftruncate to fail with EINVAL. The fix: We require fd specific information to make a decision while handling migrating files. Set the fd_ctx to indicate the fd has been opened on the dst subvol and check if it has been set while processing Phase1/Phase2 checks in the FOP callback functions. > Change-Id: I43cdcd8017b4a11e18afdd210469de7cd9a5ef14 > Signed-off-by: N Balachandran <nbalacha@redhat.com> > Reviewed-on: http://review.gluster.org/12985 > Reviewed-by: Raghavendra G <rgowdapp@redhat.com> > Tested-by: Gluster Build System <jenkins@build.gluster.com> > Reviewed-by: Dan Lambright <dlambrig@redhat.com> > Tested-by: Dan Lambright <dlambrig@redhat.com> Change-Id: I99f8aceec105f16631def06a263f0561954d14b3 BUG: 1293827 Signed-off-by: N Balachandran <nbalacha@redhat.com> Reviewed-on: http://review.gluster.org/13071 Reviewed-by: Pranith Kumar Karampuri <pkarampu@redhat.com> Tested-by: Pranith Kumar Karampuri <pkarampu@redhat.com> Smoke: Gluster Build System <jenkins@build.gluster.com> NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org> CentOS-regression: Gluster Build System <jenkins@build.gluster.com>
* all: reduce "inline" usageKaleb S KEITHLEY2016-01-181-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | There are three kinds of inline functions: plain inline, extern inline, and static inline. All three have been removed from .c files, except those in "contrib" which aren't our problem. Inlines in .h files, which are overwhelmingly "static inline" already, have generally been left alone. Over time we should be able to "lower" these into .c files, but that has to be done in a case-by-case fashion requiring more manual effort. This part was easy to do automatically without (as far as I can tell) any ill effect. In the process, several pieces of dead code were flagged by the compiler, and were removed. backport of Change-Id: I56a5e614735c9e0a6ee420dab949eac22e25c155, http://review.gluster.org/11769, BUG: 1245331 Change-Id: Iba1efb0bc578ea4a5e9bf76b7bd93dc1be9eba44 BUG: 1283302 Signed-off-by: Kaleb S KEITHLEY <kkeithle@redhat.com> Reviewed-on: http://review.gluster.org/12646 Smoke: Gluster Build System <jenkins@build.gluster.com> NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org> CentOS-regression: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Niels de Vos <ndevos@redhat.com>
* cluster/tier: do not block in synctask created from pause tierDan Lambright2015-12-251-5/+49
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | We had run sleep() in the pause tier callback. Blocking within a synctask is dangerous. The sleep() call does not inform the synctask scheduler that a thread is no longer running. It therefore believes it is running. If a second synctask already exists, it may not be able to run. This occurs if the thread limit in the pool has been reached. Note the pool size only grows when a synctask is created, not when it is moved from wait state to run state, as is the case when an FOP completes. When the tier is paused during migration, synctasks already exist waiting for responses to FOPs to the server with high probability. The fix is to yield() in the RPC callback, which will place the synctask into the wait queue and free up a thread for the FOP callback. A timer wakes the callback after sufficient time has elapsed for the pause to occur. This is a backport of 12987. > Change-Id: I6a947ee04c6e5649946cb6d8207ba17263a67fc6 > BUG: 1267950 > Signed-off-by: Dan Lambright <dlambrig@redhat.com> > Reviewed-on: http://review.gluster.org/12987 > Tested-by: Gluster Build System <jenkins@build.gluster.com> > Reviewed-by: Rajesh Joseph <rjoseph@redhat.com> Signed-off-by: Dan Lambright <dlambrig@redhat.com> Change-Id: I9bb7564d57d59abb98e89c8d54c8b1ff4a5fc3f3 BUG: 1274100 Reviewed-on: http://review.gluster.org/13078 Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Dan Lambright <dlambrig@redhat.com> Tested-by: Dan Lambright <dlambrig@redhat.com>
* cluster/tier: fix tier-max-files bookeeping and helpDan Lambright2015-12-201-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | The option tier-max-files represents the maximum number of files trasferred by a node in a gives cycle. Fix help message to reflect the "per node" aspect. The code transferred one more file per cycle than the given value. Also change the default values of max file and max bytes to very large values, effectively we will not throttle migration unless the administrator requests it via CLI. This is a backport of 12984. > Change-Id: Ic2949ed3d8c35afe7c9ae4db72195603cfb2e28f > BUG: 1292671 > Signed-off-by: Dan Lambright <dlambrig@redhat.com> > Reviewed-on: http://review.gluster.org/12984 > Tested-by: NetBSD Build System <jenkins@build.gluster.org> > Tested-by: Gluster Build System <jenkins@build.gluster.com> Signed-off-by: Dan Lambright <dlambrig@redhat.com> Change-Id: Ibb234976f912ff4b20e894f2984c63792acfe814 BUG: 1292945 Reviewed-on: http://review.gluster.org/13004 Tested-by: NetBSD Build System <jenkins@build.gluster.org> Reviewed-by: Dan Lambright <dlambrig@redhat.com> Tested-by: Dan Lambright <dlambrig@redhat.com>
* tier : Spawn promotion or demotion thread depending on local bricksJoseph Fernandes2015-12-091-22/+29
| | | | | | | | | | | | | | | | | | | | | | | | | | | Spawn Promotion or Demotion depending if there are any Cold or Hot bricks present localy. IF the local HOT brick list is empty dont spawn demote thread. IF the local COLD brick list is empty dont spawn promote thread. Backport of http://review.gluster.org/12912 > Change-Id: I524730e59414dd156c78ec0bd7a3629212697e6e > BUG: 1289578 > Signed-off-by: Joseph Fernandes <josferna@redhat.com> > Reviewed-on: http://review.gluster.org/12912 > Tested-by: NetBSD Build System <jenkins@build.gluster.org> > Reviewed-by: N Balachandran <nbalacha@redhat.com> > Tested-by: Gluster Build System <jenkins@build.gluster.com> > Reviewed-by: Dan Lambright <dlambrig@redhat.com> > Tested-by: Dan Lambright <dlambrig@redhat.com> Signed-off-by: Joseph Fernandes <josferna@redhat.com> Change-Id: Ic3113934051c7a751ae56508e00d098d010f4c0e BUG: 1290048 Reviewed-on: http://review.gluster.org/12928 Tested-by: NetBSD Build System <jenkins@build.gluster.org> Reviewed-by: Dan Lambright <dlambrig@redhat.com> Tested-by: Dan Lambright <dlambrig@redhat.com>
* tier/dht: Fix mem leak from lookup response dictJoseph Fernandes2015-12-091-0/+6
| | | | | | | | | | | | | | | | | | | | | | | | Fixing memory leak from response dict during a parent lookup to get the path. Backport of http://review.gluster.org/12867 > Change-Id: I60c23d0b25e7f763f0e53c40e71ee053aba6d555 > BUG: 1288019 > Signed-off-by: Joseph Fernandes <josferna@redhat.com> > Reviewed-on: http://review.gluster.org/12867 > Tested-by: Gluster Build System <jenkins@build.gluster.com> > Reviewed-by: Joseph Fernandes > Tested-by: Joseph Fernandes Signed-off-by: Joseph Fernandes <josferna@redhat.com> Change-Id: I77dfbb097e0200ace4c1adcf48560caf283007d7 BUG: 1288992 Reviewed-on: http://review.gluster.org/12889 Tested-by: NetBSD Build System <jenkins@build.gluster.org> Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Dan Lambright <dlambrig@redhat.com> Tested-by: Dan Lambright <dlambrig@redhat.com>
* cluster/tier : Fix double free in tier processN Balachandran2015-12-081-3/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | The tier process tries to free ipc_ctr_params twice if the syncop_ipc call in tier_process_ctr_query fails. ipc_ctr_params is freed when ctr_ipc_in_dict is freed. But ctr_ipc_out_dict is NULL when syncop_ipc fails, causing GF_FREE to be called on a non-NULL ipc_ctr_params ptr again. > Change-Id: Ia15f36dfbcd97be5524588beb7caad5cb79efdb4 > Signed-off-by: N Balachandran <nbalacha@redhat.com> > Reviewed-on: http://review.gluster.org/12890 > Reviewed-by: Joseph Fernandes > Tested-by: NetBSD Build System <jenkins@build.gluster.org> > Tested-by: Gluster Build System <jenkins@build.gluster.com> > Reviewed-by: Dan Lambright <dlambrig@redhat.com> > (cherry picked from commit 06818a0fd69bb0d6daabde73e5c3cc2661a70854) Change-Id: Ida2da0416272ff1b04cf51f76467e27b62121f41 BUG: 1289414 Signed-off-by: N Balachandran <nbalacha@redhat.com> Reviewed-on: http://review.gluster.org/12904 Reviewed-by: Joseph Fernandes Tested-by: Gluster Build System <jenkins@build.gluster.com> Tested-by: NetBSD Build System <jenkins@build.gluster.org> Reviewed-by: Dan Lambright <dlambrig@redhat.com> Tested-by: Dan Lambright <dlambrig@redhat.com>
* tier/tier: Ignoring status of already migrated filesJoseph Fernandes2015-12-061-5/+16
| | | | | | | | | | | | | | | | | | | | | | | Ignore the status of already migrated files and in the process don't count. Backport http://review.gluster.org/12758 > Change-Id: Idba6402508d51a4285ac96742c6edf797ee51b6a > BUG: 1276141 > Signed-off-by: Joseph Fernandes <josferna@redhat.com> > Reviewed-on: http://review.gluster.org/12758 > Tested-by: Gluster Build System <jenkins@build.gluster.com> > Reviewed-by: Dan Lambright <dlambrig@redhat.com> > Tested-by: Dan Lambright <dlambrig@redhat.com> Signed-off-by: Joseph Fernandes <josferna@redhat.com> Change-Id: I44b7b965ecbc34159c2233af1a74762fd410dcaf BUG: 1262860 Reviewed-on: http://review.gluster.org/12884 Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Dan Lambright <dlambrig@redhat.com> Tested-by: Dan Lambright <dlambrig@redhat.com>
* cluster/tier: fix loading tier.so into glusterdN Balachandran2015-12-041-8/+34
| | | | | | | | | | | | | | | | | | | | | | | | | | The glusterd process loads the shared libraries of client translators. This failed for tiering due to a reference to dht_methods which is defined as a global variable which is not necessary. The global variable has been removed and this is now a member of dht_conf and is now initialised in the *_init calls. > Change-Id: Ifa0a21e3962b5cd8d9b927ef1d087d3b25312953 > Signed-off-by: N Balachandran <nbalacha@redhat.com> > Reviewed-on: http://review.gluster.org/12863 > Tested-by: NetBSD Build System <jenkins@build.gluster.org> > Tested-by: Gluster Build System <jenkins@build.gluster.com> > Reviewed-by: Dan Lambright <dlambrig@redhat.com> >Tested-by: Dan Lambright <dlambrig@redhat.com> (cherry picked from commit 96fc7f64da2ef09e82845a7ab97574f511a9aae5) Change-Id: If3cc908ebfcd1f165504f15db2e3079d97f3132e BUG: 1288352 Signed-off-by: N Balachandran <nbalacha@redhat.com> Reviewed-on: http://review.gluster.org/12877 Tested-by: NetBSD Build System <jenkins@build.gluster.org> Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Dan Lambright <dlambrig@redhat.com> Tested-by: Dan Lambright <dlambrig@redhat.com>
* cluster/tier: readdirp to cold tier onlyDan Lambright2015-11-241-41/+6
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | It is possible a file would get migrated in the middle of a readdir operation. If there are four subvolumes A,B,C,D, and if readdir reads them in order and reaches subvol B, then, if a file is moved from D to A, it will not be included in the readdir output. This phenonema has pre-existed in DHT migration but is more apparent in tiering. When a file is moved off the hashed subvolume a T file is created. For tiering, we will make the cold subvolume the hashed subvolume. This will ensure the creation of a T file. Readdir will not skip T files in the tier translator. Making the cold subvolume the hashed subvolume ensures the T files created on promotions or creates will be less likely to fill the volume. Creates still put the data on the hot subvolume. This is a backport of 12530 > Change-Id: Ifde557d3d0e94a4570ca9f115adee3db2ee75407 > BUG: 1281598 > Signed-off-by: Dan Lambright <dlambrig@redhat.com> > Reviewed-on: http://review.gluster.org/12530 > Tested-by: Gluster Build System <jenkins@build.gluster.com> > Tested-by: NetBSD Build System <jenkins@build.gluster.org> > Reviewed-by: N Balachandran <nbalacha@redhat.com> > Reviewed-by: Raghavendra G <rgowdapp@redhat.com> Signed-off-by: Dan Lambright <dlambrig@redhat.com> Signed-off-by: Dan Lambright <dlambrig@redhat.com> Conflicts: xlators/cluster/dht/src/tier.c Change-Id: I5720a4cd04ae5088e5d7d23439b0f90d6bbc6265 BUG: 1283923 Reviewed-on: http://review.gluster.org/12722 Tested-by: NetBSD Build System <jenkins@build.gluster.org> Reviewed-by: N Balachandran <nbalacha@redhat.com> Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Dan Lambright <dlambrig@redhat.com> Tested-by: Dan Lambright <dlambrig@redhat.com>
* cluster/tier: Disallow detach commit when detach in progressDan Lambright2015-11-161-1/+1
| | | | | | | | | | | | | 1. Check if detach is running, disallow detach commit if so. 2. Cleanup shutdown of tier daemon on detach: do not rerun fix-layout, do not send incorrect status back to glusterd. Change-Id: I97202f748773c1176396a4ffd32a4c7fa9b9c1bc BUG: 1264441 Signed-off-by: Dan Lambright <dlambrig@redhat.com> Reviewed-on: http://review.gluster.org/12272 Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Joseph Fernandes
* tier/libgfdb: Replacing ASCII query file with binaryJoseph Fernandes2015-11-081-191/+149
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Earlier, when the database was queried we used to save all the queried records in an ASCII format in the query file. This caused issues like filename having ASCII delimiter and used to take a lot of space. The tier.c file also had a lot of parsing code. Here we changed the format of the query file to binary. All the logic of serialization and formating of query record is done by libgfdb. Libgfdb provides API, gfdb_write_query_record() and gfdb_read_query_record(), which the user i.e tier migrator and CTR xlator can use to write to and read from query file. With this binary format we save on disk space i.e reduce to 50% atleast as we are saving GFID's in binary format 16 bytes and not the string format which takes 36 bytes + We are not saving path of the file + we are also saving on ASCII delimiters. The on disk format of query record is as follows, +---------------------------------------------------------------------------+ | Length of serialized query record | Serialized Query Record | +---------------------------------------------------------------------------+ 4 bytes Length of serialized query record | | -------------------------------------------------| | | V Serialized Query Record Format: +---------------------------------------------------------------------------+ | GFID | Link count | <LINK INFO> |..... | FOOTER | +---------------------------------------------------------------------------+ 16 B 4 B Link Length 4 B | | | | -----------------------------| | | | | | V | Each <Link Info> will be serialized as | +-----------------------------------------------+ | | PGID | BASE_NAME_LENGTH | BASE_NAME | | +-----------------------------------------------+ | 16 B 4 B BASE_NAME_LENGTH | | | ------------------------------------------------------------------------| | | V FOOTER is a magic number 0xBAADF00D indicating the end of the record. This also serves as a serialized schema validator. Backport of http://review.gluster.org/#/c/12354/ > Change-Id: I9db7416fd421e118dd44eafab8b535caafe50d5a > BUG: 1272207 > Signed-off-by: Joseph Fernandes <josferna@redhat.com> > Reviewed-on: http://review.gluster.org/12354 > Reviewed-by: N Balachandran <nbalacha@redhat.com> > Tested-by: Gluster Build System <jenkins@build.gluster.com> > Reviewed-by: Dan Lambright <dlambrig@redhat.com> > Tested-by: Dan Lambright <dlambrig@redhat.com> Change-Id: I170c579027f2594a58706f826e3ddf89e34022f4 BUG: 1263619 Signed-off-by: Joseph Fernandes <josferna@redhat.com> Reviewed-on: http://review.gluster.org/12535 Tested-by: NetBSD Build System <jenkins@build.gluster.org> Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Dan Lambright <dlambrig@redhat.com> Tested-by: Dan Lambright <dlambrig@redhat.com>
* cluster/tier correct promotion cycle calculationDan Lambright2015-11-071-3/+9
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This is a backport of 12480 The tier translator should only choose candidate files for promotion from the most recent cycle, not a multiple of the most recent cycles. Otherwise user observed behavior can be inconsistent. Remove related test in tier.t that is subject to race condition. > Change-Id: I9ad1523cac00f904097ce468efa6ddd515857024 > BUG: 1275524 > Signed-off-by: root <root@rhs-cli-15.gdev.lab.eng.bos.redhat.com> > Signed-off-by: Dan Lambright <dlambrig@redhat.com> > Reviewed-on: http://review.gluster.org/12480 > Reviewed-by: Joseph Fernandes > Tested-by: Gluster Build System <jenkins@build.gluster.com> Signed-off-by: Dan Lambright <dlambrig@redhat.com> Signed-off-by: Dan Lambright <dlambrig@redhat.com> Conflicts: tests/basic/tier/tier.t xlators/cluster/dht/src/tier.c Change-Id: Ic4587bf1b5d26ba377a12a4ce8e329362988a33b BUG: 1275483 Reviewed-on: http://review.gluster.org/12536 Tested-by: NetBSD Build System <jenkins@build.gluster.org> Reviewed-by: Dan Lambright <dlambrig@redhat.com> Tested-by: Dan Lambright <dlambrig@redhat.com>
* tier/dht: Ignoring replica for migration countingJoseph Fernandes2015-11-051-7/+24
| | | | | | | | | | | | | | | | | | | | | We used to count replica files for migration counting even though they were ignore for migration as the replica brick didnt have the ownership (as per the replication xlator either AFR/EC). As a result the number of files migrated would show a wrong count, i.e each replicated file would be counted 1 + number of replica. This patch ignores such cases. Backport of http://review.gluster.org/#/c/12453/ Change-Id: Ib005fedaee16f171e0499782b91f3eeedf8860ed BUG: 1262860 Signed-off-by: Joseph Fernandes <josferna@redhat.com> Reviewed-on: http://review.gluster.org/12511 Tested-by: NetBSD Build System <jenkins@build.gluster.org> Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Dan Lambright <dlambrig@redhat.com> Tested-by: Dan Lambright <dlambrig@redhat.com>
* cluster/tier : Files skipped during tier query parsingN Balachandran2015-11-041-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | | The tier query parsing code was using fscanf to read each record. As space is a delimiter for fscanf, filenames containing spaces caused the parsing to return unexpected values causing various issues in the tier process, including crashes due to buffer overflows. > Change-Id: Ife602cb7ecb158fccbc2c89e4d2959bd97098a87 > Signed-off-by: N Balachandran <nbalacha@redhat.com> > Reviewed-on: http://review.gluster.org/12469 > Tested-by: NetBSD Build System <jenkins@build.gluster.org> > Tested-by: Gluster Build System <jenkins@build.gluster.com> > Reviewed-by: Dan Lambright <dlambrig@redhat.com> > Tested-by: Dan Lambright <dlambrig@redhat.com> (cherry picked from commit 499b43058049572e33b525ac669ef623d476fe41) Change-Id: Id60f9c484dfbb02de6ebb44032160ad4cc94cb7f BUG: 1277587 Signed-off-by: N Balachandran <nbalacha@redhat.com> Reviewed-on: http://review.gluster.org/12502 Tested-by: NetBSD Build System <jenkins@build.gluster.org> Reviewed-by: Dan Lambright <dlambrig@redhat.com> Tested-by: Dan Lambright <dlambrig@redhat.com>
* cluster/tier dont log error on lookup heal for files on hot tierDan Lambright2015-10-301-13/+14
| | | | | | | | | | | | | | | | | | | | | | | | | | | This is a backport of 12430 On fix-layout heal files are scanned. Files found are exist on the hot or cold subvolume. Those not found in the cold tier would exist on the hot. They should not be flagged as an error. Replace INFO with TRACE for common tier migration logs. Frequent migration was growing the log files too quickly. On migratation failures, do not acrue files towards cycle limit's budget. > Change-Id: Ie832ee07c43bce5477ae81c939d1fe8416a11615 > BUG: 1275383 > Signed-off-by: Dan Lambright <dlambrig@redhat.com> > Reviewed-on: http://review.gluster.org/12430 > Tested-by: Gluster Build System <jenkins@build.gluster.com> > Reviewed-by: Joseph Fernandes Signed-off-by: Dan Lambright <dlambrig@redhat.com> Change-Id: Ia1ce5c3ac9c8c43cf3f3f7e0bd6161aa13affe5f BUG: 1272398 Signed-off-by: Dan Lambright <dlambrig@redhat.com> Reviewed-on: http://review.gluster.org/12465 Tested-by: Gluster Build System <jenkins@build.gluster.com>
* cluster/tier: add pause tier for snapshotsDan Lambright2015-10-211-1/+67
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This is a backport of 12304 Snaps of tiered volumes cannot handle files undergoing migration. We implement a helper mechanism to "pause" migration. Any files undergoing migration are aborted. Clean up is done to remove sticky bits and data at the destination. Migration is restarted after snap completes. For testing an internal switch is added. It is not exposed externally. gluster volume set vol1 tier-pause [true|false] > Change-Id: Ia85bbf89ac142e9b7e73fcbef98bb9da86097799 > BUG: 1267950 > Signed-off-by: Dan Lambright <dlambrig@redhat.com> > Reviewed-on: http://review.gluster.org/12304 > Reviewed-by: N Balachandran <nbalacha@redhat.com> > Tested-by: NetBSD Build System <jenkins@build.gluster.org> > Tested-by: Gluster Build System <jenkins@build.gluster.com> Signed-off-by: Dan Lambright <dlambrig@redhat.com> Conflicts: xlators/mgmt/glusterd/src/glusterd-messages.h Change-Id: I5f039d8d38a4c915bd873969f336b96755a0b8f1 BUG: 1274101 Reviewed-on: http://review.gluster.org/12411 Tested-by: NetBSD Build System <jenkins@build.gluster.org> Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Dan Lambright <dlambrig@redhat.com> Tested-by: Dan Lambright <dlambrig@redhat.com>
* cluster/tier do not abort migration if a single brick is downDan Lambright2015-10-211-12/+12
| | | | | | | | | | | | | | | | | | | | | | | | backport fix 12397 When a bricks are down, promotion/demotion should still be possible. For example, if an EC brick is down, the other bricks are able to recover the data and migrate it. > Change-Id: I8e650c640bce22a3ad23d75c363fbb9fd027d705 > BUG: 1273215 > Signed-off-by: Dan Lambright <dlambrig@redhat.com> > Reviewed-on: http://review.gluster.org/12397 > Tested-by: NetBSD Build System <jenkins@build.gluster.org> > Tested-by: Gluster Build System <jenkins@build.gluster.com> > Reviewed-by: Joseph Fernandes Signed-off-by: Dan Lambright <dlambrig@redhat.com> Change-Id: I6688757eaf97426c8e1ea1038c598b34bf6b8ccc BUG: 1272334 Signed-off-by: Dan Lambright <dlambrig@redhat.com> Reviewed-on: http://review.gluster.org/12405 Tested-by: NetBSD Build System <jenkins@build.gluster.org> Tested-by: Gluster Build System <jenkins@build.gluster.com>
* cluster/tier remove suprious log messages on valid failed migrationDan Lambright2015-10-191-1/+8
| | | | | | | | | | | | | | | | | | | | | | | | | Backport fix 12391 > On a write to a replica volume, we record in all brick's databases an entry. > When the tier daemon runs, it will only move the file if it is the true > owner of the file as defined by the XATTR_NODE_UUID_KEY. > Change-Id: Ib82717f87a3f94f3d0d9f969773de9e88d6aaf22 > BUG: 1273043 > Signed-off-by: Dan Lambright <dlambrig@redhat.com> > Reviewed-on: http://review.gluster.org/12391 > Reviewed-by: Joseph Fernandes > Tested-by: NetBSD Build System <jenkins@build.gluster.org> > Tested-by: Gluster Build System <jenkins@build.gluster.com> Signed-off-by: Dan Lambright <dlambrig@redhat.com> Change-Id: I12147f878cd1927f845867fb7c0b84c4db017ee1 BUG: 1272398 Reviewed-on: http://review.gluster.org/12394 Reviewed-by: Joseph Fernandes Tested-by: NetBSD Build System <jenkins@build.gluster.org> Reviewed-by: Dan Lambright <dlambrig@redhat.com> Tested-by: Dan Lambright <dlambrig@redhat.com>
* cluster/tier: add watermarks and policy driverDan Lambright2015-10-101-85/+396
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Backport fix 12039 This fix introduces infrastructure to support different policies for promotion and demotion. Currently the tier feature automatically promotes and demotes files periodically based on access. This is good for testing but too stringent for most real workloads. It makes it difficult to fully utilize a hot tier- data will be demoted before it is touched- its unlikely a 100GB hot SSD will have all its data touched in a window of time. A new parameter "mode" allows the user to pick promotion/demotion polcies. The "test mode" will be used for *.t and other general testing. This is the current mechanism. The "cache mode" introduces watermarks. The watermarks represent levels of data residing on the hot tier. "cache mode" policy: The % the hot tier is full is called P. Do not promote or demote more than D MB or F files. A random number [0-100] is called R. Rules for migration: if (P < watermark_low) don't demote, always promote. if (P >= watermark_low) && (P < watermark_hi) demote if R < P; promote if R > P. if (P > watermark_hi) always demote, don't promote. gluster volume set {vol} cluster.watermark-hi % gluster volume set {vol} cluster.watermark-low % gluster volume set {vol} cluster.tier-max-mb {D} gluster volume set {vol} cluster.tier-max-files {F} gluster volume set {vol} cluster.tier-mode {test|cache} > Change-Id: I157f19667ec95aa1d53406041c1e3b073be127c2 > BUG: 1257911 > Signed-off-by: Dan Lambright <dlambrig@redhat.com> > Reviewed-on: http://review.gluster.org/12039 > Tested-by: Gluster Build System <jenkins@build.gluster.com> > Reviewed-by: Atin Mukherjee <amukherj@redhat.com> Signed-off-by: Dan Lambright <dlambrig@redhat.com> Signed-off-by: Dan Lambright <dlambrig@redhat.com> Conflicts: xlators/cluster/dht/src/dht-rebalance.c xlators/cluster/dht/src/tier.c Change-Id: Ibfe6b89563ceab98708325cf5d5ab0997c64816c BUG: 1270527 Reviewed-on: http://review.gluster.org/12330 Tested-by: NetBSD Build System <jenkins@build.gluster.org> Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Dan Lambright <dlambrig@redhat.com> Tested-by: Dan Lambright <dlambrig@redhat.com>
* tier/ctr: Solution for db locks for tier migrator and ctr using sqlite ↵Joseph Fernandes2015-10-091-18/+287
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | version less than 3.7 i.e rhel 6.7 Problem: On RHEL 6.7, we have sqlite version 3.6.2 which doesnt support WAL journaling mode, as this journaling mode is only available in sqlite 3.7 and above. As a result we cannot have to progreses concurrently accessing sqlite, without running into db locks! Well WAL is also need for performace on CTR side. Solution: This solution is to use CTR db connection for doing queries when WAL mode is absent. i,e tier migrator will send sync_op ipc calls to CTR, which in turn will do the query and create/update the query file suggested by tier migrator. Pending: Well this solution will stop the db locks but the performance is still an issue for CTR. We are developing an in-Memory Transaction Log (iMeTaL) which will help boost the CTR performance by doing in memory udpates on the IO path and later flush the updates to the db in a batch/segment flush. Master patch: http://review.gluster.org/#/c/12191 >> Change-Id: Ie3149643ded159234b5cc6aa6cf93b9022c2f124 >> BUG: 1240577 >> Signed-off-by: Joseph Fernandes <josferna@redhat.com> >> Signed-off-by: Dan Lambright <dlambrig@redhat.com> >> Signed-off-by: Joseph Fernandes <josferna@redhat.com> >> Reviewed-on: http://review.gluster.org/12191 >> Tested-by: Gluster Build System <jenkins@build.gluster.com> >> Reviewed-by: Luis Pabon <lpabon@redhat.com> Signed-off-by: Joseph Fernandes <josferna@redhat.com> Change-Id: Ie8c7a7e9566244c104531b579126bb57fbc6e32b BUG: 1270123 Reviewed-on: http://review.gluster.org/12325 Tested-by: NetBSD Build System <jenkins@build.gluster.org> Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Dan Lambright <dlambrig@redhat.com> Tested-by: Dan Lambright <dlambrig@redhat.com>
* cluster/tier: Fixed a crash in tieringNithya Balachandran2015-09-161-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | This is a backport of 12179 An incorrect check was causing the arguments to the promote thread to be cleared before the thread was done with them. This caused the process to crash when it tried to dereference a NULL pointer. > Change-Id: I8348309ef4dad33b7f648c7a2c2703487e401269 > BUG: 1263204 > Signed-off-by: Nithya Balachandran <nbalacha@redhat.com> > Reviewed-on: http://review.gluster.org/12179 > Tested-by: NetBSD Build System <jenkins@build.gluster.org> > Reviewed-by: Dan Lambright <dlambrig@redhat.com> > Reviewed-by: Joseph Fernandes Signed-off-by: Dan Lambright <dlambrig@redhat.com> Change-Id: I5cd4cb9978fc9d3a74f69ef75474fc3b593aadf0 BUG: 1263746 Reviewed-on: http://review.gluster.org/12187 Reviewed-by: N Balachandran <nbalacha@redhat.com> Tested-by: NetBSD Build System <jenkins@build.gluster.org> Tested-by: Gluster Build System <jenkins@build.gluster.com> Tested-by: Dan Lambright <dlambrig@redhat.com>
* cluster/tier do not flag migration error on already migrated fileDan Lambright2015-09-161-15/+13
| | | | | | | | | | | | | | In some cases a brick will try to migrate a file that has already been migrated. This is a legal case, e.g. when both bricks are replica pairs. Change-Id: If2578b947014cbbdfb3c6591db9044d6b1d92774 BUG: 1262408 Signed-off-by: Dan Lambright <dlambrig@redhat.com> Reviewed-on: http://review.gluster.org/12186 Tested-by: NetBSD Build System <jenkins@build.gluster.org> Reviewed-by: Joseph Fernandes Tested-by: Gluster Build System <jenkins@build.gluster.com>
* tier/ctr: Solving DB Lock issue due to write contention from db connectionsJoseph Fernandes2015-09-111-40/+95
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This is a backport of 12031. > Problem: The DB on the brick is been accessed by CTR, for write and > tier migrator, for read and write. The write from tier migrator is reseting > the heat counters after a cycle. Since we are using sqlite, two connections > trying to write would cause a db lock contention. As a result CTR used to fail > to update the db. > Solution: Using the same db connection of CTR for reseting the heat counters. > 1) Introducted a new IPC FOP for CTR > 2) After the query do a ipc syncop to the underlying client xlator associated > to the brick. > 3) CTR in brick will catch the IPC FOP and cleat the heat counters. > Change-Id: I53306bfc08dcdba479deb4ccc154896521336150 > BUG: 1260730 > Signed-off-by: Joseph Fernandes <josferna@redhat.com> > Reviewed-on: http://review.gluster.org/12031 > Tested-by: NetBSD Build System <jenkins@build.gluster.org> > Tested-by: Gluster Build System <jenkins@build.gluster.com> Signed-off-by: Dan Lambright <dlambrig@redhat.com> Conflicts: xlators/cluster/dht/src/tier.c Change-Id: I88aa289cdf21e216b42c3d8ccfb4e7e828b43772 BUG: 1262341 Reviewed-on: http://review.gluster.org/12161 Reviewed-by: Joseph Fernandes Tested-by: NetBSD Build System <jenkins@build.gluster.org> Tested-by: Gluster Build System <jenkins@build.gluster.com> Tested-by: Dan Lambright <dlambrig@redhat.com>
* cluster/tier: avoid filling /var/run with tiering filesDan Lambright2015-09-031-4/+28
| | | | | | | | | | | | | | | | | | | | This is a backport of 11931. > We failed to delete old promote/demote workfiles in /var/run. > This fix removes the <pid> postfix so there will be only a > single pair of files. > Change-Id: Ib9aafe7b4a9d4b0c05cf03a94cc1057a423a27d2 > BUG: 1253970 > Signed-off-by: Dan Lambright <dlambrig@redhat.com> > Reviewed-on: http://review.gluster.org/11931 Change-Id: Id9fb843a5ce553a79fc9f5809f84af9d317b1d3e BUG: 1259360 Signed-off-by: Dan Lambright <dlambrig@redhat.com> Reviewed-on: http://review.gluster.org/12092 Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: N Balachandran <nbalacha@redhat.com>
* cluster/tier: account for reordered layoutsDan Lambright2015-09-021-13/+30
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This is a backport of 11092 > For a tiered volume the cold subvolume is always at a fixed > position in the graph. DHT's layout array, on the other hand, > may have the cold subvolume in either the first or second > index, therefore code cannot make any assumptions. The fix > searches the layout for the correct position dynamically > rather than statically. > The bug manifested itself in NFS, in which a newly attached > subvolume had not received an existing directory. This case > is a "stale entry" and marked as such in the layout for > that directory. The code did not see this, because it > looked at the wrong index in the layout array. > The fix also adds the check for decomissioned bricks, and > fixes a problem in detach tier related to starting the > rebalance process: we never received the right defrag > command and it did not get directed to the tier translator. > Change-Id: I77cdf9fbb0a777640c98003188565a79be9d0b56 > BUG: 1214289 > Signed-off-by: Dan Lambright <dlambrig@redhat.com> Change-Id: Idb2eec9ba25812f41de7f960a0314c92341d6b5d BUG: 1259081 Signed-off-by: Dan Lambright <dlambrig@redhat.com> Reviewed-on: http://review.gluster.org/12086 Tested-by: NetBSD Build System <jenkins@build.gluster.org> Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: mohammed rafi kc <rkavunga@redhat.com>
* cluster/tier : Use dht_* versions for xlator_fopsN Balachandran2015-08-191-16/+28
| | | | | | | | | | | | | | | | The tier xlator was using the default_* versions for some xlator_fops. Changed to use the dht_* versions for all xlator_fops Change-Id: I8252fb3911b8a48a55e9eee42b89bd66bbacf799 BUG: 1254468 Signed-off-by: N Balachandran <nbalacha@redhat.com> (cherry picked from commit 0c20107a60726804030f98a7f79b94c677e6a7b6) Reviewed-on: http://review.gluster.org/11951 Tested-by: NetBSD Build System <jenkins@build.gluster.org> Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Dan Lambright <dlambrig@redhat.com> Tested-by: Dan Lambright <dlambrig@redhat.com>
* cluster/tier: fix demotion when cold tier is ECDan Lambright2015-08-121-0/+2
| | | | | | | | | | | | | | | | | | | This is a backport of 11855. We did not set the gfid in the loc structure in tier demotion. EC has a sanity check which fails FOPs when the loc gfid mismatches with the file attribute. When the FOP failed demotion was aborted. > Change-Id: I69022c9ccb135b86e1feea93b01801b6a4100509 > BUG: 1251121 > Signed-off-by: Dan Lambright <dlambrig@redhat.com> Change-Id: I266d554e3e0a2ff024a5ba3a7e9ca40866688eae BUG: 1252907 Signed-off-by: Dan Lambright <dlambrig@redhat.com> Reviewed-on: http://review.gluster.org/11901 Reviewed-by: mohammed rafi kc <rkavunga@redhat.com> Tested-by: NetBSD Build System <jenkins@build.gluster.org>
* dht: Adding log messages to the new logging frameworkarao2015-07-271-11/+15
| | | | | | | | | | | | | | | | | | | | | | | | Backported from: http://review.gluster.org/10021 > Change-Id: Ib3bb61c5223f409c23c68100f3fe884918d2dc3f > BUG: 1194640 > Reviewed-on: http://review.gluster.org/10021 > Reviewed-by: N Balachandran <nbalacha@redhat.com> > Reviewed-by: Joseph Fernandes <josferna@redhat.com> > Tested-by: Joseph Fernandes <josferna@redhat.com> > Reviewed-by: Dan Lambright <dlambrig@redhat.com> > Reviewed-by: Raghavendra G <rgowdapp@redhat.com> > Tested-by: Raghavendra G <rgowdapp@redhat.com> > Signed-off-by: arao <arao@redhat.com> BUG: 1217722 Change-Id: Ide79c6c1e6a466fb52f955c90a2b22711bec794a Signed-off-by: arao <arao@redhat.com> Signed-off-by: Anusha Rao <arao@redhat.com> Reviewed-on: http://review.gluster.org/11350 Tested-by: NetBSD Build System <jenkins@build.gluster.org> Reviewed-by: Raghavendra G <rgowdapp@redhat.com> Tested-by: Gluster Build System <jenkins@build.gluster.com>
* tier/dht: Fixing non atomic promotion/demotion w.r.t to frequency periodJoseph Fernandes2015-06-271-40/+59
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This fixes the ping-pong issue i.e files getting demoted immediately after promition, caused by off-sync promotion/demotion processes. The solution is do promotion/demotion refering to the system time. To have the fix working all the file serving nodes should have thier system time synchronized with each other either manually or using a NTP Server. NOTE: The ping-pong issue can re-appear even with this fix, if the admin have different promotion freq period and demotion freq period, but this would be under the control of the admin. Backport of http://review.gluster.org/#/c/11110/ to 3.7.x: > Change-Id: I1b33a5881d0cac143662ddb48e5b7b653aeb1271 > BUG: 1218717 > Signed-off-by: Joseph Fernandes <josferna@redhat.com> > Reviewed-on: http://review.gluster.org/11110 > Reviewed-by: Dan Lambright <dlambrig@redhat.com> > Tested-by: Dan Lambright <dlambrig@redhat.com> > Tested-by: Gluster Build System <jenkins@build.gluster.com> Signed-off-by: Joseph Fernandes <josferna@redhat.com> Change-Id: I81bd1d677487ebc0fc46df4980500102571de68e BUG: 1230857 Reviewed-on: http://review.gluster.org/11191 Reviewed-by: Niels de Vos <ndevos@redhat.com> Tested-by: NetBSD Build System <jenkins@build.gluster.org> Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Dan Lambright <dlambrig@redhat.com> Tested-by: Dan Lambright <dlambrig@redhat.com>
* tiering:static function called from a non static inline functionMohammed Rafi KC2015-06-221-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Backport of http://review.gluster.org/#/c/11032/ gcc v5.1.1 throws warning for calling a static function from a non-static inline function. <snippet from compiler warning> CC tier.lo tier.c:610:15: warning: 'tier_migrate_using_query_file' is static but used in inline function 'tier_migrate_files_using_qfile' which is not static ret = tier_migrate_using_query_file ((void *)query_cbk_args); ^ tier.c:585:47: warning: 'tier_process_brick_cbk' is static but used in inline function 'tier_build_migration_qfile' which is not static ret = dict_foreach (args->brick_list, tier_process_brick_cbk, ^ tier.c:565:176: warning: 'demotion_qfile' is static but used in inline function 'tier_build_migration_qfile' which is not static tier.c:565:158: warning: 'promotion_qfile' is static but used in inline function 'tier_build_migration_qfile' which is not static tier.c:563:58: warning: 'demotion_qfile' is static but used in inline function 'tier_build_migration_qfile' which is not static tier.c:563:40: warning: 'promotion_qfile' is static but used in inline function 'tier_build_migration_qfile' which is not static ret = remove (GET_QFILE_PATH (is_promotion)); ^ CCLD tier.la </snip> Change-Id: I46046feeb79ab4e2724b0ba6b02c9ec8b121ff4e BUG: 1231767 Signed-off-by: Mohammed Rafi KC <rkavunga@redhat.com> Reviewed-on: http://review.gluster.org/11232 Reviewed-by: Joseph Fernandes Tested-by: Joseph Fernandes Reviewed-by: Dan Lambright <dlambrig@redhat.com> Tested-by: Gluster Build System <jenkins@build.gluster.com>
* tier/volume set: Validate volume set option for tierMohammed Rafi KC2015-06-161-0/+6
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | Volume set option related to tier volume can only be set for tier volume, also currently all volume set i for tier option accepts a non-negative integer. This patch validate both condition. Back port of: >Change-Id: I3611af048ff4ab193544058cace8db205ea92336 >BUG: 1216960 >Signed-off-by: Mohammed Rafi KC <rkavunga@redhat.com> >Signed-off-by: Dan Lambright <dlambrig@redhat.com> >Reviewed-on: http://review.gluster.org/10751 >Tested-by: Gluster Build System <jenkins@build.gluster.com> >Tested-by: NetBSD Build System <jenkins@build.gluster.org> >Reviewed-by: Joseph Fernandes (cherry picked from commit f6a062044a3447bea5bf0fcf21a3f85c00fb6c7d) Change-Id: Ic6081f0ce7ae7effac69ba192bd35c8d382a11d5 BUG: 1230560 Signed-off-by: Mohammed Rafi KC <rkavunga@redhat.com> Reviewed-on: http://review.gluster.org/11173 Tested-by: NetBSD Build System <jenkins@build.gluster.org> Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Joseph Fernandes Tested-by: Joseph Fernandes
* cluster/tier: load libgfdb.so properly in all casesNiels de Vos2015-06-011-1/+1
| | | | | | | | | | | | | | | | | | | | | | We should load libgfdb.so.0, not libgfdb.so Cherry picked from commit 628406f28364f6019261a3bb37335a494ccf8dda: > Change-Id: I7a0d64018ccd9893b1685de391e99b5392bd1879 > BUG: 1222092 > Signed-off-by: Dan Lambright <dlambrig@redhat.com> > Reviewed-on: http://review.gluster.org/10796 > Reviewed-by: Kaleb KEITHLEY <kkeithle@redhat.com> > Reviewed-by: Joseph Fernandes > Reviewed-by: Niels de Vos <ndevos@redhat.com> > Tested-by: Gluster Build System <jenkins@build.gluster.com> Change-Id: I7a0d64018ccd9893b1685de391e99b5392bd1879 BUG: 1221534 Signed-off-by: Niels de Vos <ndevos@redhat.com> Reviewed-on: http://review.gluster.org/10799 Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Dan Lambright <dlambrig@redhat.com> Reviewed-by: mohammed rafi kc <rkavunga@redhat.com>
* tiering/rebalance: Use separate pid/socket file for tieringMohammed Rafi KC2015-05-311-2/+3
| | | | | | | | | | | | | | | | | | | | | | | | | Back port of http://review.gluster.org/10792 When promotion/demotion daemon starts, it uses the same pidfile as rebalance. This patch will introduce a different pid file for the same. >Change-Id: Ic484c53f51e00ae6b2d697748a9600b14829e23b >BUG: 1221970 >Signed-off-by: Mohammed Rafi KC <rkavunga@redhat.com> >Reviewed-on: http://review.gluster.org/10792 >Reviewed-by: Atin Mukherjee <amukherj@redhat.com> >Tested-by: Gluster Build System <jenkins@build.gluster.com> >Tested-by: NetBSD Build System Change-Id: Idda13e983ffd443672aee0873ee51e8cc7089c49 BUG: 1221969 Signed-off-by: Mohammed Rafi KC <rkavunga@redhat.com> Reviewed-on: http://review.gluster.org/10980 Tested-by: NetBSD Build System <jenkins@build.gluster.org> Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Joseph Fernandes Reviewed-by: Kaushal M <kaushal@redhat.com>
* glusterd: add counter support for tiered volumesDan Lambright2015-05-281-0/+6
| | | | | | | | | | | | | | | | | | | | | | | | | | back port of http://review.gluster.org/10292 This fix adds support to view the number of promoted or demoted files from the cli. The mechanism is isolmorphic to checking the status of volumes being rebalanced. gluster volume rebalance <vol> tier status >Change-Id: I1b11ca27355ceec36c488967c23531202030e205 >BUG: 1213063 >Signed-off-by: Mohammed Rafi KC <rkavunga@redhat.com> >Signed-off-by: Dan Lambright <dlambrig@redhat.com> >Reviewed-on: http://review.gluster.org/10292 >Reviewed-by: Atin Mukherjee <amukherj@redhat.com> >Tested-by: Gluster Build System <jenkins@build.gluster.com> Change-Id: I543e886f17132b544274c83fdecca5a8da9d092a BUG: 1221477 Signed-off-by: Mohammed Rafi KC <rkavunga@redhat.com> Signed-off-by: Dan Lambright <dlambrig@redhat.com> Reviewed-on: http://review.gluster.org/10775 Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Atin Mukherjee <amukherj@redhat.com>
* dht/tier/rebalancer: Fix reset of tiering client pidJoseph Fernandes2015-05-101-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | In the patch http://review.gluster.org/#/c/9657 the client pid set by tiering migration was getting over- written in dht_start_rebalance_task(). Just corrected it in dht_setxattr() before calling dht_start_rebalance_task() and removed it from dht_start_rebalance_task(). > http://review.gluster.org/#/c/10502/ > Cherry picked from commit a5fe0f594d41e1a11661d9074bb19e9c2e2c4776 > Change-Id: I37cfa111f83a4e5d498042575c93799f60b49870 > BUG: 1217937 > Signed-off-by: Joseph Fernandes <josferna@redhat.com> > Reviewed-on: http://review.gluster.org/10502 > Tested-by: Gluster Build System <jenkins@build.gluster.com> > Reviewed-by: Susant Palai <spalai@redhat.com> > Reviewed-by: Dan Lambright <dlambrig@redhat.com> Signed-off-by: Joseph Fernandes <josferna@redhat.com> Reviewed-on: http://review.gluster.org/10502 Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Susant Palai <spalai@redhat.com> Reviewed-by: Dan Lambright <dlambrig@redhat.com> Signed-off-by: Joseph Fernandes <josferna@redhat.com> Conflicts: xlators/cluster/dht/src/dht-common.c xlators/cluster/dht/src/tier.c Change-Id: Id513114c9a880c6196162dd4b35bbf1155a8cd09 BUG: 1219027 Reviewed-on: http://review.gluster.org/10609 Reviewed-by: N Balachandran <nbalacha@redhat.com> Tested-by: NetBSD Build System Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Vijay Bellur <vbellur@redhat.com>
* glusterd: support for tier volumes 'detach start' and 'detach commit'Dan Lambright2015-05-091-8/+30
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Back port of http://review.gluster.org/10108 These commands work in a manner analagous to rebalancing when removing a brick. The existing migration daemon detects "detach start" and switches to moving data off the hot tier. While in this state all lookups are directed to the cold tier. gluster v detach-tier <vol> start gluster v detach-tier <vol> commit The status and stop cli commands shall be submitted separately. >Change-Id: I24fda5cc3ba74f5fb8aa9a3234ad51f18b80a8a0 >BUG: 1205540 >Signed-off-by: Dan Lambright <dlambrig@redhat.com> >Signed-off-by: root <root@localhost.localdomain> >Signed-off-by: Dan Lambright <dlambrig@redhat.com> >Reviewed-on: http://review.gluster.org/10108 >Reviewed-by: Kaleb KEITHLEY <kkeithle@redhat.com> Change-Id: I212d748d077fb5870ee84b316c653acbafbea3f7 BUG: 1220047 Signed-off-by: Mohammed Rafi KC <rkavunga@redhat.com> Reviewed-on: http://review.gluster.org/10708 Reviewed-by: Dan Lambright <dlambrig@redhat.com> Tested-by: Gluster Build System <jenkins@build.gluster.com> Reviewed-by: Vijay Bellur <vbellur@redhat.com>
generated by cgit (git 2.34.1) at 2024-06-03 14:01:35 +0000