| Commit message (Collapse) | Author | Age | Files | Lines |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
1. Each brick on a host will get a separate query file.
2. While reading query record from these query files we
read them in a Round-Robin manner.
3. When an error occurs during migration we rename it to
query file with an time stamp and .err extension for
better debugging.
Backport of http://review.gluster.org/13293
> Change-Id: I27c4285d24fd695d2d5cbd9fd7db3879d277ecc8
> BUG: 1302772
> Signed-off-by: Joseph Fernandes <josferna@redhat.com>
> Reviewed-on: http://review.gluster.org/13293
> Smoke: Gluster Build System <jenkins@build.gluster.com>
> Tested-by: N Balachandran <nbalacha@redhat.com>
> NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org>
> Reviewed-by: Dan Lambright <dlambrig@redhat.com>
> CentOS-regression: Gluster Build System <jenkins@build.gluster.com>
Change-Id: I9448f19c10a82fff9b1b3c9387c9026adc0742f1
BUG: 1306514
Signed-off-by: Joseph Fernandes <josferna@redhat.com>
Reviewed-on: http://review.gluster.org/13427
Smoke: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Joseph Fernandes
Tested-by: Joseph Fernandes
NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org>
CentOS-regression: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Dan Lambright <dlambrig@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
There are three kinds of inline functions: plain inline, extern inline,
and static inline. All three have been removed from .c files, except
those in "contrib" which aren't our problem. Inlines in .h files, which
are overwhelmingly "static inline" already, have generally been left
alone. Over time we should be able to "lower" these into .c files, but
that has to be done in a case-by-case fashion requiring more manual
effort. This part was easy to do automatically without (as far as I can
tell) any ill effect.
In the process, several pieces of dead code were flagged by the
compiler, and were removed.
backport of Change-Id: I56a5e614735c9e0a6ee420dab949eac22e25c155,
http://review.gluster.org/11769, BUG: 1245331
Change-Id: Iba1efb0bc578ea4a5e9bf76b7bd93dc1be9eba44
BUG: 1283302
Signed-off-by: Kaleb S KEITHLEY <kkeithle@redhat.com>
Reviewed-on: http://review.gluster.org/12646
Smoke: Gluster Build System <jenkins@build.gluster.com>
NetBSD-regression: NetBSD Build System <jenkins@build.gluster.org>
CentOS-regression: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Niels de Vos <ndevos@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
recording path
1. Providing vol set option for cache size and wal autocheck point
so that performance can be tuned.
2. Removed recording of file path in the db. Trimming database columns.
Path need not be stored in the db, as PARGFID, GFID, Basename is suffice
to derive the path during migration.
Backport of http://review.gluster.org/12972
> Change-Id: I2cb590451a6d244bc91fe66c6dbffe2c2059dfb8
> BUG: 1293034
> Signed-off-by: Joseph Fernandes <josferna@redhat.com>
> Reviewed-on: http://review.gluster.org/12972
> Reviewed-by: N Balachandran <nbalacha@redhat.com>
> Tested-by: Gluster Build System <jenkins@build.gluster.com>
> Tested-by: NetBSD Build System <jenkins@build.gluster.org>
> Reviewed-by: Dan Lambright <dlambrig@redhat.com>
> Tested-by: Dan Lambright <dlambrig@redhat.com>
Signed-off-by: Joseph Fernandes <josferna@redhat.com>
Change-Id: Ia1c109983ec6ce75ed27b8c08f454f5b6283c31d
BUG: 1293659
Reviewed-on: http://review.gluster.org/13067
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Dan Lambright <dlambrig@redhat.com>
Tested-by: Dan Lambright <dlambrig@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Correcting query in the where clause
from "a & b | c"
to "a & (b | c)"
where "a" is the condition to join the gf_file_tb and gf_flink_tb
through gfids
"b" is the condition for the write heat
and "c" is the condition for read heat
Backport of http://review.gluster.org/12823
> Change-Id: I99226d82b0efb68fbef3a40f02b215bb2b4370d6
> BUG: 1286656
> Signed-off-by: Joseph Fernandes <josferna@redhat.com>
> Signed-off-by: Dan Lambright <dlambrig@redhat.com>
> Reviewed-on: http://review.gluster.org/12823
> Tested-by: NetBSD Build System <jenkins@build.gluster.org>
> Tested-by: Gluster Build System <jenkins@build.gluster.com>
Signed-off-by: Joseph Fernandes <josferna@redhat.com>
Change-Id: I70533c8589a6cd67e8737e110af0a23abed9013d
BUG: 1287583
Reviewed-on: http://review.gluster.org/12868
Tested-by: NetBSD Build System <jenkins@build.gluster.org>
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Dan Lambright <dlambrig@redhat.com>
Tested-by: Dan Lambright <dlambrig@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Problem: When a file with old_file_name and GFID_1 is renamed with a new_file_name
which already exists and with GFID_2, this is what happens in linux internaly.
a. "new_file_name" is unlinked for GFID_2
b. a hardlink "new_file_name" is created to GFID_1
c. "old_file_name" hardlink is unlinked for GFID_2.
Well this is all internal to linux, and gluster just issues a rename system call
at POSIX layer. But CTR Xlator doesn't delete the entries corresponding to the
"new_file_name" and GFID_2. Thus leaving the stale entry in the DB.
The following are the implications.
a. Promotion are tried on these stale entries which will fail and show
false results in the status of migration,
b. GFID_2 Files with 2 hardlinks, which will have only one hardlink
after the rename will not be promoted or demoted as the DB shows 2 entries.
Solution: Delete the older database entry for the replaced hardlink
Backport of http://review.gluster.org/12711
> Change-Id: I4eafa0872253e29ff1f0bec4283bcfc579ecf0e2
> BUG: 1284090
> Signed-off-by: Joseph Fernandes <josferna@redhat.com>
> Reviewed-on: http://review.gluster.org/12711
> Tested-by: NetBSD Build System <jenkins@build.gluster.org>
> Tested-by: Gluster Build System <jenkins@build.gluster.com>
> Reviewed-by: Dan Lambright <dlambrig@redhat.com>
> Tested-by: Dan Lambright <dlambrig@redhat.com>
Signed-off-by: Joseph Fernandes <josferna@redhat.com>
Change-Id: Ic35348303ec21f9bd19f20a48f3141449349668b
BUG: 1285688
Reviewed-on: http://review.gluster.org/12762
Tested-by: NetBSD Build System <jenkins@build.gluster.org>
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Dan Lambright <dlambrig@redhat.com>
Tested-by: Dan Lambright <dlambrig@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Currently we heat up a file for both data and metadata write.
Here we provide a ctr xlator option called "ctr-record-metadata-heat"
were the admin can decide on recording metadata heat i.e heatup a
file on metadata writes or not.
Metadata data operation are
a. setattr: explicit changing of atime/mtime using utimes,
changing of posix permissions of the file
b. rename: Renaming a file,
c. unlink, link: adding or deleting hardlinks
d. xattrs: setting or removal of xattrs.
NOTE: atime, mtime and ctime change through writev, readv, truncate, mknod
and create will not be considered here as these fops are data and primary
metadata fops.
Defaultly "ctr-record-metadata-heat" is off. Admin can
switch it on using gluster volume set command.
Backport of http://review.gluster.org/12540
> Change-Id: I91157509255dd5cb429cda2b6d4f64582e155e7b
> BUG: 1279166
> Signed-off-by: Joseph Fernandes <josferna@redhat.com>
> Reviewed-on: http://review.gluster.org/12540
> Tested-by: NetBSD Build System <jenkins@build.gluster.org>
> Tested-by: Gluster Build System <jenkins@build.gluster.com>
> Reviewed-by: Dan Lambright <dlambrig@redhat.com>
> Tested-by: Dan Lambright <dlambrig@redhat.com>
Signed-off-by: Joseph Fernandes <josferna@redhat.com>
Change-Id: I986c319f0cc337b0692a1dd02f71254e786afac4
BUG: 1282315
Reviewed-on: http://review.gluster.org/12582
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Tested-by: NetBSD Build System <jenkins@build.gluster.org>
Reviewed-by: Dan Lambright <dlambrig@redhat.com>
Tested-by: Dan Lambright <dlambrig@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Extending log level flexibity to relevant fops and operations
This is an extension to
http://review.gluster.org/#/c/12491/
Backport of http://review.gluster.org/#/c/12567/
> Change-Id: I33b2f7732f89f52569fb99baa692c7e3bb4c7ab1
> BUG: 1277352
> Signed-off-by: Joseph Fernandes <josferna@redhat.com>
> Reviewed-on: http://review.gluster.org/12567
> Tested-by: NetBSD Build System <jenkins@build.gluster.org>
> Tested-by: Gluster Build System <jenkins@build.gluster.com>
> Reviewed-by: Dan Lambright <dlambrig@redhat.com>
> Tested-by: Dan Lambright <dlambrig@redhat.com>
Signed-off-by: Joseph Fernandes <josferna@redhat.com>
Change-Id: Idcde04a65775ce98367967faf8a53b0f07ff022e
BUG: 1279059
Reviewed-on: http://review.gluster.org/12569
Tested-by: NetBSD Build System <jenkins@build.gluster.org>
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Dan Lambright <dlambrig@redhat.com>
Tested-by: Dan Lambright <dlambrig@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Earlier, when the database was queried we used to save
all the queried records in an ASCII format in the query file.
This caused issues like filename having ASCII delimiter and used
to take a lot of space. The tier.c file also had a lot of parsing code.
Here we changed the format of the query file to binary.
All the logic of serialization and formating of query record is done
by libgfdb. Libgfdb provides API,
gfdb_write_query_record() and gfdb_read_query_record(),
which the user i.e tier migrator and CTR xlator can use to
write to and read from query file.
With this binary format we save on disk space i.e reduce to 50% atleast
as we are saving GFID's in binary format 16 bytes and not the string format
which takes 36 bytes + We are not saving path of the file + we are also saving on
ASCII delimiters.
The on disk format of query record is as follows,
+---------------------------------------------------------------------------+
| Length of serialized query record | Serialized Query Record |
+---------------------------------------------------------------------------+
4 bytes Length of serialized query record
|
|
-------------------------------------------------|
|
|
V
Serialized Query Record Format:
+---------------------------------------------------------------------------+
| GFID | Link count | <LINK INFO> |..... | FOOTER |
+---------------------------------------------------------------------------+
16 B 4 B Link Length 4 B
| |
| |
-----------------------------| |
| |
| |
V |
Each <Link Info> will be serialized as |
+-----------------------------------------------+ |
| PGID | BASE_NAME_LENGTH | BASE_NAME | |
+-----------------------------------------------+ |
16 B 4 B BASE_NAME_LENGTH |
|
|
------------------------------------------------------------------------|
|
|
V
FOOTER is a magic number 0xBAADF00D indicating the end of the record.
This also serves as a serialized schema validator.
Backport of http://review.gluster.org/#/c/12354/
> Change-Id: I9db7416fd421e118dd44eafab8b535caafe50d5a
> BUG: 1272207
> Signed-off-by: Joseph Fernandes <josferna@redhat.com>
> Reviewed-on: http://review.gluster.org/12354
> Reviewed-by: N Balachandran <nbalacha@redhat.com>
> Tested-by: Gluster Build System <jenkins@build.gluster.com>
> Reviewed-by: Dan Lambright <dlambrig@redhat.com>
> Tested-by: Dan Lambright <dlambrig@redhat.com>
Change-Id: I170c579027f2594a58706f826e3ddf89e34022f4
BUG: 1263619
Signed-off-by: Joseph Fernandes <josferna@redhat.com>
Reviewed-on: http://review.gluster.org/12535
Tested-by: NetBSD Build System <jenkins@build.gluster.org>
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Dan Lambright <dlambrig@redhat.com>
Tested-by: Dan Lambright <dlambrig@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
CTR doesnt read from the DB, so to make sure that file records are
created it does a heal during a lookup. It remembers the decision in
the inode context cache and retrys periodically. When the volume is
restarted it looses all the inode cache from the previous time and CTR
lookup heals tries the heal again, but this time it finds that the records
are already there from sql and logs this error, and remembers this until the
volume is restarted or inode is flushed out of inode cache of the brick.
Solution: the log levels should be reduced to trace for this case and
customers need not see this.
Backport of http://review.gluster.org/#/c/12491/
Change-Id: I5f11e83c306ae912a90f12220f62bb7563b8eb96
BUG: 1279059
Signed-off-by: Joseph Fernandes <josferna@redhat.com>
Reviewed-on: http://review.gluster.org/12534
Tested-by: NetBSD Build System <jenkins@build.gluster.org>
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Dan Lambright <dlambrig@redhat.com>
Tested-by: Dan Lambright <dlambrig@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
version less than 3.7 i.e rhel 6.7
Problem: On RHEL 6.7, we have sqlite version 3.6.2 which doesnt support
WAL journaling mode, as this journaling mode is only available in sqlite 3.7 and above.
As a result we cannot have to progreses concurrently accessing sqlite, without
running into db locks! Well WAL is also need for performace on CTR side.
Solution: This solution is to use CTR db connection for doing queries when WAL mode is
absent. i,e tier migrator will send sync_op ipc calls to CTR, which in turn will
do the query and create/update the query file suggested by tier migrator.
Pending: Well this solution will stop the db locks but the performance is still an issue for CTR.
We are developing an in-Memory Transaction Log (iMeTaL) which will help boost the CTR
performance by doing in memory udpates on the IO path and later flush the updates to
the db in a batch/segment flush.
Master patch: http://review.gluster.org/#/c/12191
>> Change-Id: Ie3149643ded159234b5cc6aa6cf93b9022c2f124
>> BUG: 1240577
>> Signed-off-by: Joseph Fernandes <josferna@redhat.com>
>> Signed-off-by: Dan Lambright <dlambrig@redhat.com>
>> Signed-off-by: Joseph Fernandes <josferna@redhat.com>
>> Reviewed-on: http://review.gluster.org/12191
>> Tested-by: Gluster Build System <jenkins@build.gluster.com>
>> Reviewed-by: Luis Pabon <lpabon@redhat.com>
Signed-off-by: Joseph Fernandes <josferna@redhat.com>
Change-Id: Ie8c7a7e9566244c104531b579126bb57fbc6e32b
BUG: 1270123
Reviewed-on: http://review.gluster.org/12325
Tested-by: NetBSD Build System <jenkins@build.gluster.org>
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Dan Lambright <dlambrig@redhat.com>
Tested-by: Dan Lambright <dlambrig@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This is a backport of 12031.
> Problem: The DB on the brick is been accessed by CTR, for write and
> tier migrator, for read and write. The write from tier migrator is reseting
> the heat counters after a cycle. Since we are using sqlite, two connections
> trying to write would cause a db lock contention. As a result CTR used to fail
> to update the db.
> Solution: Using the same db connection of CTR for reseting the heat counters.
> 1) Introducted a new IPC FOP for CTR
> 2) After the query do a ipc syncop to the underlying client xlator associated
> to the brick.
> 3) CTR in brick will catch the IPC FOP and cleat the heat counters.
> Change-Id: I53306bfc08dcdba479deb4ccc154896521336150
> BUG: 1260730
> Signed-off-by: Joseph Fernandes <josferna@redhat.com>
> Reviewed-on: http://review.gluster.org/12031
> Tested-by: NetBSD Build System <jenkins@build.gluster.org>
> Tested-by: Gluster Build System <jenkins@build.gluster.com>
Signed-off-by: Dan Lambright <dlambrig@redhat.com>
Conflicts:
xlators/cluster/dht/src/tier.c
Change-Id: I88aa289cdf21e216b42c3d8ccfb4e7e828b43772
BUG: 1262341
Reviewed-on: http://review.gluster.org/12161
Reviewed-by: Joseph Fernandes
Tested-by: NetBSD Build System <jenkins@build.gluster.org>
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Tested-by: Dan Lambright <dlambrig@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This is a backport of 11984.
> Currently the tier feature piggy backs off the rebalance command
> syntax to obtain status and this is clumsy. Introduce a new
> tier command that can do tier specific operations, starting
> with volume status to display counters.
> Old commands:
> gluster volume attach-tier <vol> [replica count] {bricklist..}
> gluster volume detach-tier <vol> {start|stop|commit}
> New commands:
> gluster volume tier <vol> attach [replica count] {bricklist} |
> detach {start|stop|commit} |
> status
> Change-Id: Ic07b3c6260588162de7d34380f8cbd3d8a7f35d3
> BUG: 1255693
> Signed-off-by: Dan Lambright <dlambrig@redhat.com>
> Reviewed-on: http://review.gluster.org/11984
> Reviewed-by: Gaurav Kumar Garg <ggarg@redhat.com>
> Tested-by: Gluster Build System <jenkins@build.gluster.com>
Signed-off-by: Dan Lambright <dlambrig@redhat.com>
Change-Id: Id45bd0fa6b8606dd47863de83a694908da393229
BUG: 1261664
Signed-off-by: Dan Lambright <dlambrig@redhat.com>
Reviewed-on: http://review.gluster.org/12143
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Gaurav Kumar Garg <ggarg@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This is a backport of 11885.
> We overflowed when converting seconds to usecs in preperation for
> sql queries. The fix uses uint64_t throughout including subexpressions.
> Change-Id: I59bdb742197400dede97f54735b52030920b0d19
> BUG: 1231268
> Signed-off-by: Dan Lambright <dlambrig@redhat.com>
Change-Id: Iaef7083dac7a162080f4bc23e29dedfe700fa342
BUG: 1252903
Signed-off-by: Dan Lambright <dlambrig@redhat.com>
Reviewed-on: http://review.gluster.org/11913
Tested-by: NetBSD Build System <jenkins@build.gluster.org>
Reviewed-by: Joseph Fernandes
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Backport of http://review.gluster.org/11584 to 3.7.2
Fixing rename or updating of link in
1) libgfdb : handling return from updatelink
2) ctr xlator: correcting freeing of oldbase name in
hardlink list in the update link
>Change-Id: I1789c492b81876123cd456051a9ae495e876ad46
>BUG: 1241054
>Signed-off-by: Joseph Fernandes <josferna@redhat.com>
>Reviewed-on: http://review.gluster.org/11584
>Tested-by: Joseph Fernandes
>Reviewed-by: Dan Lambright <dlambrig@redhat.com>
>Tested-by: Dan Lambright <dlambrig@redhat.com>
>Tested-by: Gluster Build System <jenkins@build.gluster.com>
Signed-off-by: Joseph Fernandes <josferna@redhat.com>
Change-Id: I6da86c06e390f62fe8a44ade7b1efda5771e9ca8
BUG: 1242536
Signed-off-by: Joseph Fernandes <josferna@redhat.com>
Reviewed-on: http://review.gluster.org/11649
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Dan Lambright <dlambrig@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The find_unchanged query should be
"write_heat <= defined_heat" AND "read_heat <= defined_heat"
and not
"write_heat <= defined_heat" OR "read_heat <= defined_heat"
>>Backport of http://review.gluster.org/#/c/11577/
>>Change-Id: Ie82e02aafbb7ea14563007307de3350ea022049a
>>BUG: 1240970
>>Signed-off-by: Joseph Fernandes <josferna@redhat.com>
Change-Id: Ie5d06fa2e3a3552741d2e6bd76807a69808c3d65
BUG: 1241776
Signed-off-by: Joseph Fernandes <josferna@redhat.com>
Reviewed-on: http://review.gluster.org/11608
Reviewed-by: Dan Lambright <dlambrig@redhat.com>
Reviewed-by: Pamela Ousley <pousley@redhat.com>
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Tested-by: Dan Lambright <dlambrig@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This is a back port of 11334
1) Ignore creation of T file in ctr_mknod
2) Ignore lookup for T file in ctr_lookup
3) Ctr_lookup:
a. If the gfid and pgfid in empty dont record
b. Decreased log level for multiple heal attempts
c. Inode/File heal happens after an expiry period, which is configurable.
d. Hardlink heal happens after an expiry period, which is configurable.
> Change-Id: Id8eb5092e78beaec22d05f5283645081619e2452
> BUG: 1235269
> Signed-off-by: Joseph Fernandes <josferna@redhat.com>
> Reviewed-on: http://review.gluster.org/11334
> Tested-by: Gluster Build System <jenkins@build.gluster.com>
> Reviewed-by: Dan Lambright <dlambrig@redhat.com>
> Tested-by: Dan Lambright <dlambrig@redhat.com>
Change-Id: Ia28a5cf975e41d318906f707deca447aaa35630f
BUG: 1236288
Signed-off-by: Dan Lambright <dlambrig@redhat.com>
Reviewed-on: http://review.gluster.org/11446
Reviewed-by: Joseph Fernandes
Tested-by: Joseph Fernandes
Tested-by: Gluster Build System <jenkins@build.gluster.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Backport of http://review.gluster.org/#/c/10819/
Cherry picked from 2847b03a38515a625e36d52c7e0a1d37107d72ba
>Change-Id: I61874561fdf2c175d2b3eec0c85c25f12dc60552
>BUG: 1194640
>Signed-off-by: Mohamed Ashiq <ashiq333@gmail.com>
>Reviewed-on: http://review.gluster.org/10819
>Tested-by: NetBSD Build System <jenkins@build.gluster.org>
>Tested-by: Gluster Build System <jenkins@build.gluster.com>
>Reviewed-by: Joseph Fernandes
Change-Id: I61874561fdf2c175d2b3eec0c85c25f12dc60552
BUG: 1217722
Signed-off-by: Mohamed Ashiq <ashiq333@gmail.com>
Reviewed-on: http://review.gluster.org/11284
Tested-by: NetBSD Build System <jenkins@build.gluster.org>
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Dan Lambright <dlambrig@redhat.com>
Tested-by: Dan Lambright <dlambrig@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
1) If the database file exists
a. Dont try re-creating the db schema
b. Dont try re-configuring the db.
2) Dont assert in fini_db () when connection is NULL
>> Signed-off-by: Joseph Fernandes <josferna@redhat.com>
>> Reviewed-on: http://review.gluster.org/10870
>> Tested-by: NetBSD Build System
>> Tested-by: Gluster Build System <jenkins@build.gluster.com>
>> Reviewed-by: Vijay Bellur <vbellur@redhat.com>
>> Signed-off-by: Joseph Fernandes <josferna@redhat.com>
>>Change-Id: Idd2833f55caaa6b3a77d935d877d6c4d2994da6a
Change-Id: Ia2e044fce4e41bccc8fdada1cb21f240fdbd55df
BUG: 1225077
Signed-off-by: Joseph Fernandes <josferna@redhat.com>
Reviewed-on: http://review.gluster.org/10992
Reviewed-by: Vijay Bellur <vbellur@redhat.com>
Tested-by: Vijay Bellur <vbellur@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
| |
Log typo fix for CTR Xlator and Libgfdb
Change-Id: Ia39069a5ce9c48bbee937f1b5c5d749a30c9ac56
BUG: 1220100
Signed-off-by: Joseph Fernandes <josferna@redhat.com>
Reviewed-on: http://review.gluster.org/10742
Reviewed-by: N Balachandran <nbalacha@redhat.com>
Tested-by: Gluster Build System <jenkins@build.gluster.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Problem: The CTR xlator records file meta (heat/hardlinks)
into the data. This works fine for files which are created
after ctr xlator is switched ON. But for files which were
created before CTR xlator is ON, CTR xlator is not able to
record either of the meta i.e heat or hardlinks. Thus making
those files immune to promotions/demotions.
Solution: The solution that is implemented in this patch is
do ctr-db heal of all those pre-existent files, using named lookup.
For this purpose we use the inode-xlator context variable option
in gluster.
The inode-xlator context variable for ctr xlator will have the
following,
a. A Lock for the context variable
b. A hardlink list: This list represents the successful looked
up hardlinks.
These are the scenarios when the hardlink list is updated:
1) Named-Lookup: Whenever a named lookup happens on a file, in the
wind path we copy all required hardlink and inode information to
ctr_db_record structure, which resides in the frame->local variable.
We dont update the database in wind. During the unwind, we read the
information from the ctr_db_record and ,
Check if the inode context variable is created, if not we create it.
Check if the hard link is there in the hardlink list.
If its not there we add it to the list and send a update to the
database using libgfdb.
Please note: The database transaction can fail(and we ignore) as there
already might be a record in the db. This update to the db is to heal
if its not there.
If its there in the list we ignore it.
2) Inode Forget: Whenever an inode forget hits we clear the hardlink list in
the inode context variable and delete the inode context variable.
Please note: An inode forget may happen for two reason,
a. when the inode is delete.
b. the in-memory inode is evicted from the inode table due to cache limits.
3) create: whenever a create happens we create the inode context variable and
add the hardlink. The database updation is done as usual by ctr.
4) link: whenever a hardlink is created for the inode, we create the inode context
variable, if not present, and add the hardlink to the list.
5) unlink: whenever a unlink happens we delete the hardlink from the list.
6) mknod: same as create.
7) rename: whenever a rename happens we update the hardlink in list. if the hardlink
was not present for updation, we add the hardlink to the list.
What is pending:
1) This solution will only work for named lookups.
2) We dont track afr-self-heal/dht-rebalancer traffic for healing.
> http://review.gluster.org/#/c/10370/
> Cherry picked from commit cb11dd91a6cc296e4a3808364077f4eacb810e48
> Change-Id: Ia4bbaf84128ad6ce8c3ddd70bcfa82894c79585f
> BUG: 1212037
> Signed-off-by: Joseph Fernandes <josferna@redhat.com>
> Signed-off-by: Dan Lambright <dlambrig@redhat.com>
> Reviewed-on: http://review.gluster.org/10370
> Reviewed-by: Jeff Darcy <jdarcy@redhat.com>
> Tested-by: Gluster Build System <jenkins@build.gluster.com>
> Tested-by: NetBSD Build System
> Reviewed-by: Vijay Bellur <vbellur@redhat.com>
Change-Id: I367aa46c3f4b8f912248fb8be75866507f2538df
BUG: 1219075
Signed-off-by: Joseph Fernandes <josferna@redhat.com>
Signed-off-by: Dan Lambright <dlambrig@redhat.com>
Reviewed-on: http://review.gluster.org/10370
Reviewed-by: Jeff Darcy <jdarcy@redhat.com>
Reviewed-by: Vijay Bellur <vbellur@redhat.com>
Signed-off-by: Joseph Fernandes <josferna@redhat.com>
Reviewed-on: http://review.gluster.org/10615
Tested-by: Vijay Bellur <vbellur@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
1) ctr_link_consistency option for ctr xaltor is provided so that
the user can choose to switch it on or off.
/* For link consistency we do a double update i.e mark the link
* during the wind and during the unwind we update/delete the link.
* This has a performance hit. We give a choice here whether we need
* link consistency to be spoton or not using link_consistency flag.
* This will have only one link update */
2) In delete the wind time recording is moved to unwind path.
/* Special performance case:
* Updating wind time in unwind for delete. This is done here
* as in the wind path we will not know whether its the last
* link or not. For a last link there is not use to update any
* wind or unwind time!*/
> http://review.gluster.org/#/c/10170/
> Cherry picked from commit 606d9734543208542afcf9df982bf2d560235ef6
> Change-Id: I209472fb816f939db4a868b97ba053b028f17ea6
> BUG: 1217786
> Signed-off-by: Joseph Fernandes <josferna@redhat.com>
> Reviewed-on: http://review.gluster.org/10170
> Reviewed-by: Dan Lambright <dlambrig@redhat.com>
> Tested-by: Gluster Build System <jenkins@build.gluster.com>
> Signed-off-by: Joseph Fernandes <josferna@redhat.com>
Change-Id: I4a89ef80875f36cff91520f712e1f47fde258a63
BUG: 1219066
Signed-off-by: Joseph Fernandes <josferna@redhat.com>
Reviewed-on: http://review.gluster.org/10170
Reviewed-by: Dan Lambright <dlambrig@redhat.com>
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Signed-off-by: Joseph Fernandes <josferna@redhat.com>
Reviewed-on: http://review.gluster.org/10614
Tested-by: NetBSD Build System
Reviewed-by: Vijay Bellur <vbellur@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
On Linux systems we should use the libuuid from the distribution and not
bundle and statically link the contrib/uuid/ bits.
libglusterfs/src/compat-uuid.h has been introduced and should become an
abstraction layer for different UUID APIs. Non-Linux operating systems
should implement their compatibility layer there.
Once all operating systems have an implementation in compat-uuid.h, we
can remove contrib/uuid/ from the repository completely.
Change-Id: I345e5357644be2521685e00358bb8c83c4ea0577
BUG: 1206587
Signed-off-by: Niels de Vos <ndevos@redhat.com>
Reviewed-on: http://review.gluster.org/10129
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Vijay Bellur <vbellur@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This fix will solve the heating of the files during the promotion
or demotion.
Promotion:
~~~~~~~~~
When a file gets promoted it get the current time stamp
during creation only, but following writes or reads during the
migration wont heat the file.
Demotion:
~~~~~~~~
When a file gets demoted it get the wind/unwind time stamp is set to
zero. The following writes or reads during the migration wont heat
the file.
What is remaining ?
~~~~~~~~~~~~~~~~~
Bug 1209129 ( https://bugzilla.redhat.com/show_bug.cgi?id=1209129 )
Inspite of this fix there is still a issue remaining, i.e the heat of
the file is not keep intact during a internal rebalance activity i.e
a rebalance within a tier.
Change-Id: I01e82dc226355599732d40e699062cee7960b0a5
BUG: 1207867
Signed-off-by: Joseph Fernandes <josferna@redhat.com>
Signed-off-by: Dan Lambright <dlambrig@redhat.com>
Signed-off-by: Joseph Fernandes <josferna@redhat.com>
Reviewed-on: http://review.gluster.org/10080
Reviewed-by: Shyamsundar Ranganathan <srangana@redhat.com>
Tested-by: Gluster Build System <jenkins@build.gluster.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This patch resolves tiering translator issues taken from the list
in bug 1203776. These issues have been selected to be fixed
first. The rest will be fixed in a subsequent patch (or are not a
problem).
3. Replace hardcoded #defines of promote/demote file names
6. Use loc_wipe() in migrate_using_query_file()
9. Only promote/demote files on the same node on which they reside.
14. Replace calloc with GF_CALLOC in tier.c and ensure freeing done
properly.
15. Handle if parse_query_str fails
22. Only load gfdb library on server side, remove SQL references
from client.
Change-Id: I6563b11e58ab2e4c6b1ce44db755781ad6d930fb
BUG: 1203776
Signed-off-by: Dan Lambright <dlambrig@redhat.com>
Reviewed-on: http://review.gluster.org/9987
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: N Balachandran <nbalacha@redhat.com>
Reviewed-by: Niels de Vos <ndevos@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
glusterfs relies on Linux uuid implementation, which
API is incompatible with most other systems's uuid. As
a result, libglusterfs has to embed contrib/uuid,
which is the Linux implementation, on non Linux systems.
This implementation is incompatible with systtem's
built in, but the symbols have the same names.
Usually this is not a problem because when we link
with -lglusterfs, libc's symbols are trumped. However
there is a problem when a program not linked with
-lglusterfs will dlopen() glusterfs component. In
such a case, libc's uuid implementation is already
loaded in the calling program, and it will be used
instead of libglusterfs's implementation, causing
crashes.
A possible workaround is to use pre-load libglusterfs
in the calling program (using LD_PRELOAD on NetBSD for
instance), but such a mechanism is not portable, nor
is it flexible. A much better approach is to rename
libglusterfs's uuid_* functions to gf_uuid_* to avoid
any possible conflict. This is what this change attempts.
BUG: 1206587
Change-Id: I9ccd3e13afed1c7fc18508e92c7beb0f5d49f31a
Signed-off-by: Emmanuel Dreyfus <manu@netbsd.org>
Reviewed-on: http://review.gluster.org/10017
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Niels de Vos <ndevos@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Coverity CID 1288820
strncpy executed with a limit equal to the target array
size potentially leaves the target string not null terminated.
Make sure the copied string is a valid 0 terminated string.
Change-Id: I39ff6a64ca5b9e30562226dd34c5b06267b75b87
BUG: 789278
Signed-off-by: Michael Adam <obnox@samba.org>
Reviewed-on: http://review.gluster.org/10063
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Poornima G <pgurusid@redhat.com>
Reviewed-by: Kaleb KEITHLEY <kkeithle@redhat.com>
Reviewed-by: Joseph Fernandes <josferna@redhat.com>
Tested-by: Joseph Fernandes <josferna@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
1) Query fix in find_changed_with_freq()
2) Volume option typo fix for write_freq_threshold
and read_freq_threshold
Change-Id: I38e154818178aab412b2d7b2914cd29acef66ffb
BUG: 1207343
Signed-off-by: Joseph Fernandes <josferna@redhat.com>
Reviewed-on: http://review.gluster.org/10050
Reviewed-by: Dan Lambright <dlambrig@redhat.com>
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Niels de Vos <ndevos@redhat.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The tier translator shares most of DHT's code. It differs in how
subvolumes are chosen for I/Os, and how file migration (cache promotion
and demotion) is managed. That different functionality is split to either
DHT or tier logic according to the "tier_methods" structure.
A cache promotion and demotion thread is created in a manner
similar to the rebalance daemon. The thread operates a timing
wheel which periodically checks for promotion and demotion candidates
(files). Candidates are queued and then migrated. Candidates must exist on
the same node as the daemon and meet other critera per caching policies.
This patch has two authors (Dan Lambright and Joseph Fernandes). Dan
did the DHT changes and Joe wrote the cache policies. The fix depends on
DHT readidr changes and the database library which have been submitted
separately. Header files in libglusterfs/src/gfdb should be reviewed in
patch 9683.
For more background and design see the feature page [1].
[1]
http://www.gluster.org/community/documentation/index.php/Features/data-classification
Change-Id: Icc26c517ccecf5c42aef039f5b9c6f7afe83e46c
BUG: 1194753
Signed-off-by: Dan Lambright <dlambrig@redhat.com>
Reviewed-on: http://review.gluster.org/9724
Reviewed-by: Vijay Bellur <vbellur@redhat.com>
Tested-by: Vijay Bellur <vbellur@redhat.com>
|
|
|
|
|
|
|
|
|
| |
Change-Id: Id9a7d0f457d9759ab7d0a52a4000b5ae36d211f8
BUG: 1194753
Signed-off-by: Niels de Vos <ndevos@redhat.com>
Reviewed-on: http://review.gluster.org/9946
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Kaleb KEITHLEY <kkeithle@redhat.com>
|
|
*************************************************************************
Libgfdb |
*************************************************************************
Libgfdb provides abstract mechanism to record extra/rich metadata
required for data maintenance, such as data tiering/classification.
It provides consumer with API for recording and querying, keeping
the consumer abstracted from the data store used beneath for storing data.
It works in a plug-and-play model, where data stores can be plugged-in.
Presently we have plugin for Sqlite3. In the future will provide recording
and querying performance optimizer. In the current implementation the schema
of metadata is fixed.
Schema:
~~~~~~
GF_FILE_TB Table:
~~~~~~~~~~~~~~~~~
This table has one entry per file inode. It holds the metadata required to
make decisions in data maintenance.
GF_ID (Primary key) : File GFID (Universal Unique IDentifier in the namespace)
W_SEC, W_MSEC : Write wind time in sec & micro-sec
UW_SEC, UW_MSEC : Write un-wind time in sec & micro-sec
W_READ_SEC, W_READ_MSEC : Read wind time in sec & micro-sec
UW_READ_SEC, UW_READ_MSEC : Read un-wind time in sec & micro-sec
WRITE_FREQ_CNTR INTEGER : Write Frequency Counter
READ_FREQ_CNTR INTEGER : Read Frequency Counter
GF_FLINK_TABLE:
~~~~~~~~~~~~~~
This table has all the hardlinks to a file inode.
GF_ID : File GFID (Composite Primary Key)``|
GF_PID : Parent Directory GFID (Composite Primary Key) |-> Primary Key
FNAME : File Base Name (Composite Primary Key)__|
FPATH : File Full Path (Its redundant for now, this will go)
W_DEL_FLAG : This Flag is used for crash consistancy, when a link is unlinked.
i.e Set to 1 during unlink wind and during unwind this record
is deleted
LINK_UPDATE : This Flag is used when a link is changed i.e rename.
Set to 1 when rename wind and set to 0 in rename unwind
Libgfdb API:
~~~~~~~~~~~
Refer libglusterfs/src/gfdb/gfdb_data_store.h
Change-Id: I2e9fbab3878ce630a7f41221ef61017dc43db11f
BUG: 1194753
Signed-off-by: Joseph Fernandes <josferna@redhat.com>
Signed-off-by: Dan Lambright <dlambrig@redhat.com>
Signed-off-by: Joseph Fernandes <josferna@redhat.com>
Reviewed-on: http://review.gluster.org/9683
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Vijay Bellur <vbellur@redhat.com>
|