glusterfs.git/libglusterfs, branch release-3.8-fb

build: merge our 3.6 and upstream 3.8 configure/specfile

2017-11-20T20:35:31+00:00

Summary:
They have a common ancestor at 3.6, but there were hundreds of
lines of changes for each file on each side of the fork. In both
cases the easiest method was to take the upstream 3.8 version and
re-apply our own changes since we branched. Some changes were
dropped (e.g. runit) and a few other files needed new changes
(e.g. pkg-version) to keep up. Then there was more hacking to
fix stealth geo-rep dependencies, enable tirpc/IPv6, and so on.

Also added buildrpm38 and makerelease38.  These should probably
not go upstream, but not sure what else to do with them.

Test Plan: Build RPMs. Install, create volumes, mount, do I/O.

Reviewers: sshreyas, #posix_storage

Reviewed By: sshreyas

Subscribers: jbacik, aquevedo, scientist, sshreyas, calvinowens, jweiner

Differential Revision: https://phabricator.intern.facebook.com/D6259797

Tasks: T20348589

Tags: posix-2017h2, gluster, posix_storage

Change-Id: I2d43fc6f7f5603293e406c21e4ec85bf19610b77
Signature: 6259797:1510694123:fc5d2975fec134a51d4b70f7f983cd71971e175a

Replace namespace/io-stats/io-threads with 3.6-fb versions

2017-09-15T20:47:01+00:00

This rolls up multiple patches related to namespace identificaton and
throttling/QoS.  This primarily includes the following, all by Michael
Goulet .

  io-threads: Add weighted round robin queueing by namespace
  https://phabricator.facebook.com/D5615269

  io-threads: Add per-namespaces queue sizes to IO_THREADS_QUEUE_SIZE_KEY
  https://phabricator.facebook.com/D5683162

  io-threads: Implement better slot allocation algorithm
  https://phabricator.facebook.com/D5683186

  io-threads: Only enable weighted queueing on bricks
  https://phabricator.facebook.com/D5700062

  io-threads: Update queue sizes on drain
  https://phabricator.facebook.com/D5704832

  Fix parsing (-1) as default NS weight
  https://phabricator.facebook.com/D5723383

Parts of the following patches have also been applied to satisfy
dependencies.

  io-throttling: Calculate moving averages and throttle offending hosts
  https://phabricator.fb.com/D2516161
  Shreyas Siravara 

  Hook up ODS logging for FUSE clients.
  https://phabricator.facebook.com/D3963376
  Kevin Vigor 

  Add the flag --skip-nfsd-start to skip the NFS daemon stating, even if
  it is enabled
  https://phabricator.facebook.com/D4575368
  Alex Lorca 

There are also some "standard" changes: dealing with code that moved,
reindenting to comply with Gluster coding standards, gf_uuid_xxx, etc.

This patch *does* revert some changes which have occurred upstream since
3.6; these will be re-applied as apppropriate on top of this new base.

Change-Id: I69024115da7a60811e5b86beae781d602bdb558d
Signed-off-by: Jeff Darcy

added p99 support to the samples logging

2017-09-13T18:31:53+00:00

Summary:
added global and by-fop-type calculation of p99 latency
to the sampled fop data

Test Plan:
build local glusterfs mount and looked at
the stats while dd if=/dev/zero of=/mnt/fuse/groot/share1/test1 bs=5

Reviewers: sshreyas, mgoulet, jdarcy

Reviewed By: jdarcy

Subscribers: jdarcy

Differential Revision: https://phabricator.intern.facebook.com/D5597662

Change-Id: I3f5cd9c0ea59ae4357827fcbd19bbf009e661c05
Signed-off-by: Jeff Darcy 
Reviewed-on: https://review.gluster.org/18277
Reviewed-by: Jeff Darcy 
Tested-by: Jeff Darcy 
CentOS-regression: Gluster Build System 
Smoke: Gluster Build System

Disable brick daemon from incorrect brick directory

2017-09-13T17:50:36+00:00

Summary:
Currently the bricks can open any mount directory from the given volume. This patch adds a provision to prevent
bricks from opening brick directories that aren't created for them. This will help with operating gluster on large
scale.

We add a new xattr GF_XATTR_BRICK_NAME to the brick directory. When we start a brick daemon, we make sure the path on
disk matches with the config provided. For backward compatibility, we ignore if there is no value for
GF_XATTR_BRICK_NAME and set the current brick daemon's path as value.

We ignore GF_XATTR_BRICK_NAME during healing and reset GF_XATTR_BRICK_NAME on brick replace.

Test Plan: Run fb-smoke

Reviewers: jdarcy, sshreyas

Reviewed By: sshreyas

Differential Revision: https://phabricator.intern.facebook.com/D5448921

Porting note: disabled some checks to deal with the snapshot case

Change-Id: I98e62033dfd07f30ad3b99ac003ce94c8d935e5f
Signed-off-by: Jeff Darcy 
Reviewed-on: https://review.gluster.org/18275
Reviewed-by: Jeff Darcy 
Tested-by: Jeff Darcy 
CentOS-regression: Gluster Build System 
Smoke: Gluster Build System

event: Idle connection management

2017-09-12T15:12:21+00:00

Summary:
- This diff adds support for detecting and tracking idle client connections.
- It allows *service translators* (server, nfs) to opt-in to detect and close idle client connections.
- Right now it explicitly restricts the service to NFS as a safety.

Here are the debug logs when a client connection gets closed:

  [2016-03-29 17:27:06.154232] W [socket.c:2426:socket_timeout_handler] 0-socket: Shutting down idle client connection (idle=20s,fd=20,conn=[2401:db00:11:d0af:face:0:3:0:957]->[2401:db00:11:d0af:face:0:3:0:2049])!
  [2016-03-29 17:27:06.154292] D [event-epoll.c:655:__event_epoll_timeout_slot] 0-epoll: Connection on slot->fd=9 was idle for 20 seconds!
  [2016-03-29 17:27:06.163282] D [socket.c:629:__socket_rwv] 0-socket.nfs-server: EOF on socket
  [2016-03-29 17:27:06.163298] D [socket.c:2474:socket_event_handler] 0-transport: disconnecting now
  [2016-03-29 17:27:06.163316] D [event-epoll.c:614:event_dispatch_epoll_handler] 0-epoll: generation bumped on idx=9 from gen=4 to slot->gen=5, fd=20, slot->fd=20

Test Plan: - Used stuck NFS mounts to create idle clients and unstuck them.

Reviewers: kvigor, rwareing

Reviewed By: rwareing

Subscribers: dld, moox, dph

Differential Revision: https://phabricator.fb.com/D3112099

Change-Id: Ic06c89e03f87daabab7f07f892390edd1a1fcc20
Signed-off-by: Jeff Darcy 
Reviewed-on: https://review.gluster.org/18265
Reviewed-by: Jeff Darcy 
Tested-by: Jeff Darcy 
CentOS-regression: Gluster Build System 
Smoke: Gluster Build System

io-threads: nuke everything from a client when it disconnects

2017-09-09T16:54:52+00:00

Summary: These requests haven't been issued, yet alone acknowledged.  They would disappear if we crashed, which to the client is indistinguishable from any other kind of disconnection - if indeed the client itself isn't the one that died.  So we're completely within our rights to discard these.  There are strong hints that such "orphan" requests are part of how we get into the lock-revocation hangs we've been seeing for a while.  Even if that theory doesn't pan out, there's no good reason to keep them around clogging up queues and so forth.

This is a port of D5430057 & D5662545 to 3.8

Change-Id: Ie4c88f7791aac85540631f60f5c639497468ad76
Reviewed-on: https://review.gluster.org/18254
Reviewed-by: Shreyas Siravara 
CentOS-regression: Gluster Build System 
Smoke: Gluster Build System

posix-acl: Add assume-permissive option for EACCES debugging / rug-sweeping.

2017-09-09T00:36:15+00:00

Summary:
Add assume-permissive option for EACCES debugging / rug-sweeping.
Re-fetch permissions when needed if they're absent.
This is a port of D5104707 & D5131597 to 3.8

Change-Id: I900fc66876ec8e73b04049f844c428b3d225d4ad
Reviewed-on: https://review.gluster.org/18249
Reviewed-by: Shreyas Siravara 
CentOS-regression: Gluster Build System 
Smoke: Gluster Build System

features/namespace: Add namespace xlator and link into brick graph

2017-09-07T23:02:56+00:00

Summary:
This translator tags namespaces with a unique hash that corresponds to the
top-level directory (right under the gluster root) of the file the fop acts
on. The hash information is injected into the call frame by this translator,
so this namespace information can later be used to do throttling, QoS and
other namespace-specific stats collection and actions in later xlators
further down the stack.

When the translator can't find a path directly for the fd_t or loc_t, it winds
a GET_ANCESTRY_PATH_KEY down to the posix xlator to get the path manually.
Caching this namespace information in the inode makes sure that most requests
don't need to recalculate the hash, so that typically fops are just doing an
inode_ctx_get instead of the more expensive code paths that this xlator can take.

Right now the xlator is hard-coded to only hash the top-level directory, but
this could be easily extended to more sophisticated matching by modification
of the parse_path function.

Test Plan:
Run `prove -v tests/basic/namespace.t` to see that tagging works.

Change-Id: I960ddadba114120ac449d27a769d409cc3759ebc
Reviewed-on: https://review.gluster.org/18041
Smoke: Gluster Build System 
Reviewed-by: Shreyas Siravara 
CentOS-regression: Gluster Build System

Add a bounded queue implementation.

2017-09-03T15:45:26+00:00

Summary:
- This queue will be used to hold the set of directory crawl / file
migrate operations in the multi-threaded rebalance.
- This is a port of D3712047 to 3.8

Test Plan: Unit test included.

Reviewed By: sshreyas

Change-Id: I25497a64beba744430807b3512eaee5d90f089c4
Reviewed-on: https://review.gluster.org/18197
Reviewed-by: Shreyas Siravara 
CentOS-regression: Gluster Build System 
Smoke: Gluster Build System

glusterd: Log & print old clients when doing a volume set operation

2017-09-03T05:42:35+00:00

Summary:
- Prior to this diff, Gluster would simply log "One more more clients cannot ..."
- With this diff, we now show up to 20 clients that are mismatched.
- This is a port of D3313082 to 3.8

Reviewers: rwareing, kvigor

Reviewed By: kvigor

Change-Id: Ia8830f18c922bda1aee787a2e3d6033164bb64d4
Reviewed-on: https://review.gluster.org/18196
Reviewed-by: Shreyas Siravara 
CentOS-regression: Gluster Build System 
Smoke: Gluster Build System