glusterfs.git/xlators/debug, branch release-3.8-fb

io-stats: re-port changes since 3.6 on top of FB version

2017-09-15T21:16:04+00:00

Includes io-stats parts of the following patches:

  1e421a5 logging: Avoid re-initing log level in io-stats
  0facb11 io-stats: Fix overwriting of client profile by the bricks
  91004b0 debug/io-stats: Disable fop stats dump by default
  62f9659 all: fix various cppcheck warnings
  e62c0fe build: export minimum symbols from xlators for correct resolution
  1d0a0d1 core: use syscall wrappers instead of direct syscalls - tail
  0773ca6 all: reduce "inline" usage
  8a9328e build: do not #include "config.h" in each file
  320455b io-stats: Fixing dereference after null check.
  28397ca Avoid conflict between contrib/uuid and system uuid
  49d6894 io-stats : null dereference coverity fix.

Change-Id: If1bdad6244e5749c6d8c456e6c64b5c5b483e273
Signed-off-by: Jeff Darcy

Replace namespace/io-stats/io-threads with 3.6-fb versions

2017-09-15T20:47:01+00:00

This rolls up multiple patches related to namespace identificaton and
throttling/QoS.  This primarily includes the following, all by Michael
Goulet .

  io-threads: Add weighted round robin queueing by namespace
  https://phabricator.facebook.com/D5615269

  io-threads: Add per-namespaces queue sizes to IO_THREADS_QUEUE_SIZE_KEY
  https://phabricator.facebook.com/D5683162

  io-threads: Implement better slot allocation algorithm
  https://phabricator.facebook.com/D5683186

  io-threads: Only enable weighted queueing on bricks
  https://phabricator.facebook.com/D5700062

  io-threads: Update queue sizes on drain
  https://phabricator.facebook.com/D5704832

  Fix parsing (-1) as default NS weight
  https://phabricator.facebook.com/D5723383

Parts of the following patches have also been applied to satisfy
dependencies.

  io-throttling: Calculate moving averages and throttle offending hosts
  https://phabricator.fb.com/D2516161
  Shreyas Siravara 

  Hook up ODS logging for FUSE clients.
  https://phabricator.facebook.com/D3963376
  Kevin Vigor 

  Add the flag --skip-nfsd-start to skip the NFS daemon stating, even if
  it is enabled
  https://phabricator.facebook.com/D4575368
  Alex Lorca 

There are also some "standard" changes: dealing with code that moved,
reindenting to comply with Gluster coding standards, gf_uuid_xxx, etc.

This patch *does* revert some changes which have occurred upstream since
3.6; these will be re-applied as apppropriate on top of this new base.

Change-Id: I69024115da7a60811e5b86beae781d602bdb558d
Signed-off-by: Jeff Darcy

Get glusterfs to output p50, p90, and p95 latencies

2017-09-13T20:22:11+00:00

Summary:
  [done] separate p99 dumping into general funcs
  [done] add p95, p90, and p50 stats
    - add p95, p90, p50 within p99, and generalize
    - rename config to dump-percentile-lantencies

Test Plan:
  make install glusterfs on dev machine.
  gluster volume create $name ...
  mount volume on /mnt/$name 
  dd if=/dev/zero of=/mnt/$name/test
  check each brick for pn printing

    /var/lib/glusterd/stats/glusterfsd__$brick.dump

Reviewers: sshreyas, kvigor, jdarcy

Reviewed By: jdarcy

Differential Revision: https://phabricator.intern.facebook.com/D5645951

Change-Id: Ic8ada48d9772bf2d5b3a2ba3c845d91d4e03c9d3
Signed-off-by: Jeff Darcy 
Reviewed-on: https://review.gluster.org/18279
Reviewed-by: Jeff Darcy 
Tested-by: Jeff Darcy 
CentOS-regression: Gluster Build System 
Smoke: Gluster Build System

Get glusterfs to output p50, p90, and p95 latencies

2017-09-13T19:32:29+00:00

Summary:
  [done] separate p99 dumping into general funcs
  [done] add p95, p90, and p50 stats
    - add p95, p90, p50 within p99, and generalize
    - rename config to dump-percentile-lantencies

Test Plan:
  make install glusterfs on dev machine.
  gluster volume create $name ...
  mount volume on /mnt/$name 
  dd if=/dev/zero of=/mnt/$name/test
  check each brick for pn printing

    /var/lib/glusterd/stats/glusterfsd__$brick.dump

Reviewers: sshreyas, kvigor, jdarcy

Reviewed By: jdarcy

Differential Revision: https://phabricator.intern.facebook.com/D5645951

Change-Id: I7bcd7201fc3753316db0ece809491a1cbdbefd32
Signed-off-by: Jeff Darcy 
Reviewed-on: https://review.gluster.org/18278
Reviewed-by: Jeff Darcy 
Tested-by: Jeff Darcy 
CentOS-regression: Gluster Build System 
Smoke: Gluster Build System

added p99 support to the samples logging

2017-09-13T18:31:53+00:00

Summary:
added global and by-fop-type calculation of p99 latency
to the sampled fop data

Test Plan:
build local glusterfs mount and looked at
the stats while dd if=/dev/zero of=/mnt/fuse/groot/share1/test1 bs=5

Reviewers: sshreyas, mgoulet, jdarcy

Reviewed By: jdarcy

Subscribers: jdarcy

Differential Revision: https://phabricator.intern.facebook.com/D5597662

Change-Id: I3f5cd9c0ea59ae4357827fcbd19bbf009e661c05
Signed-off-by: Jeff Darcy 
Reviewed-on: https://review.gluster.org/18277
Reviewed-by: Jeff Darcy 
Tested-by: Jeff Darcy 
CentOS-regression: Gluster Build System 
Smoke: Gluster Build System

inodelk-count: Add stats to count the number of lock objects

2017-09-13T14:13:58+00:00

Summary:
We want to track the number of locks held by the locks xlator. One of the ways to do it would be to track the
total number of pl_lock objects in the system.

This patch tracks the total number of pl_lock object and exposes the stat via io-stats JSON dump.

Test Plan: WIP, haven't got a pass. Putting the diff to get a sense of this approach would yield what you guys are looking for?

Reviewers: kvigor, sshreyas, jdarcy

Reviewed By: jdarcy

Differential Revision: https://phabricator.intern.facebook.com/D5303071

Change-Id: I946debcbff61699ec28b4d6f243042440107a224
Signed-off-by: Jeff Darcy 
Reviewed-on: https://review.gluster.org/18273
Reviewed-by: Jeff Darcy 
Tested-by: Jeff Darcy 
CentOS-regression: Gluster Build System 
Smoke: Gluster Build System

debug/io-stats: Track outstanding requests

2017-09-08T04:14:24+00:00

Summary:
Add outstanding-req field to track requests that have been sent
down the stack and haven't come back.
This is a port of D4908836 to 3.8

Reviewers: sshreyas

Change-Id: I5870f63008d553416109c1808a434f526f5a633d
Reviewed-on: https://review.gluster.org/18236
Reviewed-by: Shreyas Siravara 
CentOS-regression: Gluster Build System 
Smoke: Gluster Build System

io-stats: Error count and rate collection

2017-09-07T23:42:44+00:00

Summary:
- This diff adds error counts and rates to the regular io-stats dump.
- It outputs keys that look like this:

  "storage.gluster.nfsd.groot.aggr.errors..count": "6",
  "storage.gluster.nfsd.groot.inter.errors..per_sec": "0.00"

-  is the lowercase representation of errno values (e.g., ENOENT -> enoent, etc.)
- This is a port of D4691581 to 3.8

Reviewers: dph, kvigor

Reviewed By: kvigor

Change-Id: I96857d4283c47f9d330ae1978f113013e7c78a87
Reviewed-on: https://review.gluster.org/18230
Smoke: Gluster Build System 
Reviewed-by: Shreyas Siravara 
CentOS-regression: Gluster Build System

debug/io-stats: Restrict io-thread queue depth stats to NFS

2017-09-03T18:30:38+00:00

Summary:
- Fixes the unecessary log spew in other daemons
- This is a port of D3646627 to 3.8

Reviewers: rwareing, kvigor

Reviewed By: kvigor

Change-Id: Id54ab41cdfdd2006d3af2d8774c38025c566c523
Reviewed-on: https://review.gluster.org/18199
Reviewed-by: Shreyas Siravara 
CentOS-regression: Gluster Build System 
Smoke: Gluster Build System

auditing: Sample creation & removal of filesystem entries as well as errors

2017-09-03T17:28:55+00:00

Summary:
- Adds the ability for gluster to log every single CREATE and UNLINK that happens on the bricks (right before invoking sys_unlink() or open(...| O_CREAT)
- Makes it so that CREATEs and UNLINKs are not downsampled in io-stats
- This is a port of D3268156, D3778968, D3903894 & D3301527 to 3.8

Reviewed By: kvigor

Change-Id: I1bce28068c02b7d202f094094237646b4d39794b
Reviewed-on: https://review.gluster.org/18198
Reviewed-by: Shreyas Siravara 
CentOS-regression: Gluster Build System 
Smoke: Gluster Build System